Monthly Archives: December 2014

NSX vCloud Retrofit: Controller Deployment, Host Preperation and VXLAN Config

This blog series extends my NSX Bytes Blog Posts to include a more detailed look at how to deploy NSX 6.1.x into an existing vCloud Director Environment. Initially we will be working with vCD 5.5.x which is the non SP Fork of vCD, but as soon as an upgrade path for 5.5.2 -> 5.6.x is released I’ll be including the NSX related improvements in that release.

Part 3: Controller Deployment, Host Preperation and VXLAN Config

With the NSX Manager deployed and configured and after verifying that vShield Edges are still being deployed by vCloud Director we can move onto the nuts and bolts of the NSX Configuration. There are a number of posts out there on this process so I’ll keep it relatively high level…but do check out the bottom of this post for more detailed install links.

Controller Deployment:

Just in case this step hasn’t been done in previous product installs…a best practice for most new generation VMware Applications is to have the Managed IP Address set under vCenter Server Settings -> Runtime Settings -> Managed IP Address Set the IP that of your vCenter.

Next login to the Web Client -> Networking and Security -> NSX Managers -> IP Address of NSX Manager -> Manage -> Grouping Objects -> IP Pools and Click Add.

Here we are going to preconfigure the IP Pools that are going to be used by the NSX Controllers. At this point we can also add the IP Pool that will be used by the VXLAN VMKernel Interfaces which become our VTEP’s. If we are routing our VXLAN Transport VLAN then add as many IP Pools as you need to satisfy the routed subnets in the Transport Zones.

For the NSX Controllers its recommended that 3 (5 Possible…7 max) be deployed for increased NSX Resiliency. The idea is to split them across the Management Network and on ESXi Hosts as diverse as possible. They can be split across different vCenter Clusters in the a vCenter Datacenter…Ideally there should be configured with DRS Anti Affinity Rules to ensure a single host failure doesn’t result in a loss of Cluster Quorum.

Go to the Web Client -> Networking and Security -> Installation In the NSX Controller Nodes Pane click on add

  • Select the NSX Manager, Datacenter, Cluster/Resource Pool, Datastore
  • Leave the Host blank (allow for auto placement)
  • On the Connected To, click Select and go to the Distributed PortGroup List and Select the Zone Management PortGroup
  • On the IP Pool, click Select and Choose the NSX Controller IP Pool created above

The Deployment of the NSX Controllers can be monitored via vCenter and will take about 5-10 minutes. The deployment will fail if Compute resources are not available or if the Controllers can’t talk to vCenter on the configured PortGroup.

LINK: NSX Controller Deployment Gotchyas

Host Preparation:

In the Networking & Security Menu go to Installation and Host Preparation. Here you will see the Clusters in the vCenter and their Installation Status. Once you click Install all hosts in the Cluster are Prepared at once…Preparing the hosts involves the installing of the following components:

  • UWA – User World Agent
  • DFW – Distributed Firewall Module
  • DLR – Distributed Router Module
  • VXLAN Module

The installation of these VIBs is done without interruptions to host operations and doesn’t result in Maintenance Mode being triggered during the install a reboot is not required.

VXLAN Configuration:

Once the Hosts have been prepared the Configure option becomes available in the VXLAN column of the Host Preperation tab. This process will create the initial VXLAN PortGroup under the selected Distributed Switch and add new VMKernel Interfaces to each Host in the prepared Cluster…the IP of which will act as the VTEP (VXLAN Tunnel End Point) and is from which all VM traffic passes if VXLAN enabled. There are a number of different Teaming Policies available and each choice depends on the design of your switching network…I chose Failover due to the fact LACP was not available and each ESXi host has 2x10Gig pNICs and I am comfortable with a failover scenario.

  • Select the Distributed Switch relative to each zone
  • Set the VLAN (Configured to be carried throughout the underlying Physical Network)
  • Set the MTU to 1600 (at least to allow overhead)
  • Use the VXLAN Up Pool create in previous steps
  • Set the Teaming Policy to Fail Over
  • Leave the VTEP to 1 (Can only be one if Failover is selected)

As mentioned above a new Distributed Port Group will be created and VMK NICs added to each host in the Cluster


Once a Cluster is Prepared any host that gets added to the cluster will have the Agents and Networking automatically configured…the removal of the NSX Components does require a host to be restarted. In my experience of upgrading from NSX 6.x to 6.1 host reports are also required.

Further Reading: 

NSX vCloud Retrofit: NSX Manager Configuration and vCD VSE Deployment Validation

This blog series extends my NSX Bytes Blog Posts to include a more detailed look at how to deploy NSX 6.1.x into an existing vCloud Director Environment. Initially we will be working with vCD 5.5.x which is the non SP Fork of vCD, but as soon as an upgrade path for 5.5.2 -> 5.6.x is released I’ll be including the NSX related improvements in that release.

Part 2 – NSX Manager Configuration and vCD VSE Deployment Validation

Once you have updated the VSM to the NSX Manager there are a number of configuration items to work through…some of which would have been carried over from the vCNS upgrade. For user and group management you can reference this post where I go through the configuration of the Management Services to allow users and groups to administor NSX through the vCenter Web Client.

Once you have a Green Connected Button for the Lookup Service and vCenter Service as seen above you can configure the rest of the settings. Clicking on the home Icon will give you the menu below:

Go to Manage Appliance Settings -> General and configure the Time Settings, Syslog Server and keep the Locale that is relevant to you installation. Ensure the NTP Server is set and is consistent with other NTP servers referenced in vCloud, vCenter and ESXi (Time Sync is Critical between NSX Manager, Hosts and other Management Systems)

Configure a SYSLOG or point the NSX Manager at Log Insight which has a newly released Content Pack for NSX.

Go to Network Settings and enter in new Host Name Details without the Domain Name specified (those are put of the search domains) and double check the IP and DNS Settings

Note 1: Create a DNS entry (if not already created) for the Host Name ensuring there is a reverse lookup in place for internal name resolution of the Manager.

Go to Backup and Restore and (re)configure the Backup Settings to include an FTP location and an additional Pass Phrase for NSX Manager Restores.

Once done, perform a test backup

vShield Edge Deployment and Validation:

With that done we can now move onto to testing vCloud Director initiated deployments of the VSE 5.5.3 Edges that are deployed as legacy Appliances out of the NSX Manager. If you take a look under the covers of the NSX Manager you will see that it’s DNA is vShield and more to the point…the NSX portion has been itself retrofitted ontop of the vCNS VSM which has allowed for quick integration with vCenter and legacy interoperability with current versions of vCD.

vCloud Director will call vShield APIs (not NSX) to deploy edges for use with Virtual Datacenter Networking and all current functionality in the edges up to 5.5.3 are maintained. vCD will not be able to understand an NSX 6.1 ESG and if you upgrade (the option is there as shown below) you will have a fully functional Edge with all settings and config carried over…but not manageable by the vCloud GUI.

To ensure that all previous vCloud Director Deployment mechanisms and Edge Management is still functional deploy an Edge Gateway from the vCloud Director GUI checking to make sure that the OVF is deployed correctly…the service account will now be service.nsx (or the account you chose)

Validate the vShield Version at 5.5.3, Test Internal/External Access and IP Connectivity, Service Configurations by adding rules, disabling/enabling Firewall and Create and attaching a vORG Network and Check Port Group Status

If you are interested in what the 5.5.3 VSE Management looks like under the Network & Security Section of the Web Client, click on Edges and the Name of the Edge…what you see here is similar to what you would see for the 6.1 ESGs but with less functionality and features. What’s managed in the vCD GUI is what you see here.

With that validated you have ensured that vCloud Director will continue to do it’s thing and work as expected with NSX Manager in play…at this point we are not using any VXLAN Virtualwires or NSX Transport Zones Network Pools…that’s still to come!

NSX vCloud Retrofit: Intro and VSM to NSX Manager Upgrade

I’ve been working over the past 6 months on and off looking at how to best fit NSX into existing vCloud Director Platforms and while vCD in the Enterprise is going to become less a thing…vCloud Air Network Service Providers will continue to use vCD SP…The feature set provided by NSX could greatly enhance any SPs offering around enhanced networking services as well as helping IT Operations with with SDN abstraction efficiencies.

This blog series extends my NSX Bytes Blog Posts to include a more detailed look at how to deploy NSX 6.1.x into an existing vCloud Director Environment. Initially we will be working with vCD 5.5.x which is the non SP Fork of vCD, but as soon as an upgrade path for 5.5.2 -> 5.6.x is released I’ll be including the NSX related improvements in that release.

With that, if anyone is running or still looking to run vCD in-house as a Private Cloud Abstraction of vSphere and you are looking at implementing NSX this series will still be relevant.

To protect some of the work I’ve done with Zettagrid to productise NSX there will be sections where I will be vague…can’t give you guys all the secret sauce tricks and tweaks 🙂

NSX Deployment Pre-Requisites and Build Numbers*

  • Pre Assigned VLAN for VXLAN – MTU bigger or Equal to 1600
  • vCenter 5.5 Update 2 Build 2001466
  • ESXi 5.5 Update 2 Build 2068190
  • vCloud Director 5.5.2 Build 2000523
  • vShield Manager Build 2175698
  • SSO Service Details ([email protected])
  • NSX Service Account Details (service.nsx)
  • NSX Admin Group (NSX.Admins)

* These are based on my internal testing and deployment validations

Disclaimer: At the time of posting NSX 6.1.x is not officially supported with vCloud 5.6.3 or 5.5.2. While I haven’t come across any issues in the retrofit given the current status you may want to think twice about putting this into prod until VMware validate the interoperability…I’m working to get more info on that and will update when I know more. The matrix below is not up to date and there is support for NSX 6.0.6 and NSX 6.0.7.

Part 1 – VSM to NSX Manager Upgrade:

Be wary…this is a one time upgrade…once installed we can’t roll back easily. At the time of writing the lastest version of vCNS VSM is, if you are not at the build upgrade the VSM to that before you begin.

NOTE1: vShield Data Security Installs:

If you are upgrading from NSX version 6.1.1, or do not have Data Security in your environment then you are fine to skip this step…

If you are upgrading from a release prior to NSX 6.1.1 and have Data Security in your environment…as much as is seems like an extreme PITA the following needs to be done:

  1. Un-install Data Security from all the clusters that have the service installed.
  2. Upgrade NSX Manager to version 6.1.2 – SEE STEPS BELOW. 
  3. Install or upgrade Guest Introspection and other services on appropriate clusters.
  4. Install Data Security on appropriate clusters.
  5. Upgrade the remaining components.

NOTE2: vShield Edge instances prior to version 5.5 need to be upgraded to the latest version. Pre-5.5 vShield Edge instances cannot be managed or deleted after vShield Manager has been upgraded to NSX Manager.

vCNS to NSX Upgrade Process:

Back Up and Snapshot the VSM

  • Ensure that there is a Backup of the VSM Manager Config
  • Snapshot VSM Manager
  • Reboot the VSM to ensure any existing logs are cleared and there is enough space on the filesystem to install (>4GB)

Login to the VSM -> Settings and Reports -> Updates and Click on Choose File

Click on Upload File: VMware-vShield-Manager-upgrade-bundle-to-NSX-6.1.2-2318232.tar.gz

Confirm the Version Number in the New Version and Description Field

Confirm the Install

Let the Upgrade go through it’s paces

Once the VM has rebooted go to the IP of the VSM. We should now have the NSX Manager login Screen

Login using admin and and default as the user/password combination…I’ve found in all my upgrades so far that the existing admin password had been reset, however I have had to use the previous password to access vShield API calls… (possible bug)

Verify that the build is as shown below and that vCenter is registered by going to Manage vCenter Registration

NOTE3: Shutdown VM and upgrade the VM Hardware to vCPU to 4 and vRAM to 12GB and restart the VM. This will allow the upgraded VM to meet the NSX Manager Compute Requirements.

Once completed the NSX Manager will boot and it’s time to verify the install and to ensure that no previous functionality is lost.

Part 2: NSX Manager Configuration and vCloud Director VSE Deployment Validation