NSX vCloud Retrofit: Controller Deployment, Host Preperation and VXLAN Config

This blog series extends my NSX Bytes Blog Posts to include a more detailed look at how to deploy NSX 6.1.x into an existing vCloud Director Environment. Initially we will be working with vCD 5.5.x which is the non SP Fork of vCD, but as soon as an upgrade path for 5.5.2 -> 5.6.x is released I’ll be including the NSX related improvements in that release.

Part 3: Controller Deployment, Host Preperation and VXLAN Config

With the NSX Manager deployed and configured and after verifying that vShield Edges are still being deployed by vCloud Director we can move onto the nuts and bolts of the NSX Configuration. There are a number of posts out there on this process so I’ll keep it relatively high level…but do check out the bottom of this post for more detailed install links.

Controller Deployment:

Just in case this step hasn’t been done in previous product installs…a best practice for most new generation VMware Applications is to have the Managed IP Address set under vCenter Server Settings -> Runtime Settings -> Managed IP Address Set the IP that of your vCenter.

Next login to the Web Client -> Networking and Security -> NSX Managers -> IP Address of NSX Manager -> Manage -> Grouping Objects -> IP Pools and Click Add.

Here we are going to preconfigure the IP Pools that are going to be used by the NSX Controllers. At this point we can also add the IP Pool that will be used by the VXLAN VMKernel Interfaces which become our VTEP’s. If we are routing our VXLAN Transport VLAN then add as many IP Pools as you need to satisfy the routed subnets in the Transport Zones.

For the NSX Controllers its recommended that 3 (5 Possible…7 max) be deployed for increased NSX Resiliency. The idea is to split them across the Management Network and on ESXi Hosts as diverse as possible. They can be split across different vCenter Clusters in the a vCenter Datacenter…Ideally there should be configured with DRS Anti Affinity Rules to ensure a single host failure doesn’t result in a loss of Cluster Quorum.

Go to the Web Client -> Networking and Security -> Installation In the NSX Controller Nodes Pane click on add

  • Select the NSX Manager, Datacenter, Cluster/Resource Pool, Datastore
  • Leave the Host blank (allow for auto placement)
  • On the Connected To, click Select and go to the Distributed PortGroup List and Select the Zone Management PortGroup
  • On the IP Pool, click Select and Choose the NSX Controller IP Pool created above

The Deployment of the NSX Controllers can be monitored via vCenter and will take about 5-10 minutes. The deployment will fail if Compute resources are not available or if the Controllers can’t talk to vCenter on the configured PortGroup.

LINK: NSX Controller Deployment Gotchyas

Host Preparation:

In the Networking & Security Menu go to Installation and Host Preparation. Here you will see the Clusters in the vCenter and their Installation Status. Once you click Install all hosts in the Cluster are Prepared at once…Preparing the hosts involves the installing of the following components:

  • UWA – User World Agent
  • DFW – Distributed Firewall Module
  • DLR – Distributed Router Module
  • VXLAN Module

The installation of these VIBs is done without interruptions to host operations and doesn’t result in Maintenance Mode being triggered during the install a reboot is not required.

VXLAN Configuration:

Once the Hosts have been prepared the Configure option becomes available in the VXLAN column of the Host Preperation tab. This process will create the initial VXLAN PortGroup under the selected Distributed Switch and add new VMKernel Interfaces to each Host in the prepared Cluster…the IP of which will act as the VTEP (VXLAN Tunnel End Point) and is from which all VM traffic passes if VXLAN enabled. There are a number of different Teaming Policies available and each choice depends on the design of your switching network…I chose Failover due to the fact LACP was not available and each ESXi host has 2x10Gig pNICs and I am comfortable with a failover scenario.

  • Select the Distributed Switch relative to each zone
  • Set the VLAN (Configured to be carried throughout the underlying Physical Network)
  • Set the MTU to 1600 (at least to allow overhead)
  • Use the VXLAN Up Pool create in previous steps
  • Set the Teaming Policy to Fail Over
  • Leave the VTEP to 1 (Can only be one if Failover is selected)

As mentioned above a new Distributed Port Group will be created and VMK NICs added to each host in the Cluster

 

Once a Cluster is Prepared any host that gets added to the cluster will have the Agents and Networking automatically configured…the removal of the NSX Components does require a host to be restarted. In my experience of upgrading from NSX 6.x to 6.1 host reports are also required.

Further Reading:

http://networkinferno.net/nsx-compendium

http://wahlnetwork.com/2014/06/02/working-nsx-deploying-nsx-controllers-via-gui-api/

http://wahlnetwork.com/2014/06/12/working-nsx-preparing-cluster-hosts/

http://wahlnetwork.com/2014/07/07/working-nsx-configuring-vxlan-vteps/

http://www.pluralsight.com/courses/vmware-nsx-intro-installation 

3 comments

  • Excellent Post.

    I have couple of questions –

    1. With vCloud Director 5.5.2, and vShield Manager (VSM) upgraded to NSX 6.1.2, can we use Distributed Routing and Firewall for VMs deployed via vCloud Director ? or is it only just limited to vCNS Edge 5.X via vSheild APIs ?

    2. If VSM has been upgraded to NSX 6.1.2, Can we still maintain a separate NSX instance for non-vcloud infrastructure on the same vCenter ? Knowing that vCloud Director will eventually phase out, we want to keep vShield Manager separate as we build and operate a parallel vRealize Cloud Automation infrastructure, if at all possible.

    3. Going through KB – vCloud Director 5.x and VMware NSX for vSphere 6.x interoperability guidelines (2096351)

    “Edge gateway deployment can be controlled at the Org VDC level by limiting the span of VLANs used for the uplink to the Edge VDS. However, vApp level Edges using VXLAN logical switches are deployed in the compute clusters.”

    What does this vApp thing exactly mean ?

    4. Does anything change with vCloud Director 5.6 + NSX ? is the Distributed FW and Routing available via vCloud Director?

    • Hey there Swap…I’ll answer below as best I can

      1. There is nothing theoretically adding vCD VMs to a logical switch that has the Distributed functions, but this is not nothing that can be exposed or controlled by tenants. Normal vCNS features apply when deploying from vCD that call the NSX Manager APIs…which are VSE API calls.

      2. You in place upgrade VSM Manager to NSX Manager and as far as I am aware (and logically speaking) there can be only one registered per vCenter.

      3. Stay tuned for my last Blog Post in the series that I’ll try get out next week…it explains this and finished off the retrofit

      4. Negative…NSX is not multi-tenant capable out of the box for Distributed features…best you can do at this point is utilize the enhanced ESGs to replace VSEs for vDCs…but keep your ear open towards the end of the year for possible announcements around this.

  • Hi Anthony,

    reading with very much interest this series. Just a quick note wrt NSX Controllers. It’s probably worth mentioning that although VMware says you can deploy 5 or even 7 controllers, officially and up to present, only 3 are supported. (weird I know!)

    Keep it up! 😉