Category Archives: NSX

Released: NSX-v 6.3.6

Last week VMware released NSX-v 6.3.6 (Build 8085122) that doesn’t contain any new features but addresses a number of bug fixes from previous releases. This has been done independently of any updated release of NSX-v 6.4.0 that went GA in January.

This is good to see though interesting to also see that people are still not upgrading to 6.4.0 in droves meaning VMware needs to support both versions. Going through the release notes there are a lot of known issues that should be known and there are more than a few that apply to service providers.

Some key fixes are listed below:

Important Fixes :

  • Network outage of ~40-50 seconds seen on Edge Upgrade – During Edge upgrade, there is an outage of approximately 40-50 seconds
  • After upgrading to 6.3.5, the routing loop between DLR and ESG’s causes connectivity issues in certain BGP configurations –  A routing loop is causing a connectivity issue
  • NSX Manager CPU high due to edge in read-only file system mode – NSX Manager is slow to respond because it keeps 100% CPU and receives a lot of read-only file system events from edge.
  • After upgrade from vCNS edge 5.5.4 to NSX 6.3.6, customers could not configure Health-Check-Monitor port nor make any changes directly from vCD – Customers will not be able to configure Health-Check-Monitor port nor make any changes directly from vCD.
  • Distributed Firewall stays in Publishing state with certain firewall configurations – Distributed Firewall stays in “Publishing” state if you have a security group that contains an IPSet with 0.0.0.0/0 as an EXCLUDE member, an INCLUDE member, or as a part of ‘dynamic membership containing Intersection (AND)’

Those with the correct entitlements can download NSX-v 6.3.6 here.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_636.html

NSX Bytes: Updated – NSX Edge Feature and Performance Matrix

For a few years now i’ve been compiling features and throughput numbers for NSX Edge Services Gateways. This started off comparing features and performance metrics between vShield Edges and NSX Edges. As the product evolves, so does it’s capabilities and given the last time I updated this was around the time of NSX-v 6.2 I thought it was time for an update.

A reminder that VMware announced the End of Availability (“EOA”) of the VMware vCloud Networking and Security 5.5.x that kicked in on the September of 19, 2016 and that from vCloud Director 8.10 and above vShield Edges are no longer supported…hence why I don’t have the VSE listed in the tables. For those still running VSEs for what ever reason, you can reference my original post here.

As a refresher…what is an Edge device?

The Edge Services Gateway (NSX-v) connects isolated, stub networks to shared (uplink) networks by providing common gateway services such as DHCP, VPN, NAT, dynamic routing, and Load Balancing. Common deployments of Edges include in the DMZ, VPN Extranets, and multi-tenant Cloud environments where the Edge creates virtual boundaries for each tenant.

The following relates to ESG maximums per NSX and ESXi maximums.

Item Maximums
ESGs per NSX Manager 2,000
ESGs per ESXi Host 250
ESG Interfaces 10 (Including Internal, Uplink and Trunk)
ESG Subinterfaces 200
The function of an ESG is as follows:

The ESG gives you access to all NSX Edge services such as firewall, NAT, DHCP, VPN, load balancing, and high availability. You can install multiple ESG virtual appliances in a datacenter. Each ESG virtual appliance can have a total of ten uplink and internal network interfaces. With a trunk, an ESG can have up to 200 subinterfaces. The internal interfaces connect to secured port groups and act as the gateway for all protected virtual machines in the port group. The subnet assigned to the internal interface can be a publicly routed IP space or a NATed/routed RFC 1918 private space. Firewall rules and other NSX Edge services are enforced on traffic between network interfaces.

Below is a list of services provided by the NSX Edge.

Service Description
Firewall Supported rules include IP 5-tuple configuration with IP and port ranges for stateful inspection for all protocols
NAT Separate controls for Source and Destination IP addresses, as well as port translation
DHCP Configuration of IP pools, gateways, DNS servers, and search domains
Site to Site VPN Uses standardized IPsec protocol settings to interoperate with all major VPN vendors
SSL VPN SSL VPN-Plus enables remote users to connect securely to private networks behind a NSX Edge gateway
Load Balancing Simple and dynamically configurable virtual IP addresses and server groups
High Availability High availability ensures an active NSX Edge on the network in case the primary NSX Edge virtual machine is unavailable
Syslog Syslog export for all services to remote servers
L2 VPN Provides the ability to stretch your L2 network.
Dynamic Routing Provides the necessary forwarding information between layer 2 broadcast domains, thereby allowing you to decrease layer 2 broadcast domains and improve network efficiency and scale. Provides North-South connectivity, thereby enabling tenants to access public networks.

Below is a table that shows the different sizes of each edge appliance and what (if any) impact that has to the performance of each service. As a disclaimer the below numbers have been cherry picked from different sources and are subject to change.

NSX Edge (Compact) NSX Edge (Large) NSX Edge (Quad-Large) NSX Edge (X-Large)
vCPU 1 2 4 6
Memory 512MB 1GB 1GB 8GB
Disk 512MB 512MB 512MB 4.5GB + 4GB
Interfaces 10 10 10 10
Sub Interfaces (Trunk) 200 200 200 200
NAT Rules 2,048 4,096 4,096 8,192
ARP Entries
Until Overwrite
1,024 2,048 2,048 2,048
FW Rules 2000 2000 2000 2000
FW Performance 3Gbps 9.7Gbps 9.7Gbps 9.7Gbps
DHCP Pools 20,000  20,000  20,000  20,000
ECMP Paths 8 8 8 8
Static Routes 2,048 2,048 2,048 2,048
LB Pools 64 64 64 1,024
LB Virtual Servers 64 64 64 1,024
LB Server / Pool 32 32 32 32
LB Health Checks 320 320 320 3,072
LB Application Rules 4,096 4,096 4,096 4,096
L2VPN Clients Hub to Spoke 5 5 5 5
L2VPN Networks per Client/Server 200 200 200 200
IPSec Tunnels 512 1,600 4,096 6,000
SSLVPN Tunnels 50 100 100 1,000
SSLVPN Private Networks 16 16 16 16
Concurrent Sessions 64,000 1,000,000 1,000,000 1,000,000
Sessions/Second 8,000 50,000 50,000 50,000
LB Throughput L7 Proxy) 2.2Gbps 2.2Gbps 3Gbps
LB Throughput L4 Mode) 6Gbps 6Gbps 6Gbps
LB Connections/s (L7 Proxy) 46,000 50,000 50,000
LB Concurrent Connections (L7 Proxy) 8,000 60,000 60,000
LB Connections/s (L4 Mode) 50,000 50,000 50,000
LB Concurrent Connections (L4 Mode) 600,000 1,000,000 1,000,000
BGP Routes 20,000 50,000 250,000 250,000
BGP Neighbors 10 20 100 100
BGP Routes Redistributed No Limit No Limit No Limit No Limit
OSPF Routes 20,000 50,000 100,000 100,000
OSPF LSA Entries Max 750 Type-1 20,000 50,000 100,000 100,000
OSPF Adjacencies 10 20 40 40
OSPF Routes Redistributed 2000 5000 20,000 20,000
Total Routes 20,000 50,000 250,000 250,000

Of interest from the above table it doesn’t list any Load Balancing performance number for the NSX Compact Edge…take that to mean that if you want to do any sort of load balancing you will need NSX Large and above. To finish up, below is a table describing each NSX Edge size use case.

Use Case
NSX Edge (Compact) Small Deployment, POCs and single service use
NSX Edge (Large) Small/Medium DC or mult-tenant
NSX Edge (Quad-Large) High Throughput ECMP or High Performance Firewall
NSX Edge (X-Large) L7 Load Balancing, Dedicated Core

The Quad Large model is suitable for high performance firewall abilities and the X-Large is suitable for both high performance load balancing and routing. You can convert between NSX Edge service gateway sizes upon demand using a non-disruptive upgrade process, so the recommendation is to begin with the Large model and scale up if necessary. A Large NSX Edge service gateway is suitable for medium firewall performance but as detailed later, the NSX Edge service gateway does not perform the majority of firewall functions.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.4/NSX%20for%20vSphere%20Recommended%20Configuration%20Maximums_64.pdf

https://docs.vmware.com/en/VMware-Validated-Design/4.2/com.vmware.vvd.sddc-design.doc/GUID-FCEA948E-7F8B-4FF0-857B-12D6E045BF1D.html

NSX Bytes: NSX 6.4 UI Enhancements and Upgrade Coordinator

NSX-v 6.4 was released a couple of weeks ago and as I talked about in my launch post, there are a lot of new features and enhancements that make this release significant. A big focus for this release was around enhancing NSX’s ease of use and serviceability. There have been a number of additions to the UI with additional dashboards and menu items. Also importantly, a first port of the NSX Web Client functionality over the to HTML5 Web Client.

What’s interesting about the approach that the NSX product team has taken is that they have decided to have each new feature in the HTML5 Web Client accessible from the old Flash based Web Client as well. They have also continued to improve on the layout and usability of the flash based vSphere Web Client so what you have now is a combination of Flash and HTML5 inside the old Web Client as well as a limited pure HTML5 NSX experience in the new Web Client.

UI Enhancements:

Among the enhancements to the UI is the improvement in the navigation menu where some commonly used menu items that where clicks away have been brought into the main tree. As you can see below there is a lot more happening in the 6.4 menu tree on the right vs the previous releases on the left.

The HTML5 menu is a little shorter with only a couple of items added however it shows you what it will look like when the porting is complete. Also shown in the picture below is the new System Scale Dashboard that provides visibility into the current usage of various NSX components and system capacity relative to configuration maximums with warning thresholds configurable.

Highlighting the Flash+HTML cross over in the Flash Web Client, the System Scale Dashboard is also present in the old Web Client and shown below.

In terms of other UI additions there is now an EAM status monitor in the Host Preparation Tab and a direct way from the Web Client to generate Support Bundle…which again, is available from both Web Clients.

NSX Upgrade Coordinator:

Probably one of the coolest features in NSX-v 6.4 is the Upgrade Coordinator.

When you upgrade using Upgrade Coordinator, you can select to perform a One Click Upgrade, where everything is upgraded during one upgrade session. Or you can select to Plan Your Upgrade, and customize which components are upgraded, and organize component objects into upgrade groups.

Working you way through the wizard you can select which components to upgrade.

For me have control of the NSX Edge upgrades is super important as this has historically been a monotonous task for Service Providers with lots of customer using vCloud Director Edge services. The Upgrade Coordinator streamlines this upgrade task and makes the process a lot more efficient.

Having the ability to group and order the upgrade process for Edges (and Service VMs) is also an excellent enhancement. Once the wizard has been completed you are shown a progress dashboard which you can click into to view the current state of upgrading components.

Once completed, you should have all components upgraded and you can go through the post upgrade tasks and once completed you can always get an overview of the NSX environment by clicking on the main dashboard.

Conclusion:

There is a lot to like about where the NSX team is taking the user interface and it’s good to see an initial move over to the HTML5 Web Client while also having that same functionality still accessible via the Flash Web Client. To have a loot at what is currently supported and what is not in the HTML5 vs Flash Client head to this page and check out the support tables.

I’m looking forward to future updates that will look to push more functionality directly into the HTML5 Web Client.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.4/rn/nsx-vsphere-client-65-functionality-support.html

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.4/com.vmware.nsx.upgrade.doc/GUID-A539869B-9858-48B3-90ED-2336698EE386.html

NSX-v 6.4.0 Released! What’s in it for Service Providers

This week VMware released NSX-v 6.4.0 (Build 7564187) and with it comes a new UI Plug-in for vSphere Client (HTML5) which includes some new dashboards including a new Update Lifecycle Manager built right into the Web Client. Reading through the release notes, for me the biggest improvements seem to be around NSX Edges and Edge services. These are central to Service Providers who offer NSX services with vCloud Director or otherwise via their service offerings. There are also as usual, a number of Resolved Issues which can be skimmed through in the release notes page.

What’s New:

As mentioned above there is a lot to get through and there are a lot of new enhancements and features packed into this release. I’ve gone through and picked the major ones as they might pertain to Service Providers running NSX on their platforms. I’ve basically followed the sections in the Release Notes but summarised for those that don’t want to troll through the page. Ad the end of each section i’ve commented on the benefits of the improvements.

Security Services

  • Identity Firewall now supports user sessions on remote desktop and application servers (RDSH) sharing a single IP address, new “fast-path” architecture improves processing speed of IDFW rules. Active Directory integration now allows selective synchronization for faster AD updates.
  • Distributed Firewall adds layer-7 application-based context for flow control and micro-segmentation planning.
  • Distributed Firewall rules can now be created as stateless rules at a per DFW section level.
  • Distributed Firewall supports VM IP realization in the hypervisor. This allows users to verify if a particular VM IP is part of a securitygroup/cluster/resourcepool/host.

These security features listed above will make a lot of people happy and improves end user experience and the DFW supporting within the VM is a small but important feature.

NSX User Interface

  • Support for vSphere Client (HTML5): Introduces VMware NSX UI Plug-in for vSphere Client (HTML5).
  • HTML5 Compatibility with vSphere Web Client (Flash): NSX functionality developed in HTML5 (for example, Dashboard) remains compatible with both vSphere Client and vSphere Web Client, offering seamless experience for users who are unable to transition immediately to vSphere Client.
  • Improved Navigation Menu: Reduced number of clicks to access key functionality, such as Grouping Objects, Tags, Exclusion List and System Configuration.

It’s great to see NSX jump over to the HTML5 Web Client and even though it’s a small first step its a great preview of what’s to come in future releases. The fact that it goes both ways, meaning older flash clients still have the features is important as well.

Operations and Troubleshooting

  • Upgrade Coordinator provides a single portal to simplify the planning and execution of an NSX upgrade. Upgrade Coordinator provides a complete system view of all NSX components with current and target versions, upgrade progress meters, one-click or custom upgrade plans and pre- and post-checks.
  • A new improved HTML5 dashboard is available along with many new components. Dashboard is now your default homepage. You can also customize existing system-defined widgets, and can create your own custom widgets through API.
  • New System Scale dashboard collects information about the current system scale and displays the configuration maximums for the supported scale parameters. Warnings and alerts can also be configured when limits are approached or exceeded.
  • A Central CLI for logical switch, logical router and edge distributed firewall reduces troubleshooting time with centralized access to distributed network functions.
  • New Support Bundle tab is available to help you collect the support bundle through UI on a single click. You can now collect the support bundle data for NSX components like NSX Manager, hosts, edges, and controllers.
  • New Packet Capture tab is available to capture packets through UI.
  • Multi-syslog support for up to 5 syslog servers.
  • API improvements including JSON support. NSX now offers the choice or JSON or XML for data formats. XML remains the default for backwards compatibility.

There is a lot going on here but for me it continues to solidify the vision that Martin Casado had around Nicira in it being efficient in software to get a deep view of what’s happened and what’s happening in your network. The System Scale dashboard (shown below) also is a great way to get an understanding of how loaded an NSX environment is…one of my favourite news features.

NSX Edge Enhancements

  • Enhancement to Edge load balancer health check. Three new health check monitors have been added: DNS, LDAP, and SQL.
  • You can now filter routes for redistribution based on LE/GE in prefix length in the destination IP.
  • Support for BGP and static routing over GRE tunnels.
  • NAT64 provides IPv6 to IPv4 translation.
  • Faster failover of edge routing services.
  • Routing events now generate system events in NSX Manager.
  • Improvements to L3 VPN performance and resiliency.

I’ve highlighted this in red because the improvements above continue to build on a very strong foundation that is the NSX Edge Gateway that still continues vShield DNA. Though I’ve been away from the day to day of a service provider for almost a year and a half I recognise that these new features create a more enterprise class of edge device. The little thing added will make network engineers happy.

Conclusion:

Overall this looks like a strong release for NSX-v and good to see that there is still a ton of development going into the platform. Service providers have the most to gain from this release which is a good thing! The only thing that I do hope is that as a 6.x.0 release that it’s stable and without any major bugs…the history of these first major release builds hasn’t been great but hopefully that’s a thing of the past with 6.4.0.

EDIT: Just to clarify after a couple of comments, it seems that for the moment vCD 9.0 and 8.20 is not compatible with NSX-v 6.4.0 just yet. More news when it comes to hand.

Resources:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.4/rn/releasenotes_nsx_vsphere_640.html

NSX Bytes – What’s new in NSX-T 2.1

In Feburary of this year VMware released NSX-T 2.0 and with it came a variety of updates that looked to continue to push of NSX-T beyond that of NSX-v while catching up in some areas where the NSX-v was ahead. The NSBU has big plans for NSX beyond vSphere and during the NSX vExpert session we saw how the future of networking is all in software…having just come back from AWS re:Invent I tend to agree with this statement as organisations look to extend networks beyond traditional on-premises or cloud locations.

NSX-T’s main drivers relate to new data centre and cloud architectures with more hetrogeneality driving a different set of requirements to that of vSphere that focuses around multi-domain environments leading to a multi-hypervisor NSX platform. NSX-T is highly extensible and will address more endpoint heterogeneity in future releases including containers, public clouds and other hypervisors. As you can see before the existing use cases for NSX-T are mainly focused around devops, micro-segmentation and multi-tenant infrastructure.

Layer 3 accessibility across all types of platforms.

What’s new in NSX-T 2.1:

Today at Pivotal SpringOne, VMware is launching version 2.1 of NSX-T and with it comes a networking stack underpinning Pivotal Container Services, direct integration with Pivotal Cloud Foundry and significant enhancements to load balancing capabilities for OpenStack Neutron and Kubernetes ingress. These load balancers can be virtual or bare metal. There is also native networking and security for containers and Pivotal operations manager integration.

NSX-T Native Load Balancer:
NSX-T has two levels of routers as shown above…then ones that connect to the physical world and the ones which are labeled T1 in the diagram above. Load balancing will be active on the T1 routers and have the following features:

  • Algorithms – Round Robin, Weighted Round Robin, Least Connections and Source IP Hash
  • Protocols – TCP, UDP, HTTP, HTTPS with passthrough, SSL Offload and End to end SSL
  • Health Checks – ICMP, TCP, UDP, HTTP, HTTPS
  • Persistance – Source IP, Cookie
  • Translation – SNAT, SNAT Automap and No SNAT

As well as the above it will have L7 manipulation as will as OpenStack and Kubernetes ingress. Like NSX-v these edges can be deployed in various sizes depending on the workload.

Pivotal Cloud Foundry and NSX-T:

For those that may not know, PCF is a cloud native platform for deploying and operating modern applications and in that NSX-T providers the networking to support those modern application. This is achieved via the Network Container Plugin. Cloud Foundry NSX-T topology include a separate network topology per orginization with every organization getting one T1 router. Logical switches are then attached per space. High performance north/south routing uses NSX routing infrastructure, including dynamic routing to the physical network.

For east/west traffic that happens container to container with every container having distributed firewall rules applied on it’s interface. There is also a number of visibility and troubleshooting counters attached to every container. NSX also controls the IP management by supplying subnets from IP blocks to namespaces and individual IPs and MACs to containers.

Log Insight Content Pack:

As part of this release there is also a new Log Insight NSX-T Content Pack that builds on the new visibility and troubleshooting enhancements mentioned above and allows Log Insight to monitor a lot of the container infrastructure with NSX.

Conclusion:

When it comes to the NSX-T 2.1 feature capabilities, the load balancing is a case of bringing NSX-T up to speed to where NSX-v is, however the thing to think about is that how those capabilities will or could be used beyond vSphere environments…that is the big picture to consider here around the future of NSX and it can be seen with the deeper integration into Pivotal Cloud Foundry.

Released: NSX-v 6.3.5 and New Features and Fixes

Last week VMware released NSX-v 6.3.5 (Build 7119875) that contains a few new features and addresses a number of bug fixes from previous releases. Going through the release notes there are a lot of known issues that should be known and there are more than a few that apply to service providers…specifically there are a lot around Logical and Edge Routing functions. The other interesting point to highlight about this release is that this is apparently the same build that runs on VMware on AWS instances as mentioned by Ray Budavari.

The new features in this build are:

  • For vCenter 6.5 and later, Guest Introspection VM’s, on deployment, will be named Guest Introspection (XX.XX.XX.XX), where XX.XX.XX.XX is the IPv4 address of the host on which the GI machine resides. This occurs during the initial deployment of GI.
  • Guest Introspection service VM will now ignore network events sent by guest VMs unless Identify Firewall or Endpoint Monitoring is enabled
  • You can also modify the threshold for CPU and memory usage system events with this API: PUT /api/2.0/endpointsecurity/usvmstats/usvmhealththresholds
  • Serviceability enhancements to L2 VPN including
    • Changing and/or enabling logging on the fly, without a process restart
    • Enhanced logging
    • Tunnel state and statistics
    • CLI enhancements
    • Events for tunnel status changes
  • Forwarded syslog messages now include additional details previously only visible on the vSphere Web Client
  • Host prep now has troubleshooting enhancements, including additional information for “not ready” errors

That last new feature above is seen below…you can see the EAM Status message just below the NSX Manager IP which is a nice touch given the issues that can happen if EAM is down.

If you click on the Not Ready Installation Status you now get a more detailed report of what could be wrong and suggestions of how to resolve.

Important Fixes :

  • VMs migrated from 6.0.x can cause host PSOD When upgrading a cluster from 6.0.x to 6.2.3-6.2.8 or 6.3.x, the VM state exported can be corrupted and cause the receiving host to PSOD
  • “Upgrade Available” link not shown if cluster has an alarm. Users are not be able to push the new service spec to EAM because the link is missing and the service will not be upgraded
  • NSX Manager crashes with high NSX Manager CPU NSX Manager has an OOM (out of memory) error and continuously restarts
  • NSX Controller memory increases with hardware VTEP configuration causing high CPU usage A controller process memory increase is seen with hardware VTEP configurations running for few days. The memory increase causes high CPU usage that lasts for some time (minutes) while the controller recovers the memory. During this time the data path is affected
  • Translated IPs are not getting added to vNIC filters which is causing Distributed Firewall to drop traffic When new VMs are deployed, the vNIC filters do not get updated with the right set of IPs causing Distributed Firewall to block the traffic.

Those with the correct entitlements can download NSX-v 6.3.5 here.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_635.html

Awarded vExpert Cloud – A New vExpert Sub Program

Last week Corey Romero announced the inaugural members of the vExpert Cloud sub-program. This is the third vExpert sub-program following the vSAN and NSX programs announced last year. There are 135 initial vExpert Cloud members who have been awarded the title. As it so happens I am now a member of all three which reflects on the focus I’ve had and still have around VMware’s cloud, storage and networking products leading up to and after my move to Veeam last year.

Even with my move, that hasn’t stopped me working around these VMware vertices as Veeam works closely with VMware to offer supportability and integration with vCloud Director as well as being certified with vSAN for data protection. And more recently as it pertains specifically to the vExpert Cloud program, we are going to be supporting vCloud
Director in v10 of Backup & Replication for Cloud Connect Replication and also at VMworld 2017 we where announced as a launch partner for data protection for VMware Cloud on AWS.

For those wondering what does it take to be a part of the vExpert Cloud program:

We are looking for vExperts who are evangelizing VMware Cloud and delivering on the principles of the multi-cloud world being the new normal. Specificity we are looking for community activities which follow the same format as the vExpert program (blogs, books, videos, public speaking, VMUG Leadership, conference sessions speaking and so on).

And in terms of the focus of the vExpert Cloud program:

The program is focused on VMware Cloud influencer activities, VMware, AWS and other cloud environments and use of the products and services in way that delivers the VMware Cloud reality of consistency across multi-cloud environments.

Again, thank you to Corey and team for the award and I look forward to continuing to spread the community messaging around Cloud, NSX and vSAN.

Released: NSX-v 6.3.4 and Upgrade Notes and Fixes

Last week VMware released NSX-v 6.3.4 (Build 6845891) that contains no specific new features but addresses a couple of bug fixes from previous releases. Going through the release notes there are a lot of known issues that should be known and there are more than a few that apply to service providers…specifically there are a lot around NSX Edge functions. The other interesting point to highlight about this release is that for those on NSX-v 6.3.3 there is are a couple of scripts to run against the API before upgrading to ensure all controllers are upgradable.

As mentioned, before upgrading the release notes stage that for those on NSX-v 6.3.3 they follow this VMwareKB. In a nutshell there is a bug in 6.3.3 where the NSX Controllers are reported as disconnected in the Web Client as shown below.

To fix that situation you need to execute a couple of API calls that POSTs a script to the NSX Manager as documented in the VMwareKB. This needs to be done as the NSX Manager Admin user as I found this didn’t work with an NSX Domain User or an SSO Administrator Account with NSX Org admin level permissions.

Once the second script has been run you should see a similar output to what’s shown above and have all NSX Controllers ready in a connected state which allows you to prepare for the upgrade. Once done, you can go through the normal NSX upgrade steps which will get you to the latest build.

Important Fixes :

  • Fixed Issue 1970527: ARP fails to resolve for VMs when Logical Distributed Router ARP table crosses 5K limit
  • Fixed Issue 1961105: Hardware VTEP connection goes down upon controller rebootA BufferOverFlow exception is seen when certain hardware VTEP configurations are pushed from the NSX Manager to the NSX Controller. This overflow issue prevents the NSX Controller from getting a complete hardware gateway configuration. Fixed in 6.3.4.
  • Fixed Issue 1955855: Controller API could fail due to cleanup of API server reference filesUpon cleanup of required files, workflows such as traceflow and central CLI will fail. If external events disrupt the persistent TCP connections between NSX Manager and controller, NSX Manager will lose the ability to make API connections to controllers, and the UI will display the controllers as disconnected. There is no datapath impact.

Those with the correct entitlements can download NSX-v 6.3.4 here.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_634.html

https://kb.vmware.com/kb/2151719

 

NSX Bytes: NSX-v 6.3.3 Released – Upgrade Notes and Enhancements

Last week VMware released NSX-v 6.3.3 (Build 6276725) and with it comes a new operating system for the NSX Controllers. Once upgraded the new controllers will be powered by Photon OS which is more and more making it’s way into VMware’s appliances. There are a few other new bits in this release but more importantly a number of Resolved Issues. For those running homelabs with one NSX Controller there are some important upgrade notes to be made aware of before kicking off…i’ll go into those below.

Compatibility:

Before moving to the upgrade there are some important notes around interoperability and supported ESXi versions as is explained in this VMwareKB. The minimum supported version of ESXi running with NSX-v 6.3.3 is as shown below:

  • NSX-v 6.3.3 installed in a vSphere 5.5 environment requires a minimum version of ESXi 5.5 GA
  • NSX-v 6.3.3 installed in a vSphere 6.0 environment requires a minimum version of ESXi 6.0 Update 2
  • NSX-v 6.3.3 installed in a vSphere 6.5 environment requires a minimum version of ESXi 6.5a
If NSX 6.3.3 is installed on an earlier version of 5.5/6.0 ESXi, the netcpa service will fail to start preventing communication between ESXi hosts and the NSX Controllers.
In terms of upgrading from previous versions of NSX-v you can see that the upgrade path does have some stoppers. Below is the interoperability matrix that included vCloud Director 8.20 which, at the moment is not supported with NSX-v 6.3.3…I expect that to change over the next couple of weeks.
Upgrading to NSX-v 6.3.3:

 

As mentioned there are things to look out for during and after the upgrade from previous builds of NSX-v. There are detailed upgrade notes in the release notes so as always, make sure to read those as well, but below is a brief walk through of the upgrade process I conducted in one of my NestedESXi labs.
Once the NSX Manager has been upgraded you should have the following in your Summary tab:
Once the NSX Manager has been upgraded you should restart the vCenter Web Client to ensure any lingering parts of the previous version are removed. Login to the Web Client and click through to Networking & Security -> Installation and then the Management Tab where you will see Upgrade Available.
IMPORTANT NOTE: The upgrade notes state that you need to have a minimum of three NSX Controllers which I’d say is linked to the fact that the underlying OS of the Controllers has been shifted to Photon OS. This is likely to impact anyone running NSX in a NestedESXi or homelab as generally, only one was deployed to preserve resources. Once you click on upgrade you will get a special upgrade warning before committing to the upgrade as shown below:
  • The NSX Controller cluster must contain three controller nodes to upgrade to NSX 6.3.3. If it has fewer than three controllers, you must add controllers before starting the upgrade
  • When you upgrade to NSX-v 6.3.3, instead of an in-place software upgrade, the existing controllers are deleted one at a time, and new Photon OS based controllers are deployed using the same IP addresses

There is also a slight increase to the size of the storage for the controllers from 20GB to 28GB. Once upgraded the NSX Controllers will be at version 6.3.6235594.

The last major step is to upgrade the Host components from the Host Preparation tab. On vSphere 6.0 and above once you have upgraded to NSX 6.3.x, all future NSX VIB changes do not trigger a reboot…only maintenance mode is required to complete the VIB change. In NSX 6.3.3 there is a change to the NSX VIB names on ESXi 6.0 and later where the esx-vxlan and esx-vsip VIBs have been merged and replaced with esx-nsxv as shown below.

VIB names on ESXi 5.5 remain the same.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_633.html

https://kb.vmware.com/kb/2151267

 

vCloud Director SP 8.20 – NSX Advanced Networking Overview

Many, including myself thought that the day would never come where we would be talking about a new UI for vCloud Director…but a a month on from the 8.20 release of vCloud Director SP (which was the 8th major release of vCD) I’m happy to be writing about the new Advanced Networking features of 8.20 based on NSX-v. Full NSX compatibility and interoperability has been a long time coming, however the wait has been worthwhile as the vCloud Director team opted to fully integrate the network management into the vCD Cloud Cells over the initial approach that had a seperate appliance acting as a proxy between the NSX Manager and vCD Cells.

But before I dive into the new HTML5 goodness, I thought it would be good to recap the Advanced Networking Services of vCD and how we got to where we are today…

No More vShield…Sort Of:

As everyone should know by now, the vCloud Networking & Security was made end of life late last year and from the release of vCD SP 8.10 vShield Edges should have been upgraded to their NSX equivalents. These Edges will remain as basic Edges within vCloud Director and even though at the backend they would be on NSX-v versioning, no extra features or functionality beyond what was available in the existing vCD portal would be available to tenants.

  • DHCP
  • NAT
  • Firewall
  • Static Routing
  • IPSec VPN
  • Basic Load Balancer

The version of NSX-v deployed dictates the build number of the NSX Edge, however as can be seen below it’s still listed as a vShield Edge in vCenter.

As anyone who has worked closely would know, NSX-v has a lot of vShield DNA in it and in truth it’s more vShield than NSX when talking about the features that pertain to vCloud Director. However the power of NSX-v can be taken advantage of once an basic edge is upgraded to an Advanced Edge.

Advanced Edge Services:

Before the major UI additions that came with vCD SP 8.20 the previous 8.10 version did give us a taste of what was to come with the introduction of a new menu option when you right clicked on an Edge Gateway.

This option was greyed out unless you where running the initial beta of the Advanced Networking Services or ANS. The option can be executed by anyone with the rights to upgrade the edge gateway, but by default this can only be done by a System Administrator or the Org Admin. So it’s worthwhile double checking the roles you have allocated to your tenant’s to ensure that these upgrades can be controlled.

Once you click on the Convert to Advanced Gateway option you get a warning referring to a VMwareKB that warns you about an API change that may make previous calling methods obsolete. Something to take note of for anyone automating this process. On execution of this conversion there is no physical change to the Virtual Machine, however if you now click on the Edge Gateway Services option of the Edge Gateway you will be taken to the new HTML5 Web Interface for NSX Advanced Networking Services to access all the advanced features:

  • Firewall
  • DHCP
  • NAT
  • Routing (Dynamic)
  • Load Balancer (Advanced)
  • SSL VPN Plus
  • Certificates
  • Grouping Objects
  • Statistics
  • Edge Settings

All new Advanced Networking features are configured from the new HTML5 web interface which retains the base vCD URL but now adds:

/tenant/network-edges/{ID}?org=ORGNAME

Everything is self contained the tenant doesn’t have to authenticate again to get to the new user interface. However, if you just upgrade the Edge and go to configure the Advanced Network Services out of the box you will only see a couple of the items listed above.

In order to use the new features a System Administrator must use the vCloud API to grant the new rights that the organisation requires. This process has been explained very well by my good friend Giuliano Bertello here. This process uses the vCloud API to Grant Distributed Firewall and Advanced Networking Services Rights to roles in vCloud Director 8.20 using the new granular role based access control mechanisms that where introduced in 8.20. Once configured your tenant’s can now see all the services listed above to configure the Edge Gateway.

Organisational Distributed Firewall:

Something that is very much new in the 8.20 release is the ability to take advantage of mircosegmentation using the NSX-v Distributed Firewall service. The ability to configure organisation wide rules logically, without the need for a virtual Edge Gateway is a significant step forward for vCD tenants and I hope that this feature enhancement is exposed by service providers and it’s value sold to their tenants. To access the Distributed Firewall, in the Virtual Datacenters windows of the Administration tab, right click on the Virtual Datacenter name and select Manage Firewall.

Once again you will be taken to the new HTML5 user interface and once the correct permissions have been applied to the user you can enable the Distributed Firewall and start configuring your rules. The URL is slightly different to the Edge Gateway URL:

/tenant/dwf/{ID}?org=ORGNAME

But the look and feel is familiar.

Conclusion:

vCloud Director SP 8.20 has finally delivered on the what most members of the vCloud Air Network had wanted for some time…that is, full NSX interoperability and feature set access as well as a new user interface. Over the next few weeks, I am going to expand on all the features of the Advanced and Distributed Networking features of vCD and NSX and walk through how to configure elements through the UI and API as well as give a looks into what’s happening at the backend in terms of how NSX stores rules and policy items for vCD tenant use.

Compatibility with vSphere 6.5 and NSX-v 6.3.x:

vCloud Director SP 8.20 is compatible with vSphere 6.5 and NSX 6.3.0 and supports full interoperability with other versions as shown in the VMware Product Interoperability Matrix. As of vCD 8.20 GA, vCD 8.20 passed the functional interoperability test and limited scale testing for these versions:

  • vCD 8.20 with vSphere 6.0 and NSX 6.3.0
  • vCD 8.20 with vSphere 6.5 and NSX 6.3.0

References:

https://kb.vmware.com/kb/2149042
https://kb.vmware.com/kb/2147625

« Older Entries