Tag Archives: NSX-V

NSX Bytes: Important Bug in 6.2.4 to be Aware of

[UPDATE] In light of this post being quoted on The Register I wanted to clarify a couple of things. First off, as mentioned there is a fix for this issue (the KB should be rewritten to clearly state that) and secondly, if you read below, you will see that I did not state that just about anyone running NSX-v 6.2.4 will be impacted. Greenfield deployments are not impacted.

Here we go again…I thought maybe we where over these, but it looks like NSX-v 6.2.4 contains a fairly serious bug impacting VMs after vMotion operations. I had intended to write about this earlier in the week when I first became aware of the issue, however the last couple of days have gotten away from me. That said, please be aware of this issue as it will impact those who have upgraded NSX-v from 6.1.x to 6.2.4.

As the KB states, the issue appears if you have the Distributed Firewall enabled (it’s enabled and inline by default) and you have upgraded NSX-v from 6.1.x to 6.2.3 and above, though for most this should be applicable to 6.2.4 upgrades due to all this issues in 6.2.3. If VM’s are migrated between upgraded hosts they will loose network connectivity and require a reboot to bring back connectivity.

If you check the vmkernal.log file you will see similar entries to that below.

Cause

This issue occurs when the VSIP module at the kernel level does not handle the export_version deployed in NSX for vSphere 6.1.x correctly during the upgrade process.

The is no current resolution to the issue apart from the VM reboot but there is a workaround in the form of a script that can be obtained via GSS if you reference KB2146171. Hopefully there will be a proper fix in future NSX releases.

<RANT>

I can’t believe something as serious as this was missed by QA for what is VMware’s flagship product. It’s beyond me that this sort of error wasn’t picked up in testing before it was released. It’s simply not good enough that a major release goes out with this sort of bug and I don’t know how it keeps on happening. This one specifically impacted customers and for service providers or enterprises that upgraded in good faith, it puts egg of the faces of those who approve, update and execute the upgrades that results in unhappy customers or internal users.

Most organisations can’t fully replicate production situations when testing upgrades due to lack or resources or lack of real world situation testing…VMware could and should have the resources to stop these bugs leaking into release builds. For now, if possible I would suggest that people add more stringent vMotion tests as part of NSX-v lab testing before promoting into production moving forward.

VMware customers shouldn’t have to be the ones discovering these bugs!

</RANT>

[UPDATE] While I am obviously not happy about this issue coming in the wake of previous issues, I still believe in NSX and would recommend all shops looking to automate networking still have faith in what the platform offers. Bug’s will happen…I get that, but I know in the long run there is huge benefit in running NSX.

References:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146171

NSX Bytes: Updated – NSX Edge Feature and Performance Matrix

A question came up today around throughput numbers for an NSX Edge Services Gateway and that jogged my memory back to a previous blog post where I compared features and performance metrics between vShield Edges and NSX Edges. In the original post I had left out some key metrics, specifically around firewall and load balance throughput so thought it was time for an update. Thanks to a couple of people in the vExpert NSX Slack Channel I was able to fill some gaps and update the tables below.

A reminder that VMware has announced the End of Availability (“EOA”) of the VMware vCloud Networking and Security 5.5.x that kicked in on the September  of 19, 2016 and that vCloud Director 8.10 does not support vShield Edges anymore…hence why I have removed the VSE from the tables.

As a refresher…what is an Edge device?

The Edge Services Gateway (NSX-v) connects isolated, stub networks to shared (uplink) networks by providing common gateway services such as DHCP, VPN, NAT, dynamic routing, and Load Balancing. Common deployments of Edges include in the DMZ, VPN Extranets, and multi-tenant Cloud environments where the Edge creates virtual boundaries for each tenant.

Below is a list of services provided by the NSX Edge.

Service Description
Firewall Supported rules include IP 5-tuple configuration with IP and port ranges for stateful inspection for all protocols
NAT Separate controls for Source and Destination IP addresses, as well as port translation
DHCP Configuration of IP pools, gateways, DNS servers, and search domains
Site to Site VPN Uses standardized IPsec protocol settings to interoperate with all major VPN vendors
SSL VPN SSL VPN-Plus enables remote users to connect securely to private networks behind a NSX Edge gateway
Load Balancing Simple and dynamically configurable virtual IP addresses and server groups
High Availability High availability ensures an active NSX Edge on the network in case the primary NSX Edge virtual machine is unavailable
Syslog Syslog export for all services to remote servers
L2 VPN Provides the ability to stretch your L2 network.
Dynamic Routing Provides the necessary forwarding information between layer 2 broadcast domains, thereby allowing you to decrease layer 2 broadcast domains and improve network efficiency and scale. Provides North-South connectivity, thereby enabling tenants to access public networks.

Below is a table that shows the different sizes of each edge appliance and what (if any) impact that has to the performance of each service. As a disclaimer the below numbers have been cherry picked from different sources and are subject to change…I’ll keep them as up to date as possible.

NSX Edge (Compact) NSX Edge (Large) NSX Edge (Quad-Large) NSX Edge (X-Large)
vCPU 1 2 4 6
Memory 512MB 1GB 1GB 8GB
Disk 512MB 512MB 512MB 4.5GB
Interfaces 10 10 10 10
Sub Interfaces (Trunk) 200 200 200 200
NAT Rules 2000 2000 2000 2000
FW Rules 2000 2000 2000 2000
FW Performance 3Gbps 9.7Gbps 9.7Gbps 9.7Gbps
DHCP Pools 25 25 25 25
Static Routes 2048 2048 2048 2048
LB Pools 64 64 64 64
LB Virtual Servers 64 64 64 64
LB Server / Pool 32 32 32 32
IPSec Tunnels 512 1600 4096 6000
SSLVPN Tunnels 50 100 100 1000
Concurrent Sessions 64,000 1,000,000 1,000,000 1,000,000
Sessions/Second 8,000 50,000 50,000 50,000
LB Throughput L7 Proxy) 2.2Gbps 2.2Gbps 3Gbps
LB Throughput L4 Mode) 6Gbps 6Gbps 6Gbps
LB Connections/s (L7 Proxy) 46,000 50,000 50,000
LB Concurrent Connections (L7 Proxy) 8,000 60,000 60,000
LB Connections/s (L4 Mode) 50,000 50,000 50,000
LB Concurrent Connections (L4 Mode) 600,000 1,000,000 1,000,000
BGP Routes 20,000 50,000 250,000 250,000
BGP Neighbors 10 20 50 50
BGP Routes Redistributed No Limit No Limit No Limit No Limit
OSPF Routes 20,000 50,000 100,000 100,000
OSPF Adjacencies 10 20 40 40
OSPF Routes Redistributed 2000 5000 20,000 20,000
Total Routes 20,000 50,000 250,000 250,000

Of interest from the above table it doesn’t list any Load Balancing performance number for the NSX Compact Edge…take that to mean that if you want to do any sort of load balancing you will need NSX Large and above. To finish up, below is a table describing each NSX Edge size use case.

Use Case
NSX Edge (Compact) Small Deployment, POCs and single service use
NSX Edge (Large) Small/Medium DC or mult-tenant
NSX Edge (Quad-Large) High Throughput ECMP or High Performance Firewall
NSX Edge (X-Large) L7 Load Balancing, Dedicated Core

References:

https://www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf

https://pubs.vmware.com/NSX-6/index.jsp#com.vmware.nsx.admin.doc/GUID-3F96DECE-33FB-43EE-88D7-124A730830A4.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2042799

First Look: vRealize Network Insight (Arkin)

Last year Arkin burst onto the scene offering a solution that focused on virtual and physical deep network analytics. Arkin was recognised at VMworld 2015 by nearly taking out the best of show and fast forward twelve months, Arkin was acquired by VMware with the product later rebadged as vRealize Network Insight. One of the products main strengths that attracted VMware into making the acquisition was it’s tight integration into NSX by way of a simple and intuitive user interface that lets admins easily manage and troubleshoot NSX while offering best practice checks that can guide users through VXLAN and firewall implementations and alert them to any issues in their design and implementation of NSX.

Arkin removes barriers to SDDC adoption and operation by providing converged visibility, and contextual analytics across virtual and physical, an ability to implement newer security models such as micro-segmentation, and by ensuring application uptime, while letting IT collaborate better. The platform helps IT organizations plan, operate, visualize, analyze, and troubleshoot their complex software-defined data center environments.

As vRealize Network Insight the key benefits are:

  • East-west traffic analytics for security and micro-segmentation design
  • Control and tracking to meet audit and compliance requirements for virtual distributed firewalls
  • 360 Overlay-underlay visibility and topology mapping
  • Extensive 3rd party physical switch integrations
  • VXLAN to VLAN logical path mappings
  • Advanced NSX Operations Management
  • Natural language search and enhanced user experience for rapid troubleshooting

What I was surprised to find when I was able to dig into the product was that it offered more than just Network insights…in fact it offered surprisingly deep analytics and metrics for Hosts and Virtual Machines that rival most similar products out on the market today.

Installation Overview:

To install Network Insight you download two OVA’s from MyVMware and deploy the two appliances into vCenter. It’s got an interesting setup that’s shown below and after deployment you are left with two appliances, a Platform, and a Proxy that have the following specifications.

Platform OVA

  • 6 CPU cores (reservation 3072) Mhz
  • 32 GigaBytes RAM (reservation)
  • 600 Gigabytes HDD (thin provisioned)

Proxy OVA

  • 2 CPU cores (reservation 1024 Mhz)
  • 4 Gigabytes RAM (reservation 4GB)
  • 100 Gigabytes HDD (thin provisioned)

A note before continuing…only Chrome is supported as a browser at this stage.

You start the install by deploying the Platform appliance…once the Platform OVA is deployed and the appliance VM settings have been configured you can hit the IP specified in the OVA deployment process and continue the installation.

After the license key has been validated you are then asked to Generate a shared secret that is used to pair the Platform with the Proxy appliance.

From here you can initiate the deployment of the Proxy appliance. During the OVA deployment you are asked to enter in the shared key before continuing to configure the appliance networking and naming. As shown below, the configuration wizard waits to detect the deployed Proxy appliance at which point the installation is complete and you can login.

The default username name is [email protected] with a password of admin.

When you login for the first time you are presented with a Product Evaluation pop up letting you know you are in NSX Assessment Mode and that you can switch to Full Product Mode at the bottom right of the window. NSX Assessment Mode is an interesting feature that looks like it will be used to install Network Insight as part of an on boarding or discovery engagement and produce reports on what is happening inside an NSX environment.

In either mode you need to register at least one vCenter and, if in a site with NSX, register the NSX Manager as well. As mentioned in the opening you can also plug into a small subset of popular physical networking equipment such as Cisco, Arista, DELL, Brocade and Juniper.

Once the vCenter has been connected and verified you then have the option to select the vDS and PortGroups you want to have monitored. This enabled Netflow (IPFIX) across all PortGroups selected…it does these changes live so be wary of any possible breaks in vDS traffic flow just in case.

Due to a rather serious PSOD bug in previous version of ESXi when Netflow is enabled, the configurator blocks any host that doesn’t meet the minimum ESXi builds as shown below.

Below is the minimum requirements for Network Insight to be configured and start collecting and analyzing.

Infrastructure

  • vCenter 5.5 or above
  • ESXi 5,5, update 2 (build 2068190) and above
  • ESXi 6.0, update 1b (3380124) and above
  • NSX for vSphere 6.1 or greater
  • Netflow enabled on vDS

Reading through the FAQ, you get to learn about IPFIX and how it’s used with the vDS to collect network traffic data…it’s worth spending some time going through the FAQ however I’ve pulled an overview on how it all works below.

IPFIX is an IETF protocol for exporting flow information. A flow is defined as a set of packets transmitted in a specific timeslot, and sharing 5-tuple values – source IP address, source port, destination IP address, destination port, and protocol. The flow information may include properties such as timestamps, packets/bytes count, Input/output interfaces, TCP Flags, VXLAN Id, Encapsulated flow information and so on.

 

Network Insight uses VMware VDS IPFIX to collect network traffic data. Every session has two paths. For example: Session A↔C has A→C packets and C→A packets. To analyze the complete information of any session, IPFIX data about packets in both the directions is required. Refer following diagram where VM-A is connected to DVPG-A and is talking to VM-C. Here DVPG-A will only provide data about the C→A packets, and DVPG-Uplink will provide data about A→C packets. To get the complete information of A’s traffic, Ipfix should be enabled on DVPG-A, DVPG-uplink

That wraps up this post…I’ll be looking at doing a followup post that looks at the Network Insight user interface and what information about network traffic, flows and routing can be viewed and analysed as well as taking a look at the surprisingly good VM, Host and Cluster level metrics

References:

http://www.arkin.net/

https://www.vmware.com/products/vrealize-network-insight.html

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vmware-vrealize-network-insight-faq.pdf

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vmware-vrealize-network-insight-user-guide.pdf

NSX Bytes: vCloud Director Can’t Deploy NSX Edges

Over the weekend I was tasked with the recovery of a #NestedESXi lab that had vCloud Director and NSX-v components as part of the lab platform. Rather than being a straight forward restore from the Veeam backup I also needed to downgrade the NSX-v version from 6.2.4 to 6.1.4 for testing purposes. That process was relatively straight forward and involved essentially working backwards in terms of installing and configuring NSX and removing all the components from vCenter and the ESXi hosts.

To complete the NSX-v downgrade I deployed a new 6.1.4 appliance and connected it back up to vCenter, configured the hosts, setup VXLAN, transport components and tested NSX Edge deployments through the vCenter Web Client. However, when it came time to test Edge deployments from vCloud Director I kept on getting the following error shown below.

Checking through the NSX Manager logs there was no reference to any API call hitting the endpoint as is suggested by the error detail above. Moving over to the vCloud Director Cells I was able to trace the error message in the log folder…eventually seeing the error generated below in the vcloud-container-info.log file.

As a test I hit the API endpoint referenced in the error message from a browser and got the same result.

This got me thinking that the error was either DNS related or permission related. After confirming that the vCloud Cells where resolving the NSX Manager host name correctly, as suggested by the error I looked at permissions as the cause of the 403 error. vCloud Director was configured to use the service.vcloud service account to connect to the previous NSX/vShield Manager and it dawned on me that I hadn’t setup user rights in the Web Client under Networking & Security. Under the Users section of the Manage Tab the service account used by vCloud Director wasn’t configured and needed to be added. After adding the user I retried the vCD job and the Edge deployed successfully.

While I was in this menu I thought I’d test what level of NSX User was required to for that service account to have in order to execute operations against vCloud Director and NSX. As shown below anything but NSX or Enterprise Administrator triggered a “VSM response error (254). User is not authorized to access object” error.

At the very least to deploy edges, you require the service account to be NSX Administrator…The Auditor and Security Administrator levels are not enough to perform the operations required. More importantly don’t forget to add the service account as configured in vCloud Director to the NSX Manager instance otherwise you won’t be able to have vCloud Director deploy edges using NSX-v.

 

 

NSX Bytes: NSX-v 6.2.4 Released …Important Upgrade!

NSX-v 6.2.4 was released the week before VMworld US so might have gotten somewhat lost in the VMworld noise…For those that where fortunate enough to not upgrade to or deploy a greenfield 6.2.3 site you can now safely do so without the nasty bugs that existed in the 6.2.3 build. In a nutshell this new build delivers all the significant features and enhancements announced in 6.2.3 without the dFW or Edge Gateway bugs that forced the build being pulled from distribution a few weeks back.

In terms of how and when to upgrade from previous versions the following table gives a great overview of the pathways required to get to 6.2.4.

The take away from the table above is that if possible you need to get onto NSX-v 6.2.4 as soon as possible and with good reason:

  • VMware NSX 6.2.4 provides critical bug fixes identified in NSX 6.2.3, and 6.2.4 delivers a security patch for CVE-2016-2079 which is a critical input validation vulnerability for sites that uses NSX SSL VPN.
  • For customers who use SSL VPN, VMware strongly recommends a review of CVE-2016-2079 and an upgrade to NSX 6.2.4.
  • For customers who have installed NSX 6.2.3 or 6.2.3a, VMware recommends installing NSX 6.2.4 to address critical bug fixes.

Prior to this release if you had upgraded to NSX-v 6.1.7 you where stuck and not able to upgrade to 6.2.3. The Upgrade matrix is now reporting that you can upgrade 6.1.7 to 6.2.4 as shown below.

I was able to validate this in my lab going from 6.1.7 to 6.2.4 without any issues.

NSX-v 6.1.4 is also fully supported by vCloud Director SP 8.0.1 and 8.10

References:

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.4/releasenotes_nsx_vsphere_624.html

http://www.theregister.co.uk/2016/07/22/please_dont_upgrade_nsx_just_now_says_vmware/

NSX Bytes: 6.1.x General Support Extended and 6.2.3 Edge Upgrade Issues

A while ago VMware announced that NSX-v general support would come to an end on this October to pave the way for current 6.1.x users to upgrade to 6.2.x. A problem has arisen in that people who patched NSX-v to the latest patch release 6.1.7 to cover a security venerability are left being unable to upgrade to 6.2.3 which also covers the same venerability in the 6.2.x release.

NSX Bytes: Critical Update for NSX-v and vCNS

As of June 9, 2016 with the release of NSX for vSphere 6.1.7, the EOGS date has been extended by 3 months, to January 15th, 2017. This is to allow customers to have time to upgrade from NSX for vSphere 6.1.7,  which contains an important security patch improving input validation of the system, to the latest 6.2.x release. For recommended upgrade paths, refer to the latest NSX for vSphere 6.2
.
It’s not the first time that current releases of NSX-v have blocked upgrades to future releases, and in this case NSX-v 6.2.3 also includes this security patch and along with 6.2.2, remains the suggested release for NSX-v. Repeating that upgrades from NSX 6.1.7 to 6.2.3 are not supported. Once VMware release the patch version beyond 6.1.7 upgrading to 6.2.x will be possible. That said it’s great of VMware to extend the end of support by three months to give themselves time to get the patch out.
.
6.2.3 ESG Catch-22:

For those than can upgrade to NSX-v 6.2.3 there is a current issue around the upgrading of NSX and existing edges possibly becoming unmanageable. This issue occurs when the load balancer is configured for serverSsl or clientSsl but ciphers value is set as NULL in the previous version. NSX-v 6.2.3 introduces a new approved cipher list in NSX Manager and does not allow the ciphers to be NULL when configuring the load balancer…as was the previous default option.

Since the ciphers value defaults to NULL in the earlier version, if this is not set NSX Manager 6.2.3 considers this ciphers value as invalid the Edges in turn become unmanageable. There should be a fix coming and there is a workaround as described in the VMwareKB here.

 

References:

Released: NSX 6.2.3 – Packed Full Of New Features!

Last week VMware released NSX-v 6.2.3 Build 3979471 and it’s anything but your standard point release. Running through the list off the release notes this could have easily been a major dot release. In good news for vCloud Air Network Service Providers there have been some major enhancements to the Edge Services Gateways which adds availability and protocol enhancements as well as added general stability through bug fixes and security updates.

There has also been additional management and monitoring built into the Web Client and other UI enhancements. The new licensing features as previously discussed in this post have come into effect as of this build so you will now see the license type and number of licenses used for VXLAN and DFW in the Web Client under NSX Managers -> Summary

As this is a big release I am going to filter through the release notes and pick the best features and fixes as it pertains to Service Providers and highlight the ones that I feel improve the ability to SPs to deliver strong networking services based on NSX-v as part of their service offerings.

Web Client Additions:

As mentioned above there have been a few UI enhancements in the 6.2.3 release including a new NSX Dashboard (shown below) that provides visibility into the overall health of NSX components in one view, Traceflow Enhancement for Network Introspection Services and the Firewall rules UI now displays configured IP protocols and TCP/UDP port numbers associated with services.

Going through the upgrade from previous NSX versions I noticed a few other UI additions. Once the Controllers are upgraded you can now see Disk Latency of each controller disk. The Controllers are extremely disk sensitive so it’s good to see this worked into the UI.

In addition to that new installations of NSX 6.2.3 will deploy NSX Controllers with updated disk partitions to provide extra cluster resiliency. Previously log overflow on the controller disk might impact controller stability. If you upgrade to NSX 6.2.3 the Controller will retain their original disk layout.

I also noticed a Channel Health option in the Host Preparation Tab that shows the status of the NSX Host agents and there are some other UI additions letting you modify the UUID of the NSX Instance and modify the VXLAN Port which can be done under Logical Network Preperation -> VXLAN Transport.

NSX Edge Service Gateway Changes:

As mentioned there have been a number of enhancements to the NSX ESGs which have further added to the maturity of the Edge appliance and makes it even more attractive for use with vCloud Director offering Hybrid Networking solutions…or just as a web frontend for key internet services. IS-IS has also been removed as a routing protocol option under dynamic routing as support has been pulled. TLS 1.0 has been depreciated and there have been some Cipher support changes for the IPSec, SSLVPN and L2VPN.

  • New Edge DHCP Options: DHCP Option 121 supports static route option, which is used for DHCP server to publish static routes to DHCP client; DHCP Options 66, 67, 150 supports DHCP options for PXE Boot; and DHCP Option 26 supports configuration of DHCP client network interface MTU by DHCP server.
  • Increase in DHCP Pool, static binding limits: The following are the new limit numbers for various form factors: Compact: 2048; Large: 4096; Quad large: 4096; and X-large: 8192.
  • Edge Firewall adds SYN flood protection: Avoid service disruptions by enabling SYN flood protection for transit traffic. Feature is disabled by default, use the NSX REST API to enable it.
  • NSX Edge — Resource Reservation: Reserves CPU/Memory for NSX Edge during creation. Admin user can modify the CPU/Memory settings after NSX Edge deployment using REST API to configure VM appliances.
  • Change in NSX Edge Upgrade Behavior: Replacement NSX Edge VMs are deployed before upgrade or redeploy. The host must have sufficient resources for four NSX Edge VMs during the upgrade or redeploy of an Edge HA pair. Default value for TCP connection timeout is changed to 21600 seconds from the previous value of 3600 seconds.
  • Flexible SNAT / DNAT rule creation: vnicId no longer needed as an input parameter; removed requirement that the DNAT address must be the address of an NSX Edge VNIC.
  • Maximum number of NAT rules: For NSX Edge versions prior to 6.2, a user could configure 2048 SNAT and 2048 DNAT rules separately, giving a total limit of 4096 rules. Since NSX Edge version 6.2 onwards, a limit is enforced for the maximum allowed NAT rules, based on the NSX Edge appliance size: 1024 SNAT and 1024 DNAT rules for a total limit of 2048 rules for COMPACT edge. 2048 SNAT and 2048 DNAT for a total limit of 4096 rules for LARGE edge and QUADLARGE edge. 4096 SNAT and 4096 DNAT rules for a total limit of 8192 rules for XLARGE edge.
  • Logging is now enabled by default for SSL VPN and L2 VPN. The default log level is notice.
  • NSX Edge technical support logs have been enhanced to report memory consumption per process.

Other Key Features and Additions:

  • NSX Hardware Layer 2 Gateway Integration: expands physical connectivity options by integrating 3rd-party hardware gateway switches into the NSX logical network
  • New VXLAN Port 4789 in NSX 6.2.3 and later: Before version 6.2.3, the default VXLAN UDP port number was 8472. See the NSX Upgrade Guide for details.
  • Firewall — Granular Rule Filtering: simplifies troubleshooting by providing granular rule filters in UI, based on Source, Destination, Action, Enabled/Disabled, Logging, Name, Comments, Rule ID, Tag, Service, Protocol.
  • Guest Introspection — Windows 10 support
  • SSL VPN ClientMac OS El Capitan support
  • Service Composer — Performance Improvements: enables faster startup/reboot of NSX Manager by optimizing synchronization between security policy and firewall service, and disabling auto-save of firewall drafts by default
  • VMware vRealize Log Insight 3.3.2 for NSX provides intelligent log analytics for NSX, This version accepts NSX Standard/Advanced/Enterprise edition license keys issued for NSX 6.2.2+

Upgrade Notes – RTFM:

In the release notes there is a detailed section on the upgrade and interoprability of this version of NSX with other key VMware components. It’s important that it’s read so as to not have a poor experience during the upgrade.

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html#upgradenotes

Resolved Issues:

There are a large number of Resolved Issues which can be found on the release notes…below are the ones that relating to Service Providers running Edge Services Gateways.

  • Extended HA failover times for Edge Services Gateway (ESG) or DLR with Edge VM when using only static routes
  • NAT does not translate IP addresses when NSX Edge firewall is disabled
  • vCenter 6.0 restart/reboot may result in duplicate VTEPs on VXLAN prepared ESX hosts
  • After upgrading the NSX Edge from 6.1.x to 6.2.x, the NSX Manager vsm.log shows “INVALID DHCP CONFIG”
  • Unexpected TCP interruption on TCP sessions during Edge High Availability (HA) failover in NSX 6.2.x

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html#resolvedissues

NSX Design Guide v3:

https://communities.vmware.com/servlet/JiveServlet/previewBody/27683-102-8-41631/NSX%20Reference%20Design%20Version%203.0.pdf

Overall a huge release for NSX-v. If you have the right entitlements you can login to MyVMware and download the binaries.

References:

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html

NSX Bytes: Critical Update for NSX-v and vCNS

I generally don’t post around security releases but after going through the notes on CVE-2016-2079 I thought it was important enough to dedicate a post around. Mainly because it could impact those running NSX Edge Services Gateways or vShield Edges with the SSL-VPN service enabled for clients.

Most vCloud Director based instances won’t have the SSL-VPN enabled due to it not being exposed through the vCD UI however some Service Providers may offer this as a managed service as it’s one of the strongest features of the Edge Gateways. The issue detailed in the CVE is summarized below.

VMware NSX and vCNS with SSL-VPN enabled contain a critical input validation vulnerability. This issue may allow a remote attacker to gain access to sensitive information.

In a nutshell you need to upgrade an existing version of NSX-v or vCNS to the version below. As per usual if you have the entitlements go ahead and download the updates from the links below.

  • NSX Edge: 6.2 -> 6.2.3
  • NSX Edge: 6.1 -> 6.1.7
  • vCNS Edge: 5.5 -> 5.5.4.3

NSX-v  Downloads: https://www.vmware.com/go/download-nsx-vsphere

vCNS Downloads: https://www.vmware.com/go/download-vcd-ns

References:

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-2079

NSX Bytes: Friends Don’t Let Friends Delete The VTEP PortGroup

Last week I posted a tweet saying “Friends don’t let friends delete the NSX-v VTEP PortGroup” and as most of us do in our industry we learn by doing and I found out the hard way that you shouldn’t mess with the PortGroup created during the Host Preparation of the NSX setup and configuration stage. This PortGroup is used by the Hosts in an NSX Enabled Cluster for the VMKernel Interfaces that are the VTEPs or VXLAN Tunnel End Points.

In a production environment this action is actually near on impossible to do because you can’t delete a PortGroup when it’s in use. Where I found myself in this situation was in trying to clone off a lab environment and restore components of the existing lab into new lab with new hosts. With that the following is something that could be handy in lab environments.

Once the new hosts have been prepared I went to configure the VXLAN against the cluster which creates a new VMKernel Interface on each host and assigns it a VTEP address from DHCP or from a pre-configured IP Pool but got an error. When I looked at the event logs in vCenter I saw the following error.

DVPortGroup dvportgroup-148806 couldnot be found
 The object or item referred to could not be found

Instantly I remembered that I had “cleaned up” the cloned vCenter configuration and removed any surplus PortGroups…in doing so I deleted the PortGroup NSX was referencing. I tried to recreate the PortGroup with the same name but it was clear that the configuration was referencing the MOID of the PortGroup and asking vCenter to use that to complete the job. Even an export/import of the Distributed Switch configuration from the original vCenter didn’t do the trick as the import increments the MOID already contained in the vCenter Database.

GSS Support Fix:

Thinking back to previous NSX related cases I’ve raised with VMware support I knew that the NSX Manager Database kept a very simple structure of vCenter objects and I guessed that some backend SQL search and replace could do the trick. After raising a case I had the guys in GSS enter into the NSX Manager backend, that can only be access with a secret VMware password and search for the table that referenced the MOID of the PortGroup. As can be seen below the fix is simple if you know the MOID of the old and the new PortGroup.

Note: Only VMware Support can action this fix.

With that modification committed I was able configure the VTEPs for the new hosts and continue to rebuild up the cloned instance. So if you ever get yourself in a situation where you have managed to do as I have done…there is a fix that can be done to avoid a complete start from scratch scenario.

New NSX License Tier Thoughts and Transformers

Overnight there was a couple of NSX related events that took place…one was expected and one not so much. I have known about the next release of NSX code named Transformers going back to VMworld last year where I attended a couple of NDA sessions…What I wasn’t expecting was the announcement of separate licensing tiers for NSX-V which is detailed in this KB and this landing page.

Transformers – NSX-MH:

Before going into my thoughts around the features options that come with each licensing tier the release of NSX Transformers 1.0 represents a fundamental shift in the way that NSX is released in that it now is the one code base that unifies NSX-MH and NSX-V. For the moment though the 1.0 release which was available earlier in the day is only for current MH customers and we probably shouldn’t expect any release for V customers for a while…and when that drops it will be interesting to see the upgrade path and product interoprabilites.

NSX-V Edition Thoughts:

Going back to the new pricing its clear to me that this is aimed at trying to keep the momentum going for the NSBU and try to entice the market to a lower entry point, however going through the edition feature matrix I’m left a little confused at some of the choices especially for those who are looking to replace vCloud Networking and Security editions that’s set to be end of lifed later this year. (click here for details on that)

  • NSX Standard Edition – For organizations needing agility and automation of the network.
  • NSX Advanced Edition – For organizations needing Standard, plus a fundamentally more secure data center with micro-segmentation.
  • NSX Enterprise Edition – For organizations needing Advanced, plus networking and security across multiple domains.

VMware have increased the cost of the full featured Enterprise edition which existing customers have while creating the Standard and Advanced editions. Where previously the list (USD) for NSX was $5,996 per Socket the new editions come in at $1,995, $4,995 and $6,995 per Socket. The Standard edition is well priced but taking a look through the Matrix in the official KB you are getting an extremely slimmed down version of NSX…short of the bells and whistles that make it the awesome SDN platform that it is, however I’m sure the feature set will be attractive for some.

For existing customers, as stated in the FAQ

Customers with active support contracts who have purchased NSX prior to the new licensing model goes into effect in May, 2016 will be entitled to the same functionality in the Enterprise offering.

vCNS Edition Match:

Looking at the current functionality of vCNS and then trying to match it to the available functions in each edition as per the KB there doesn’t seem to me to be a perfect fit except for the Enterprise Edition…which isn’t what those looking to upgrade from vCNS want to hear. The new NSX-V Standard Edition doesn’t include VPN, SSLVPN or Load Balancing functionality, though it does include the third party integration which I suspect would represent a large part of the current vCNS users that are not Service Providers.

Feature Standard Advanced Enterprise
Hypervisors Supported      
ESXi 5.5 Yes Yes Yes
ESXi 6.0 Yes Yes Yes
vCenter 5.5 Yes Yes Yes
vCenter 6.0 Yes Yes Yes
Switching Encapsulation Format      
VXLAN Yes Yes Yes
Replication Mode for VXLAN      
Multicast Yes Yes Yes
Edge Routing (N-S)      
Edge Routing Static – IPv4 Yes Yes Yes
Edge Routing Static – IPv6 Yes Yes Yes
DHCP Relay Yes Yes Yes
VLAN Trunk (sub-interface) support Yes Yes Yes
NAT Support for NSX Edge      
NAT Support for NSX Edge Yes Yes Yes
Source NAT Yes Yes Yes
Destination NAT Yes Yes Yes
DDI      
DHCP Server Yes Yes Yes
DHCP Relay Yes Yes Yes
DNS Relay Yes Yes Yes
VPN      
IPSEC VPN No No Yes
SSL VPN No No Yes
Edge Firewall      
Edge Firewall Yes Yes Yes
Edge High-Availability Yes Yes Yes
Third Party Integration      
Endpoint Service Insertion – Guest Introspection Yes Yes Yes 
Public API based Integration Yes Yes  Yes
Edge Load-Balancing      
TCP (L4 – L7) No Yes Yes
HTTP No Yes Yes
HTTPS (Pass-through) No Yes Yes
LB Methods No Yes Yes
Round Robin No Yes Yes
Src IP Hash No Yes Yes
Least Connection No Yes Yes
Health Checks
TCP No Yes Yes
HTTP No Yes Yes
HTTPS No Yes Yes

What about the vCAN and Service Provider Pricing?:

At this point in time I haven’t heard through any official channels of changes to the NSX Bundle options that come with the vCAN Program and I don’t really expect any changes to what currently exists for SP consumption of NSX as anything but the full Enterprise feature list would be required for SPs to take full advantage of NSX.

Final Thoughts:

Time will tell if these changes lead to greater uptake of non Enterprise or Service provider customers. One thing is forsure…VMware are without question needing NSX to be a success and I’ve got no doubt that with what currently exists in addition to what’s coming in future releases NSX will continue to gain market penetration and be the defacto SDN platform going round.

References:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2145269

http://www.crn.com/news/data-center/300080572/vmware-debuts-cheaper-nsx-software-defined-networking-options-hikes-pricing-for-premium-version.htm?itc=refresh

http://www.vmware.com/files/pdf/products/nsx/vmware-nsx-editions-faq.pdf