Tag Archives: NSX-V

NSX Bytes: NSX-v 6.3 Host Preparation Fails with Agent VIB module not installed

NSX-v 6.3 was released last week with an impressive list of new enhancements and I wasted no time in looking to upgrades my NestedESXi lab instance from 6.2.5 to 6.3 however I ran into an issue that at first I thought was related to a previous VIB upgrade issue caused by VMware Update Manager not being available during NSX Host upgrades…in this case it presented with the same error message in the vCenter Events view:

VIB module for agent is not installed on host <hostname> (_VCNS_xxx_Cluster_VMware Network Fabri)

After ensuring that my Update Manager was in a good state I was left scratching my head…that was until some back and forth in the vExpert Slack #NSX channel relating to a new VMwareKB that was released the same day as NSX-v 6.3.

https://kb.vmware.com/kb/2053782

This issue occurs if vSphere Update Manager (VUM) is unavailable. EAM depends on VUM to approve the installation or uninstallation of VIBs to and from the ESXi host.

Even though my Upgrade Manager was available I was not able to upgrade through Host Preparation. It seem’s like vSphere 6.x instances might be impacted by this bug but the good news is there is a relatively easy workaround as mentioned in the VMwareKB that bypasses the VUM install mechanism. To enable the workaround you need to enter into the Managed Object Browser of the vCenter EAM by going to the following URL and entering in vCenter admin credentials.

https://vCenter_Server_IP/eam/mob/ 

Once logged in you are presented with a (or list of) agencies. In my case I had more than one, but I selected the first one in the list which was agency-11

The value that needs to be changed is the bypassVumEnabled boolean value as shown below.

To set that flag to True enter in the following URL:

https://vCenter_Server_IP/eam/mob/?moid=agency-x&method=Update

Making sure that the agency number matches your vCenter EAM instance. From there you need to change the existing configuration for that value by removing all the text in the value box and invoking the value listed below:

Once invoked you should be able to go back into the Web Client and click on Resolve under the Cluster name in the Host Preparation Tab of the NSX Installation window.

Once done I was in an all Green state and all hosts where upgraded to 6.3.0.5007049. Once all hosts have been upgraded it might be a useful idea to reverse the workaround and wait for an official fix from VMware.

References:

https://kb.vmware.com/kb/2053782

NSX Bytes: NSX for vSphere 6.3 and NSX-T 1.1 Release Information

VMware’s NSX has been in the wild for almost three years and while the initial adoption was slow, of recent times there has been a calculated push to make NSX more mainstream. The change in licensing that happened last year has not only been done to help drive adoption by traditional VMware customers running vSphere that previously couldn’t look at NSX due to price but also the Transformers project has looked to build on Nicira’s roots in the heterogeneous hypervisor market and offer network virutalization beyond vSphere and beyond Open source platforms and into the public cloud space. The vision for VMware with NSX is to manage security and connectivity for heterogeneous end points through:

  • Security
  • Automation
  • Application Continuity

NSX has seen significant growth for VMware over the past twelve to eighteen months driven mostly from customer demand focusing around micro-segmentation, IT automation and efficiency and also the need to have extended multiple data centre locations that can be pooled together. To highlight the potential that remains with NSX-v less that 5% of the total available vSphere install base has NSX-v installed…and while that could have something to do with the initial restrictions and cost of the software it still represents enormous opportunity for VMware and their partners.

Last week the NSX vExpert group was given a first look at what’s coming in the new releases…below is a summation of what to expect from both NSX-v 6.3 and NSX-T 1.1. Note that we where not given an indication on vSphere 6.5 support so, like the rest of you we are all waiting for the offical release notes.

[Update] vSphere 6.5 will be supported with NSX-v 6.3

Please note that VMware vSphere 6.5a is the minimum supported version with NSX for vSphere 6.3.0. For the most up-to-date information, see the VMware Product Interoperability Matrix. Also, see 2148841.

NSX for vSphere 6.3 Enhancements:

Security:

  • NSX Pre-Assessment Tool based on vRealize Network Insight
  • Micro-Segmentation Planning and application visibility
  • New Security Certifications around ICSA, FIPS, Common Criteria and STIG
  • Linux Guest VM Introspection
  • Increase performance in service chaining
  • Larger scalability of VDI up to 50K desktops
  • NSX IDFW for VDI
  • Active Directory Integration for VDI at scale

Automation:

  • Routing Enhancements
  • Centralized Dashboard for service and ops
  • Reduced Upgrade windows with rebootless upgrades
  • Integration with vRA 7.2 enhancing LB,NAT
  • vCloud Director 8.20 support with advanced routing, DFW, VPN
  • VIO Updates to include multi-vc deployments
  • vSphere Integrated Container Support
  • New Automation Frameworks for PowerNSX, PyNSXv, vRO

Application Continuity:

  • Multi-DC deployments with Cross VC NSX enhancements for security tags
  • Operations enhancements with improved availability
  • L2VPN performance enhancements for cross DC/Cloud Connectivity

Where does NSX-T Fit:

Given there was some confusion about NSX-v vs. NSX-t in terms of everything going to a common code base starting from the transformers release it was highlighted that VMware’s primary focus for 2017 hasn’t shifted away from NSX for vSphere and will still be heavily invested in to add new capabilities in and beyond 6.3 and that there will be a robust roadmap of new capabilities in future releases with support extended will into the future.

NSX-t’s main drivers related to new data centre and cloud architectures with more hetrogeneality driving a different set of requirements to that of vSphere that focuses around multi-domain environments leading to a multi-hypervisor NSX platform. NSX-t is highly extensible and will address more endpoint heterogeneity in future releases including containers, public clouds and other hypervisors. As you can see before the existing use cases for NSX-t are mainly focused around devops, micro-segmentation and multi-tenant infrastructure.

NSX-T 1.1 Brief Overview:

Again the focus is around private IaaS and multi-hypervisor support for development teams using dev clouds and employing more devops methodologies. There isn’t too much to write home about in the 1.1.0 release but there is some extended hypervisor support for KVM and ESXi, more single or multi-tenant support and some performance and resiliency optimizations.

Conclusion:

There is a lot to like about where VMware is taking NSX and both product streams offer strong network virtualization capabilities for customers to take advantage of. There is no doubt in my mind that the release of NSX-v 6.3 will continue to build on the great foundation laid by the previous NSX versions. When the release notes are made available I will do take a deeper look into all the new features and enhancements and tie them into what’s most useful for service providers.

NSX Bytes: Important Bug in 6.2.4 to be Aware of

[UPDATE] In light of this post being quoted on The Register I wanted to clarify a couple of things. First off, as mentioned there is a fix for this issue (the KB should be rewritten to clearly state that) and secondly, if you read below, you will see that I did not state that just about anyone running NSX-v 6.2.4 will be impacted. Greenfield deployments are not impacted.

Here we go again…I thought maybe we where over these, but it looks like NSX-v 6.2.4 contains a fairly serious bug impacting VMs after vMotion operations. I had intended to write about this earlier in the week when I first became aware of the issue, however the last couple of days have gotten away from me. That said, please be aware of this issue as it will impact those who have upgraded NSX-v from 6.1.x to 6.2.4.

As the KB states, the issue appears if you have the Distributed Firewall enabled (it’s enabled and inline by default) and you have upgraded NSX-v from 6.1.x to 6.2.3 and above, though for most this should be applicable to 6.2.4 upgrades due to all this issues in 6.2.3. If VM’s are migrated between upgraded hosts they will loose network connectivity and require a reboot to bring back connectivity.

If you check the vmkernal.log file you will see similar entries to that below.

Cause

This issue occurs when the VSIP module at the kernel level does not handle the export_version deployed in NSX for vSphere 6.1.x correctly during the upgrade process.

The is no current resolution to the issue apart from the VM reboot but there is a workaround in the form of a script that can be obtained via GSS if you reference KB2146171. Hopefully there will be a proper fix in future NSX releases.

<RANT>

I can’t believe something as serious as this was missed by QA for what is VMware’s flagship product. It’s beyond me that this sort of error wasn’t picked up in testing before it was released. It’s simply not good enough that a major release goes out with this sort of bug and I don’t know how it keeps on happening. This one specifically impacted customers and for service providers or enterprises that upgraded in good faith, it puts egg of the faces of those who approve, update and execute the upgrades that results in unhappy customers or internal users.

Most organisations can’t fully replicate production situations when testing upgrades due to lack or resources or lack of real world situation testing…VMware could and should have the resources to stop these bugs leaking into release builds. For now, if possible I would suggest that people add more stringent vMotion tests as part of NSX-v lab testing before promoting into production moving forward.

VMware customers shouldn’t have to be the ones discovering these bugs!

</RANT>

[UPDATE] While I am obviously not happy about this issue coming in the wake of previous issues, I still believe in NSX and would recommend all shops looking to automate networking still have faith in what the platform offers. Bug’s will happen…I get that, but I know in the long run there is huge benefit in running NSX.

References:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146171

NSX Bytes: Updated – NSX Edge Feature and Performance Matrix

A question came up today around throughput numbers for an NSX Edge Services Gateway and that jogged my memory back to a previous blog post where I compared features and performance metrics between vShield Edges and NSX Edges. In the original post I had left out some key metrics, specifically around firewall and load balance throughput so thought it was time for an update. Thanks to a couple of people in the vExpert NSX Slack Channel I was able to fill some gaps and update the tables below.

A reminder that VMware has announced the End of Availability (“EOA”) of the VMware vCloud Networking and Security 5.5.x that kicked in on the September  of 19, 2016 and that vCloud Director 8.10 does not support vShield Edges anymore…hence why I have removed the VSE from the tables.

As a refresher…what is an Edge device?

The Edge Services Gateway (NSX-v) connects isolated, stub networks to shared (uplink) networks by providing common gateway services such as DHCP, VPN, NAT, dynamic routing, and Load Balancing. Common deployments of Edges include in the DMZ, VPN Extranets, and multi-tenant Cloud environments where the Edge creates virtual boundaries for each tenant.

Below is a list of services provided by the NSX Edge.

Service Description
Firewall Supported rules include IP 5-tuple configuration with IP and port ranges for stateful inspection for all protocols
NAT Separate controls for Source and Destination IP addresses, as well as port translation
DHCP Configuration of IP pools, gateways, DNS servers, and search domains
Site to Site VPN Uses standardized IPsec protocol settings to interoperate with all major VPN vendors
SSL VPN SSL VPN-Plus enables remote users to connect securely to private networks behind a NSX Edge gateway
Load Balancing Simple and dynamically configurable virtual IP addresses and server groups
High Availability High availability ensures an active NSX Edge on the network in case the primary NSX Edge virtual machine is unavailable
Syslog Syslog export for all services to remote servers
L2 VPN Provides the ability to stretch your L2 network.
Dynamic Routing Provides the necessary forwarding information between layer 2 broadcast domains, thereby allowing you to decrease layer 2 broadcast domains and improve network efficiency and scale. Provides North-South connectivity, thereby enabling tenants to access public networks.

Below is a table that shows the different sizes of each edge appliance and what (if any) impact that has to the performance of each service. As a disclaimer the below numbers have been cherry picked from different sources and are subject to change…I’ll keep them as up to date as possible.

NSX Edge (Compact) NSX Edge (Large) NSX Edge (Quad-Large) NSX Edge (X-Large)
vCPU 1 2 4 6
Memory 512MB 1GB 1GB 8GB
Disk 512MB 512MB 512MB 4.5GB
Interfaces 10 10 10 10
Sub Interfaces (Trunk) 200 200 200 200
NAT Rules 2000 2000 2000 2000
FW Rules 2000 2000 2000 2000
FW Performance 3Gbps 9.7Gbps 9.7Gbps 9.7Gbps
DHCP Pools 25 25 25 25
Static Routes 2048 2048 2048 2048
LB Pools 64 64 64 64
LB Virtual Servers 64 64 64 64
LB Server / Pool 32 32 32 32
IPSec Tunnels 512 1600 4096 6000
SSLVPN Tunnels 50 100 100 1000
Concurrent Sessions 64,000 1,000,000 1,000,000 1,000,000
Sessions/Second 8,000 50,000 50,000 50,000
LB Throughput L7 Proxy) 2.2Gbps 2.2Gbps 3Gbps
LB Throughput L4 Mode) 6Gbps 6Gbps 6Gbps
LB Connections/s (L7 Proxy) 46,000 50,000 50,000
LB Concurrent Connections (L7 Proxy) 8,000 60,000 60,000
LB Connections/s (L4 Mode) 50,000 50,000 50,000
LB Concurrent Connections (L4 Mode) 600,000 1,000,000 1,000,000
BGP Routes 20,000 50,000 250,000 250,000
BGP Neighbors 10 20 50 50
BGP Routes Redistributed No Limit No Limit No Limit No Limit
OSPF Routes 20,000 50,000 100,000 100,000
OSPF Adjacencies 10 20 40 40
OSPF Routes Redistributed 2000 5000 20,000 20,000
Total Routes 20,000 50,000 250,000 250,000

Of interest from the above table it doesn’t list any Load Balancing performance number for the NSX Compact Edge…take that to mean that if you want to do any sort of load balancing you will need NSX Large and above. To finish up, below is a table describing each NSX Edge size use case.

Use Case
NSX Edge (Compact) Small Deployment, POCs and single service use
NSX Edge (Large) Small/Medium DC or mult-tenant
NSX Edge (Quad-Large) High Throughput ECMP or High Performance Firewall
NSX Edge (X-Large) L7 Load Balancing, Dedicated Core

References:

https://www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf

https://pubs.vmware.com/NSX-6/index.jsp#com.vmware.nsx.admin.doc/GUID-3F96DECE-33FB-43EE-88D7-124A730830A4.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2042799

First Look: vRealize Network Insight (Arkin)

Last year Arkin burst onto the scene offering a solution that focused on virtual and physical deep network analytics. Arkin was recognised at VMworld 2015 by nearly taking out the best of show and fast forward twelve months, Arkin was acquired by VMware with the product later rebadged as vRealize Network Insight. One of the products main strengths that attracted VMware into making the acquisition was it’s tight integration into NSX by way of a simple and intuitive user interface that lets admins easily manage and troubleshoot NSX while offering best practice checks that can guide users through VXLAN and firewall implementations and alert them to any issues in their design and implementation of NSX.

Arkin removes barriers to SDDC adoption and operation by providing converged visibility, and contextual analytics across virtual and physical, an ability to implement newer security models such as micro-segmentation, and by ensuring application uptime, while letting IT collaborate better. The platform helps IT organizations plan, operate, visualize, analyze, and troubleshoot their complex software-defined data center environments.

As vRealize Network Insight the key benefits are:

  • East-west traffic analytics for security and micro-segmentation design
  • Control and tracking to meet audit and compliance requirements for virtual distributed firewalls
  • 360 Overlay-underlay visibility and topology mapping
  • Extensive 3rd party physical switch integrations
  • VXLAN to VLAN logical path mappings
  • Advanced NSX Operations Management
  • Natural language search and enhanced user experience for rapid troubleshooting

What I was surprised to find when I was able to dig into the product was that it offered more than just Network insights…in fact it offered surprisingly deep analytics and metrics for Hosts and Virtual Machines that rival most similar products out on the market today.

Installation Overview:

To install Network Insight you download two OVA’s from MyVMware and deploy the two appliances into vCenter. It’s got an interesting setup that’s shown below and after deployment you are left with two appliances, a Platform, and a Proxy that have the following specifications.

Platform OVA

  • 6 CPU cores (reservation 3072) Mhz
  • 32 GigaBytes RAM (reservation)
  • 600 Gigabytes HDD (thin provisioned)

Proxy OVA

  • 2 CPU cores (reservation 1024 Mhz)
  • 4 Gigabytes RAM (reservation 4GB)
  • 100 Gigabytes HDD (thin provisioned)

A note before continuing…only Chrome is supported as a browser at this stage.

You start the install by deploying the Platform appliance…once the Platform OVA is deployed and the appliance VM settings have been configured you can hit the IP specified in the OVA deployment process and continue the installation.

After the license key has been validated you are then asked to Generate a shared secret that is used to pair the Platform with the Proxy appliance.

From here you can initiate the deployment of the Proxy appliance. During the OVA deployment you are asked to enter in the shared key before continuing to configure the appliance networking and naming. As shown below, the configuration wizard waits to detect the deployed Proxy appliance at which point the installation is complete and you can login.

The default username name is [email protected] with a password of admin.

When you login for the first time you are presented with a Product Evaluation pop up letting you know you are in NSX Assessment Mode and that you can switch to Full Product Mode at the bottom right of the window. NSX Assessment Mode is an interesting feature that looks like it will be used to install Network Insight as part of an on boarding or discovery engagement and produce reports on what is happening inside an NSX environment.

In either mode you need to register at least one vCenter and, if in a site with NSX, register the NSX Manager as well. As mentioned in the opening you can also plug into a small subset of popular physical networking equipment such as Cisco, Arista, DELL, Brocade and Juniper.

Once the vCenter has been connected and verified you then have the option to select the vDS and PortGroups you want to have monitored. This enabled Netflow (IPFIX) across all PortGroups selected…it does these changes live so be wary of any possible breaks in vDS traffic flow just in case.

Due to a rather serious PSOD bug in previous version of ESXi when Netflow is enabled, the configurator blocks any host that doesn’t meet the minimum ESXi builds as shown below.

Below is the minimum requirements for Network Insight to be configured and start collecting and analyzing.

Infrastructure

  • vCenter 5.5 or above
  • ESXi 5,5, update 2 (build 2068190) and above
  • ESXi 6.0, update 1b (3380124) and above
  • NSX for vSphere 6.1 or greater
  • Netflow enabled on vDS

Reading through the FAQ, you get to learn about IPFIX and how it’s used with the vDS to collect network traffic data…it’s worth spending some time going through the FAQ however I’ve pulled an overview on how it all works below.

IPFIX is an IETF protocol for exporting flow information. A flow is defined as a set of packets transmitted in a specific timeslot, and sharing 5-tuple values – source IP address, source port, destination IP address, destination port, and protocol. The flow information may include properties such as timestamps, packets/bytes count, Input/output interfaces, TCP Flags, VXLAN Id, Encapsulated flow information and so on.

 

Network Insight uses VMware VDS IPFIX to collect network traffic data. Every session has two paths. For example: Session A↔C has A→C packets and C→A packets. To analyze the complete information of any session, IPFIX data about packets in both the directions is required. Refer following diagram where VM-A is connected to DVPG-A and is talking to VM-C. Here DVPG-A will only provide data about the C→A packets, and DVPG-Uplink will provide data about A→C packets. To get the complete information of A’s traffic, Ipfix should be enabled on DVPG-A, DVPG-uplink

That wraps up this post…I’ll be looking at doing a followup post that looks at the Network Insight user interface and what information about network traffic, flows and routing can be viewed and analysed as well as taking a look at the surprisingly good VM, Host and Cluster level metrics

References:

http://www.arkin.net/

https://www.vmware.com/products/vrealize-network-insight.html

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vmware-vrealize-network-insight-faq.pdf

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vmware-vrealize-network-insight-user-guide.pdf

NSX Bytes: vCloud Director Can’t Deploy NSX Edges

Over the weekend I was tasked with the recovery of a #NestedESXi lab that had vCloud Director and NSX-v components as part of the lab platform. Rather than being a straight forward restore from the Veeam backup I also needed to downgrade the NSX-v version from 6.2.4 to 6.1.4 for testing purposes. That process was relatively straight forward and involved essentially working backwards in terms of installing and configuring NSX and removing all the components from vCenter and the ESXi hosts.

To complete the NSX-v downgrade I deployed a new 6.1.4 appliance and connected it back up to vCenter, configured the hosts, setup VXLAN, transport components and tested NSX Edge deployments through the vCenter Web Client. However, when it came time to test Edge deployments from vCloud Director I kept on getting the following error shown below.

Checking through the NSX Manager logs there was no reference to any API call hitting the endpoint as is suggested by the error detail above. Moving over to the vCloud Director Cells I was able to trace the error message in the log folder…eventually seeing the error generated below in the vcloud-container-info.log file.

As a test I hit the API endpoint referenced in the error message from a browser and got the same result.

This got me thinking that the error was either DNS related or permission related. After confirming that the vCloud Cells where resolving the NSX Manager host name correctly, as suggested by the error I looked at permissions as the cause of the 403 error. vCloud Director was configured to use the service.vcloud service account to connect to the previous NSX/vShield Manager and it dawned on me that I hadn’t setup user rights in the Web Client under Networking & Security. Under the Users section of the Manage Tab the service account used by vCloud Director wasn’t configured and needed to be added. After adding the user I retried the vCD job and the Edge deployed successfully.

While I was in this menu I thought I’d test what level of NSX User was required to for that service account to have in order to execute operations against vCloud Director and NSX. As shown below anything but NSX or Enterprise Administrator triggered a “VSM response error (254). User is not authorized to access object” error.

At the very least to deploy edges, you require the service account to be NSX Administrator…The Auditor and Security Administrator levels are not enough to perform the operations required. More importantly don’t forget to add the service account as configured in vCloud Director to the NSX Manager instance otherwise you won’t be able to have vCloud Director deploy edges using NSX-v.

 

 

NSX Bytes: NSX-v 6.2.4 Released …Important Upgrade!

NSX-v 6.2.4 was released the week before VMworld US so might have gotten somewhat lost in the VMworld noise…For those that where fortunate enough to not upgrade to or deploy a greenfield 6.2.3 site you can now safely do so without the nasty bugs that existed in the 6.2.3 build. In a nutshell this new build delivers all the significant features and enhancements announced in 6.2.3 without the dFW or Edge Gateway bugs that forced the build being pulled from distribution a few weeks back.

In terms of how and when to upgrade from previous versions the following table gives a great overview of the pathways required to get to 6.2.4.

The take away from the table above is that if possible you need to get onto NSX-v 6.2.4 as soon as possible and with good reason:

  • VMware NSX 6.2.4 provides critical bug fixes identified in NSX 6.2.3, and 6.2.4 delivers a security patch for CVE-2016-2079 which is a critical input validation vulnerability for sites that uses NSX SSL VPN.
  • For customers who use SSL VPN, VMware strongly recommends a review of CVE-2016-2079 and an upgrade to NSX 6.2.4.
  • For customers who have installed NSX 6.2.3 or 6.2.3a, VMware recommends installing NSX 6.2.4 to address critical bug fixes.

Prior to this release if you had upgraded to NSX-v 6.1.7 you where stuck and not able to upgrade to 6.2.3. The Upgrade matrix is now reporting that you can upgrade 6.1.7 to 6.2.4 as shown below.

I was able to validate this in my lab going from 6.1.7 to 6.2.4 without any issues.

NSX-v 6.1.4 is also fully supported by vCloud Director SP 8.0.1 and 8.10

References:

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.4/releasenotes_nsx_vsphere_624.html

http://www.theregister.co.uk/2016/07/22/please_dont_upgrade_nsx_just_now_says_vmware/

NSX Bytes: 6.1.x General Support Extended and 6.2.3 Edge Upgrade Issues

A while ago VMware announced that NSX-v general support would come to an end on this October to pave the way for current 6.1.x users to upgrade to 6.2.x. A problem has arisen in that people who patched NSX-v to the latest patch release 6.1.7 to cover a security venerability are left being unable to upgrade to 6.2.3 which also covers the same venerability in the 6.2.x release.

NSX Bytes: Critical Update for NSX-v and vCNS

As of June 9, 2016 with the release of NSX for vSphere 6.1.7, the EOGS date has been extended by 3 months, to January 15th, 2017. This is to allow customers to have time to upgrade from NSX for vSphere 6.1.7,  which contains an important security patch improving input validation of the system, to the latest 6.2.x release. For recommended upgrade paths, refer to the latest NSX for vSphere 6.2
.
It’s not the first time that current releases of NSX-v have blocked upgrades to future releases, and in this case NSX-v 6.2.3 also includes this security patch and along with 6.2.2, remains the suggested release for NSX-v. Repeating that upgrades from NSX 6.1.7 to 6.2.3 are not supported. Once VMware release the patch version beyond 6.1.7 upgrading to 6.2.x will be possible. That said it’s great of VMware to extend the end of support by three months to give themselves time to get the patch out.
.
6.2.3 ESG Catch-22:

For those than can upgrade to NSX-v 6.2.3 there is a current issue around the upgrading of NSX and existing edges possibly becoming unmanageable. This issue occurs when the load balancer is configured for serverSsl or clientSsl but ciphers value is set as NULL in the previous version. NSX-v 6.2.3 introduces a new approved cipher list in NSX Manager and does not allow the ciphers to be NULL when configuring the load balancer…as was the previous default option.

Since the ciphers value defaults to NULL in the earlier version, if this is not set NSX Manager 6.2.3 considers this ciphers value as invalid the Edges in turn become unmanageable. There should be a fix coming and there is a workaround as described in the VMwareKB here.

 

References:

Released: NSX 6.2.3 – Packed Full Of New Features!

Last week VMware released NSX-v 6.2.3 Build 3979471 and it’s anything but your standard point release. Running through the list off the release notes this could have easily been a major dot release. In good news for vCloud Air Network Service Providers there have been some major enhancements to the Edge Services Gateways which adds availability and protocol enhancements as well as added general stability through bug fixes and security updates.

There has also been additional management and monitoring built into the Web Client and other UI enhancements. The new licensing features as previously discussed in this post have come into effect as of this build so you will now see the license type and number of licenses used for VXLAN and DFW in the Web Client under NSX Managers -> Summary

As this is a big release I am going to filter through the release notes and pick the best features and fixes as it pertains to Service Providers and highlight the ones that I feel improve the ability to SPs to deliver strong networking services based on NSX-v as part of their service offerings.

Web Client Additions:

As mentioned above there have been a few UI enhancements in the 6.2.3 release including a new NSX Dashboard (shown below) that provides visibility into the overall health of NSX components in one view, Traceflow Enhancement for Network Introspection Services and the Firewall rules UI now displays configured IP protocols and TCP/UDP port numbers associated with services.

Going through the upgrade from previous NSX versions I noticed a few other UI additions. Once the Controllers are upgraded you can now see Disk Latency of each controller disk. The Controllers are extremely disk sensitive so it’s good to see this worked into the UI.

In addition to that new installations of NSX 6.2.3 will deploy NSX Controllers with updated disk partitions to provide extra cluster resiliency. Previously log overflow on the controller disk might impact controller stability. If you upgrade to NSX 6.2.3 the Controller will retain their original disk layout.

I also noticed a Channel Health option in the Host Preparation Tab that shows the status of the NSX Host agents and there are some other UI additions letting you modify the UUID of the NSX Instance and modify the VXLAN Port which can be done under Logical Network Preperation -> VXLAN Transport.

NSX Edge Service Gateway Changes:

As mentioned there have been a number of enhancements to the NSX ESGs which have further added to the maturity of the Edge appliance and makes it even more attractive for use with vCloud Director offering Hybrid Networking solutions…or just as a web frontend for key internet services. IS-IS has also been removed as a routing protocol option under dynamic routing as support has been pulled. TLS 1.0 has been depreciated and there have been some Cipher support changes for the IPSec, SSLVPN and L2VPN.

  • New Edge DHCP Options: DHCP Option 121 supports static route option, which is used for DHCP server to publish static routes to DHCP client; DHCP Options 66, 67, 150 supports DHCP options for PXE Boot; and DHCP Option 26 supports configuration of DHCP client network interface MTU by DHCP server.
  • Increase in DHCP Pool, static binding limits: The following are the new limit numbers for various form factors: Compact: 2048; Large: 4096; Quad large: 4096; and X-large: 8192.
  • Edge Firewall adds SYN flood protection: Avoid service disruptions by enabling SYN flood protection for transit traffic. Feature is disabled by default, use the NSX REST API to enable it.
  • NSX Edge — Resource Reservation: Reserves CPU/Memory for NSX Edge during creation. Admin user can modify the CPU/Memory settings after NSX Edge deployment using REST API to configure VM appliances.
  • Change in NSX Edge Upgrade Behavior: Replacement NSX Edge VMs are deployed before upgrade or redeploy. The host must have sufficient resources for four NSX Edge VMs during the upgrade or redeploy of an Edge HA pair. Default value for TCP connection timeout is changed to 21600 seconds from the previous value of 3600 seconds.
  • Flexible SNAT / DNAT rule creation: vnicId no longer needed as an input parameter; removed requirement that the DNAT address must be the address of an NSX Edge VNIC.
  • Maximum number of NAT rules: For NSX Edge versions prior to 6.2, a user could configure 2048 SNAT and 2048 DNAT rules separately, giving a total limit of 4096 rules. Since NSX Edge version 6.2 onwards, a limit is enforced for the maximum allowed NAT rules, based on the NSX Edge appliance size: 1024 SNAT and 1024 DNAT rules for a total limit of 2048 rules for COMPACT edge. 2048 SNAT and 2048 DNAT for a total limit of 4096 rules for LARGE edge and QUADLARGE edge. 4096 SNAT and 4096 DNAT rules for a total limit of 8192 rules for XLARGE edge.
  • Logging is now enabled by default for SSL VPN and L2 VPN. The default log level is notice.
  • NSX Edge technical support logs have been enhanced to report memory consumption per process.

Other Key Features and Additions:

  • NSX Hardware Layer 2 Gateway Integration: expands physical connectivity options by integrating 3rd-party hardware gateway switches into the NSX logical network
  • New VXLAN Port 4789 in NSX 6.2.3 and later: Before version 6.2.3, the default VXLAN UDP port number was 8472. See the NSX Upgrade Guide for details.
  • Firewall — Granular Rule Filtering: simplifies troubleshooting by providing granular rule filters in UI, based on Source, Destination, Action, Enabled/Disabled, Logging, Name, Comments, Rule ID, Tag, Service, Protocol.
  • Guest Introspection — Windows 10 support
  • SSL VPN ClientMac OS El Capitan support
  • Service Composer — Performance Improvements: enables faster startup/reboot of NSX Manager by optimizing synchronization between security policy and firewall service, and disabling auto-save of firewall drafts by default
  • VMware vRealize Log Insight 3.3.2 for NSX provides intelligent log analytics for NSX, This version accepts NSX Standard/Advanced/Enterprise edition license keys issued for NSX 6.2.2+

Upgrade Notes – RTFM:

In the release notes there is a detailed section on the upgrade and interoprability of this version of NSX with other key VMware components. It’s important that it’s read so as to not have a poor experience during the upgrade.

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html#upgradenotes

Resolved Issues:

There are a large number of Resolved Issues which can be found on the release notes…below are the ones that relating to Service Providers running Edge Services Gateways.

  • Extended HA failover times for Edge Services Gateway (ESG) or DLR with Edge VM when using only static routes
  • NAT does not translate IP addresses when NSX Edge firewall is disabled
  • vCenter 6.0 restart/reboot may result in duplicate VTEPs on VXLAN prepared ESX hosts
  • After upgrading the NSX Edge from 6.1.x to 6.2.x, the NSX Manager vsm.log shows “INVALID DHCP CONFIG”
  • Unexpected TCP interruption on TCP sessions during Edge High Availability (HA) failover in NSX 6.2.x

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html#resolvedissues

NSX Design Guide v3:

https://communities.vmware.com/servlet/JiveServlet/previewBody/27683-102-8-41631/NSX%20Reference%20Design%20Version%203.0.pdf

Overall a huge release for NSX-v. If you have the right entitlements you can login to MyVMware and download the binaries.

References:

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html

NSX Bytes: Critical Update for NSX-v and vCNS

I generally don’t post around security releases but after going through the notes on CVE-2016-2079 I thought it was important enough to dedicate a post around. Mainly because it could impact those running NSX Edge Services Gateways or vShield Edges with the SSL-VPN service enabled for clients.

Most vCloud Director based instances won’t have the SSL-VPN enabled due to it not being exposed through the vCD UI however some Service Providers may offer this as a managed service as it’s one of the strongest features of the Edge Gateways. The issue detailed in the CVE is summarized below.

VMware NSX and vCNS with SSL-VPN enabled contain a critical input validation vulnerability. This issue may allow a remote attacker to gain access to sensitive information.

In a nutshell you need to upgrade an existing version of NSX-v or vCNS to the version below. As per usual if you have the entitlements go ahead and download the updates from the links below.

  • NSX Edge: 6.2 -> 6.2.3
  • NSX Edge: 6.1 -> 6.1.7
  • vCNS Edge: 5.5 -> 5.5.4.3

NSX-v  Downloads: https://www.vmware.com/go/download-nsx-vsphere

vCNS Downloads: https://www.vmware.com/go/download-vcd-ns

References:

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-2079

« Older Entries