Category Archives: VMware

Quick Fix: VCSA 503 Service Unavailable Error

I’ve just had to fix one of my VCSA’s again from the infamous 503 Service Unavailable error that seems to be fairly common with the VCSA even though it’s was claimed to be fixed in vCenter version 6.5d. I’ve had this error pop up fairly regularly since deploying my homelab’s vCenter Server Appliance as a version 6.5 GA instance and for the most part I’ve refrained from rebooting the VCSA just in case the error pops up upon reboot and have even kept a snapshot against the VM just in case I needed to revert to it on the high change that it would error out.

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000559b1531ef80] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

After doing a Google search for any permanent solutions to the issue, I came across a couple of posts referencing USB passthrough devices that could trigger the error which was plausible given I was using an external USB Hard Drive. IP changes seem to also be a trigger for the error though in my case, it wasn’t the cause. There is a good Reddit thread here that talks about duplicate keys…again related to USB passthrough. It also links externally to some other solutions that where not relevant to my VCSA.

Solution:

As referenced in this VMware communities forum post, to fix the issue I had to first find out if I did have a duplicate key error in the VCSA logs. To do that I dropped into the VCSA shell and went into /var/logs and did a search for any file containing device_key + already exists. As shown in the image above this returned a number of entries confirming that I had duplicate keys and that it was causing the issue.

The VMware vCenter Server Appliance vpxd 6.5 logs are located in the /var/log/vmware/vmware-vpx folder

What was required next was to delete the duplicate embedded PostGres database table entries. To connect to the embedded postgres database you need to run the following command from the VCSA shell:

To remove the duplicate key I ran the following command and rebooted the appliance, noting that the id and device_key will vary.

Once everything rebooted all the services started up and I had a functional vCenter again which was a relief given I was about five minutes away from a restore or a complete rebuild…and ain’t nobody got time for that!

vCenter (VCSA) 6.5 broken after restart from vmware

Reference:

https://communities.vmware.com/thread/556490

 

VMware Flings: Top 5 – 2017 Edition

VMware has had their Lab Flings program going for a number of years now and in 2015 I wrote this post listing out my Top 5 Flings. Since then there have been some awesome Flings released and I thought it was a good time to update my Top 5 Flings to reflect the continued awesomeness generated within the VMware Labs. Since my last post there have also been a number of flings that have found their way into product releases:

Flings are apps and tools built by our engineers that are intended to be played with and explored.

There are 128 (up from 57 from August of 2015) Flings listed on the site though some have been depreciated. They range across most of VMware’s Product stack…most of them have been created out of some requirement or function that was/is lacking in the current toolset for their respective products. Most of them solve usability issues or look to resolve performance bottlenecks and look to optimize product experience.

Fling Number 5 – Storage Profile Updater

This Fling is a simple tool that enables the migration of vCloud Director virtual machines and templates from the default any storage profile to a specific storage profile. The tool can be run from the command-line with the help of a configuration file, and it allows you to change storage profiles in a batch style of processing.

For those that upgraded vCloud Director from 1.5 to 5.x you would know about the Any profile issue…this fling allows you to migrate all VMs from that default storage policy to any new one you might have configured in your Provider vDC.

Fling Number 4 – Cross vCenter VM Mobility – CLI

Cross vCenter VM Mobility – CLI is a command line interface (CLI) tool that can be used to migrate or clone a VM from one host to another host managed by a linked or isolated vCenter (VC) instance. It has been built using vSphere Java-based SDK APIs.

Currently, as of vSphere 6.0, the vSphere HTML5 Web Client allows users to perform Cross-VC operations like migration and cloning if two VCs are linked. If VCs are not linked, users cannot view the infrastructure across multiple VCs and thus, cannot utilize this functionality through UI. This Fling provides a way for users to access this vSphere feature through simple CLI commands. It also supports cross-cluster placement and shared storage vMotion between two VCs.

Cross vCenter migrations is probably one of the most underrated features VMware has released and has been present since vSphere 6.0. Originally exposed via the API’s William Lam blogged about a wrapper he wrote to use the functionality and this Fling sits beside that as possible tools to perform the cross vCenter actions.

Fling Number 3 – Embedded Host Client

The ESXi Embedded Host Client is a native HTML and JavaScript application and is served directly from your ESXi host! It should perform much better than any of the existing solutions

This Fling was a revelation when it was first released and adds a very usable and functional HTML5 web interface from which to manage your ESXi hosts. It’s now productized and packaged into ESXi 5.5, 6.0 and 6.5 and there is continues on going development of the tool along with bug fixes and features that can be installed via the vib on the Fling site.

Fling Number 2 – VMware Tools for Nested ESXi

This VIB package provides a VMware Tools service (vmtoolsd) for running inside a nested ESXi virtual machine. The following capabilities are exposed through VMware Tools:

Provides guest OS information of the nested ESXi Hypervisor (eg. IP address, configured hostname, etc.).
Allows the nested ESXi VM to be cleanly shut down or restarted when performing power operations with the vSphere Web/C# Client or vSphere APIs.
Executes scripts that help automate ESXi guest OS operations when the guest’s power state changes.
Supports the Guest Operations API (formally known as the VIX API).

The release of this Fling was met with a lot of thankyou’s from those who had battled with NestedESXi Hosts not having VMTools available. If anything, the ability to cleanly shutdown or restart the ESXi Guest was welcomed. With the release of ESXi 6.0 (and subsequent 6.5 release) the Tools are included in the OS by default…but for those running 5.x Nested Hosts its a must have.

Fling Number 1 – ESXi Mac Learning dvFilter v2.0

MAC learning functionality solves performance problems for use cases like nested ESX.  This ESX extension adds functionality to ESX to support MAC-learning on vswitch ports. For most ESX use cases, MAC learning is not required as ESX knows exactly which MAC address will be used by a VM. However, for applications like running nested ESX, i.e. ESX as a guest-VM on ESX, the situation is different. As an ESX VM may emit packets for a multitude of different MAC addresses, it currently requires the vswitch port to be put in “promiscuous mode”. That however will lead to too many packets delivered into the ESX VM, as it leads to all packets on the vswitch being seen by all ESX VMs. When running several ESX VMs, this can lead to very significant CPU overhead and noticeable degradation in network throughput. Combining MAC learning with “promiscuous mode” solves this problem. The MAC learning functionality is delivered as a high speed VMkernel extension that can be enabled on a per-port basis. It works on legacy standard switches as well as Virtual Distributed Switches

This Fling is close to my heart as I learnt at VMworld 2014 that it was born out of a blog post I did on Promiscuous Mode that triggered William Lam to approach Christian Dickmann with the issues and look for a way to solve the issue. As you can see from my followup post it works as designed and is the single must have Fling for those who run Nested ESXi labs. It was recently upgraded to version 2.0 to support ESXi 6.5.

As of last week there a new ESXi Learnswitch Fling was released which builds upon (but can’t be used with) the MAC Learning fling.

ESXi Learnswitch is a complete implementation of MAC Learning and Filtering and is designed as a wrapper around the host virtual switch. It supports learning multiple source MAC addresses on virtual network interface cards (vNIC) and filters packets from egressing the wrong port based on destination MAC lookup. This substantially improves overall network throughput and system performance for nested ESX and container use cases.

To learn more, read ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter.

For a full list of the Flings available for download, head to this link

https://labs.vmware.com/flings/?utf8=%E2%9C%93&order=date+DESC

vSAN 6.6 – What’s In It For Service Providers

Last February when VMware released VSAN 6.2 I stated that “Things had gotten Interesting” with regards to the 6.2 release of vSAN finally marking it’s arrival as a serious player in the Hyper-converged Infrastructure (HCI) market. vSAN was ready to be taken very seriously by VMware’s competitors. Fast forward fourteen months and apart from the fact we have confirmed the v in vSAN is a lower case with the product name officially changing from Virtual SAN to vSAN…Version 6.6 was announced last week is set to GA today, and with it comes the biggest list of new features and enhancements in vSANs history.

VMware has decided to break with the normal vSphere release cycle for vSAN and move to patch releases for vSphere that are actually major updates of vSAN. This is why this release is labeled vSAN 6.6 and will be included in the vSphere 6.5EP2 build. The move allows the vSAN team to continue to enhance the platform outside of the core vSphere platform and I believe it will deliver at least 2 update releases per year.

Looking at the new features and enhancements of the vSAN 6.6 release it’s clear to see that the platform has matured and given the 7000+ strong customer base it’s also clear to see that its being accepted more and more for critical workloads. From a service provider point of view I know of a lot more vCloud Air Network partners that have implemented vSAN as not only their Management HCI platform, but also now their customer HCI compute and storage  platforms.

A lot for Service Providers to like:

As shown in the feature timeline above there are over 20+ new features and enhancements but for me the following ones are most relative to vCAN Service Providers who are using, or looking to use vSAN in their offerings. I will expand on the ones in red as I see them as being the most significant of the new features and enhancements for service providers.

  • Native encryption for data-at-rest
  • Compliance certifications
  • vSAN Proactive Drive HA for failing drives
  • Resilient management independent of vCenter
  • Rapid recovery with smart, efficient rebuilds
  • Certified file service & data protection solutions
  • Enhanced vSAN SDK and PowerCLI
  • Simple networking with Unicast
  • vSAN Cloud Analytics for performance
  • vSAN Cloud Analytics with real-time support notification and recommendations*
  • vSAN Config Assist with 1-click hardware lifecycle management
  • Extended Health Services
  • Up to 50% greater IOPS for all-flash with optimized checksum and dedupe
  • Optimized for latest flash technologies
  • Expanded caching tier choice
  • New Docker Volume Driver

Simple networking with Unicast:

As John Nicholson wrote on the Virtual Blocks blog…it’s time to say goodbye to the multicast requirements around vSAN networking traffic. For a history as to why multicast was used, click here. Also it’s worth reading John’s post and also the he goes through the upgrade process as if you are upgrading from previous versions, multicast will still be used unless you make the change as also specified here.

I can attest first hand to the added complexity when it comes to setting up vSAN with multicast and have gone through a couple of painful deployments where the multicast configuration was an issue during initial setup and also caused issue with switching infrastructure that needed to be upgraded to before vSAN could work reliably. In my mind unicast offers a simpler less complex solution with minimal overheads and makes it more transportable across networks.

Performance Improvements:

Service Providers are always trying to squeeze the most out of their hardware purchases and with VMware claiming 50% greater IOPS for all-flash through optimized data services that in theory can enable 150K IOPS per host it appears they will be served well. in addition to optimized checksum and dedupe along with support for the latest flash technologies. The increased performance helps accelerate tenant workloads and provides higher consolidation ratios for those workloads.

Service providers can accelerate new hardware technologies with the support of the latest flash technologies, including solutions like the new breed of NVMe SSDs. These solutions can deliver up to 250% greater performance for write-intensive applications. vSAN 6.6 now offers larger caching drive options that includes 1.6TB flash drives, so that service providers can take advantage of larger capacity flash drives.

Disk Performance Enhancements:

For those that have gone through a vSAN rebuild operation you would know that is can be a long exercise depending on the amount of data and configuration of the vSAN datastore. vSAN 6.6 introduces a new smart rebuild and rebalancing feature along with partial repairs of degraded or absent components. There is also resync throttling and improved visibility into the rebuilding status through the Health Status. Cormac Hogan goes through the improvements in detail here.

From a Service Provider point of view having these enhanced features around the rebuilds it critical to continued quality of service for IaaS customer who live on shared vSAN storage. Shorter and more efficient rebuild times means less impact to customers.

Health Checks and Monitoring Improvements:

vSAN Encryption:

VMware has introduced VM encryption native at the vSAN datastore level. This can be enabled per vSAN Cluster and works with deduplication and compression across hybrid and all-flash cluster configurations. vSAN 6.6 data Encryption is hardware agnostic, there is no requirement to use specialized and more expensive Self-Encrypting Drives (SEDs) which is also a bonus. Jase McCarty has another Virtual Blocks article here that goes through this feature in great detail.

From a Service Provider point of view you can now potentially offer two classes of vSAN backed storage for IaaS customers. One that lives on an Encrypted enabled cluster that’s charged at a premium over non Encrypted clusters. In talking with service providers across the globe, data at rest encryption has become something that potential customers are asking for and most leading storage companies have an encryption story…now so does vSAN and it appears to be market leading.

vSAN 6.6 Licensing:

In terms of the licensing Matrix, nothing too drastic has changed except for the addition of Data at Rest Encryption in the Enterprise bundle, however in a significant move for vCAN Service Providers, QoS IOPS Limiting has been extended across all license types and can now be taken advantage across the board. This is good for Service Providers who look to offer different tiers or storage performance based on IOPS limited…previously it was only available under Enterprise licensing.

Bootstrapping UI:

As a bonus feature that I think will assist vCAN Service Providers is the new Native Bootstrap installer in vSAN 6.6. William Lam has written about the feature here, but for those looking to install their first vSAN node without vSphere available the ability to bootstrap is invaluable. The old manual process is still worth looking at as it’s always beneficial to know what’s going on in the background, but it’s all GUI based now via the VCSA installer.

Conclusion:

vSAN 6.6 appears to be a great step forward for VMware and Service Providers will no doubt be keen to upgrade as soon as possible to take advantage of the features and enhancements that have been delivered in this 6.6 release.

References:

http://cormachogan.com/2017/04/11/whats-new-vsan-6-6/ 

https://storagehub.vmware.com/#!/vmware-vsan/vmware-vsan-6-5-technical-overview

http://vsphere-land.com/news/an-overview-of-whats-new-in-vmware-vsan-6-6.html

https://storagehub.vmware.com/#!/vmware-vsan/vsan-multicast-removal/multicast-removal-steps-and-requirements/1

vSAN 6.6 Encryption Configuration

vSAN 6.6 – Native Data-at-Rest Encryption

Goodbye Multicast

Native VCSA bootstrap installer in vSAN 6.6

Worth a Repost: “VMware Doubles Down” vCloud Director 8.20

It seems that with the announcement last week that VMware was offloading vCloud Air to OVH people where again asking what is happening with vCloud Director….and the vCloud Air Network in general. While vCD is still not available for VMware’s enterprise customers, the vCloud Director platform has officially never been in a stronger position.

Those outside the vCAN inner circles probably are not aware of this and I still personally field a lot of questions about vCD and where it sits in regards to VMware’s plans. Apparently the vCloud Team has again sought to clear the air about vCloud Director’s future and posted this fairly emotive blog post overnight.

I’ve reposted part of the article below:

Blogger Blast: VMware vCloud Director 8.20

We are pleased to confirm that vCloud Director continues to be owned and developed by VMware’s Cloud Provider Software Business Unit and is the strategic cloud management platform for vCloud Air Network service providers. VMware has been and continues to be committed to its investment and innovation in vCloud Director.

With the recent release of vCloud Director 8.20 in February 2017 VMware has doubled down on its dedication to enhancing the product, and, in addition, is working to expand its training program to keep pace with the evolving needs of its users. In December 2016 we launched the Instructor Led Training for vCloud Director 8.10 (information and registration link) and in June 2017 we are pleased to be able to offer a Instructor Led Training program for vCloud Director 8.20.

Exciting progress is also occurring with vCloud Director’s expanding partner ecosystem. We are working to provide ISVs with streamlined access and certification to vCloud Director to provide service providers with access to more pre-certified capabilities with the ongoing new releases of vCloud Director. By extending our ecosystem service providers are able to more rapidly monetize services for their customers

Again, this is exciting times for those who are running vCloud Director SP and those looking to implement vCD into their IaaS offerings. It should be an interesting year and I look forward to VMware building on this renewed momentum for vCloud Director. There are many people blogging about vCD again which is awesome to see and it gives everyone in the vCloud Air Network an excellent content from which to leach from.

The vCloud Director Team also has a VMLive session that will provide a sneak peek at vCloud Director.Next roadmap. So if you are not a VMware Partner Central member and work for a vCloud Air Network provider wanting to know about where vCD is heading…sign up.

#LongLivevCD

vCloud Air Sold to OVH – Final Thoughts On Project Zephyr

I’ve just spent the last fifteen minutes looking back through all my posts on vCloud Air over the last four or five years and given yesterday’s announcement that VMware was selling what remains of vCloud Air to OVH Going over the content I thought it would be pertinent to write up one last piece on VMware’s attempt to build a public cloud that tried compete against the might of AWS, Azure, Google and the other well established hyper-scalers.

Project Zephyr:

Project Zephyr was first rumoured during 2012 and later launched as VMware Cloud Hybrid Services or vCHS…and while VMware pushed the cloud platform as a competitor to the hyper-scalers, the fact that it was built upon vCloud Director was probably one of it’s biggest downfalls. That might come as a shock to a lot of you reading this to hear me talk bad about vCD, however it wasn’t so much the fact that vCD was used as the backend, it was more what the consumer saw at the frontend that for me posed a significant problem for it’s initial uptake.

VMworld – Where is the Zephyr?

It was the perfect opportunity for VMware to deliver a completely new and modern UI for vCD and even though they did front the legacy vCD UI will a new frontend it wasn’t game changing enough to draw people in. It was utilitarian at best, but given that you only had to provision VMs it didn’t do enough to show that the service was cutting edge.  Obviously the UI wasn’t the only reason why it failed to take off…using vCD meant that vCloud Air was limited by the fact that vCD wasn’t built for hyper-scale operations such as individual VM instance management or for platform as a service offerings. The lack of PaaS offerings in effect meant it was a glorified extension of existing vCloud Air Network provider clouds…which in fact was some of the key messaging VMware used in the early days.

The use of vCD did deliver benefits to the vCloud Air Network and in truth might have saved vCD from being put on the scrapheap before VMware renewed their commitment to develop the SP version which has resulted in a new UI being introduced for Advanced Networking in 8.20.

vCloud Air Struggles:

There was no hiding the fact that vCloud Air was struggling to gain traction world wide and even as other zones where opening around the world it seemed like VMware where always playing catchup with the hyper-scalers…but the reality of what the platform was meant that there never a chance vCloud Air would grow to rival AWS, Azure and others.

By late 2015 there was a joint venture between EMC’s Virtustream and VMware vCloud Air that looked to join the best of both offerings under the Virtustream banner where they looked to form a new hybrid cloud services business but the DELL/EMC merger looked to get in the way of that deal and by December 2015 the idea has been squashed.

vCloud Air and Virtustream – Just kill vCloud Air Already?!?

vCloud Air and Virtustream – Ok…So This Might Not Happen!

It appeared from the outside that vCloud Air never recovered from that missed opportunity and through 2016 there where a number of announcements that started in March when it was reported that vCloud Air Japan was to be sold to the company that was effectively funding the zone and effectively closed down.

HOTP: vCloud Air Japan to be Shutdown!

Then in June VMware announced that Credit Card payments would no longer be accepted for any vCloud Air online transactions and that the service had to be bought with pre purchased credits through partners. For me this was the final nail in the coffin in terms of vCloud Air being able to compete in the Public Cloud space.

vCloud Air – Pulling Back Credit Card Payments

From this point forward the messaging for the use case of vCloud Air had shifted to Disaster Recovery services via the Hybrid Cloud Manager and vSphere Replication services that where built to work directly from vSphere to vCloud Air endpoints.

vCloud Air Network:

Stepping back, just before VMworld 2014, VMware announced the rebranding of vCHS to what is now called vCloud Air and also launched the vCloud Air Network. Myself and others where pretty happy at the time that VMware looked to reconnect with their service provider partners.

With the announcement around the full rebranding of vCHS to vCloud Air and Transforming the VSPP and vCloud Powered programs to the vCloud Air Network it would appear that VMware has in fact gone the other way and recommitted their support to all vCloud Server Providers and has even sort out to make the partner relationship stronger. The premise being that together, there is a ready made network (Including vCloud Air) of providers around the world ready to take on the greater uptake of Hybrid Cloud that’s expected over the next couple of years.

So while vCloud Air existed VMware acknowledged that more success was possible through support the vCloud Air Network ecosystem as the enabler of hybrid cloud services.

Final Final Thoughts:

To say that I’ve had a love hate relationship with the idea of VMware having a public cloud is reflected in my posts over the years. In truth myself and others who formed part of the vCloud Air Network of VMware based service providers where never really thrilled about the idea of VMware competing directly against their own partners.

vCHS vs. vCloud Providers: The Elephant in the Cloud

I would now say that many would be glad to see it handed over to OVH…because now VMware does not compete against it’s vCAN Service Providers directly, but can continue to hopefully focus on enabling them with the best tools to power their own cloud or provider platforms and help the network grow successfully as what the likes of OVH, iLand, Zettagrid and others have been able to so.

Pat Gelsinger statement in regards to the sale to OVH are very postive for the vCloud Air Network and I believe for VMware hybrid cloud vision that it revealed at VMworld last year can now proceed without this lingering in the corner.

“We remain committed to delivering our broader cross-cloud architecture that extends our hybrid cloud strategy, enabling customers to run, manage, connect, and secure their applications across clouds and devices in a common operating environment”

The VMware vCloud blog here talks about what OVH will bring to the table for the customers that remain on vCloud Air. Overall it’s extremely positive for those customers and they can take advantage of the technical ability and execution of the vCloud Air Networks leading service provider. Overall I think this is a great move by VMware and will hopefully lead to the vCloud Air Network becoming stronger…not weaker.

vCloud Director SP 8.20 – NSX Advanced Networking Overview

Many, including myself thought that the day would never come where we would be talking about a new UI for vCloud Director…but a a month on from the 8.20 release of vCloud Director SP (which was the 8th major release of vCD) I’m happy to be writing about the new Advanced Networking features of 8.20 based on NSX-v. Full NSX compatibility and interoperability has been a long time coming, however the wait has been worthwhile as the vCloud Director team opted to fully integrate the network management into the vCD Cloud Cells over the initial approach that had a seperate appliance acting as a proxy between the NSX Manager and vCD Cells.

But before I dive into the new HTML5 goodness, I thought it would be good to recap the Advanced Networking Services of vCD and how we got to where we are today…

No More vShield…Sort Of:

As everyone should know by now, the vCloud Networking & Security was made end of life late last year and from the release of vCD SP 8.10 vShield Edges should have been upgraded to their NSX equivalents. These Edges will remain as basic Edges within vCloud Director and even though at the backend they would be on NSX-v versioning, no extra features or functionality beyond what was available in the existing vCD portal would be available to tenants.

  • DHCP
  • NAT
  • Firewall
  • Static Routing
  • IPSec VPN
  • Basic Load Balancer

The version of NSX-v deployed dictates the build number of the NSX Edge, however as can be seen below it’s still listed as a vShield Edge in vCenter.

As anyone who has worked closely would know, NSX-v has a lot of vShield DNA in it and in truth it’s more vShield than NSX when talking about the features that pertain to vCloud Director. However the power of NSX-v can be taken advantage of once an basic edge is upgraded to an Advanced Edge.

Advanced Edge Services:

Before the major UI additions that came with vCD SP 8.20 the previous 8.10 version did give us a taste of what was to come with the introduction of a new menu option when you right clicked on an Edge Gateway.

This option was greyed out unless you where running the initial beta of the Advanced Networking Services or ANS. The option can be executed by anyone with the rights to upgrade the edge gateway, but by default this can only be done by a System Administrator or the Org Admin. So it’s worthwhile double checking the roles you have allocated to your tenant’s to ensure that these upgrades can be controlled.

Once you click on the Convert to Advanced Gateway option you get a warning referring to a VMwareKB that warns you about an API change that may make previous calling methods obsolete. Something to take note of for anyone automating this process. On execution of this conversion there is no physical change to the Virtual Machine, however if you now click on the Edge Gateway Services option of the Edge Gateway you will be taken to the new HTML5 Web Interface for NSX Advanced Networking Services to access all the advanced features:

  • Firewall
  • DHCP
  • NAT
  • Routing (Dynamic)
  • Load Balancer (Advanced)
  • SSL VPN Plus
  • Certificates
  • Grouping Objects
  • Statistics
  • Edge Settings

All new Advanced Networking features are configured from the new HTML5 web interface which retains the base vCD URL but now adds:

/tenant/network-edges/{ID}?org=ORGNAME

Everything is self contained the tenant doesn’t have to authenticate again to get to the new user interface. However, if you just upgrade the Edge and go to configure the Advanced Network Services out of the box you will only see a couple of the items listed above.

In order to use the new features a System Administrator must use the vCloud API to grant the new rights that the organisation requires. This process has been explained very well by my good friend Giuliano Bertello here. This process uses the vCloud API to Grant Distributed Firewall and Advanced Networking Services Rights to roles in vCloud Director 8.20 using the new granular role based access control mechanisms that where introduced in 8.20. Once configured your tenant’s can now see all the services listed above to configure the Edge Gateway.

Organisational Distributed Firewall:

Something that is very much new in the 8.20 release is the ability to take advantage of mircosegmentation using the NSX-v Distributed Firewall service. The ability to configure organisation wide rules logically, without the need for a virtual Edge Gateway is a significant step forward for vCD tenants and I hope that this feature enhancement is exposed by service providers and it’s value sold to their tenants. To access the Distributed Firewall, in the Virtual Datacenters windows of the Administration tab, right click on the Virtual Datacenter name and select Manage Firewall.

Once again you will be taken to the new HTML5 user interface and once the correct permissions have been applied to the user you can enable the Distributed Firewall and start configuring your rules. The URL is slightly different to the Edge Gateway URL:

/tenant/dwf/{ID}?org=ORGNAME

But the look and feel is familiar.

Conclusion:

vCloud Director SP 8.20 has finally delivered on the what most members of the vCloud Air Network had wanted for some time…that is, full NSX interoperability and feature set access as well as a new user interface. Over the next few weeks, I am going to expand on all the features of the Advanced and Distributed Networking features of vCD and NSX and walk through how to configure elements through the UI and API as well as give a looks into what’s happening at the backend in terms of how NSX stores rules and policy items for vCD tenant use.

Compatibility with vSphere 6.5 and NSX-v 6.3.x:

vCloud Director SP 8.20 is compatible with vSphere 6.5 and NSX 6.3.0 and supports full interoperability with other versions as shown in the VMware Product Interoperability Matrix. As of vCD 8.20 GA, vCD 8.20 passed the functional interoperability test and limited scale testing for these versions:

  • vCD 8.20 with vSphere 6.0 and NSX 6.3.0
  • vCD 8.20 with vSphere 6.5 and NSX 6.3.0

References:

https://kb.vmware.com/kb/2149042
https://kb.vmware.com/kb/2147625

VMworld 2017 : Session Analysis and Voting Open…Already!

Well this has crept up on us quickly this year! It’s time to vote for the VMworld Sessions that will be part of the US and Europe VMworld’s held later in the year. The Session Voting is more Session liking as you have the ability to mark multiple sessions as ones that you would like to see. Apparently there are 1900 odd sessions, however the voting site lists 1500 that where submitted and are listed in the Online Catalog.

[EDIT]: I totally forgot that David Hill submitted a joint session

David Hill, VMware
Anthony Spiteri, Veeam
Service Providers globally are enthusiastically embracing hybrid cloud as both a way of reducing costs and improving the quality of service they provide to end customers. To achieve this, Service Providers are looking to VMware vCloud Air Network and Veeam to help them build a scalable cost effect cloud solution. In this session we will get into the details of the technology. We’ll focus on how these solutions are architected and what that implies in real-life implementations. A participant in this session will leave with a technical understanding of how to leverage technology to provide a successful cloud based storage service.
Session Type:  Breakout Session
Track :  Integrate Public Clouds
Integrate Public Clouds Subtrack:  Leverage Hybrid Clouds
Market Segment:  No Specific Segment
Session Audience:  IT – All, IT – Operations, Technical Support
Product and Topics:  vCloud
Technical Level:  Technical – Advanced
Event Submitted For:  Both

Having gone through the session catalog it’s pleasing to see the number of cloud and service provider content feature prominently again. NSX, vSAN and vCloud Director all have a number of decent sessions. Click below on the links to view the related sessions.

I was surprised with the number of sessions submitted for NSX-T, but there is obviously a lot of interest in the hypervisor agnostic NSX…along side the huge number of NSX-v sessions listed. AWS is mentioned in 51 sessions showing you another change of direction at VMworld and there is a decent amount of automation and container based sessions. It will be interesting to see what the final cut is and how those numbers look when the offical catalog becomes available.

As per the VMworld Session Voting FAQ Session Voting is open March 28 to April 16…which is a lot earlier than last years May 12 to May 26 voting period. So it’s not just me that thinks this has come around sooner than previous years.

VMworld 2017 registration goes live on April 4!

It’s ok to steal… VMUG UserCon Key Take Aways

Last week I attended the Sydney and Melbourne VMUG UserCons and apart from sitting in on some great sessions I came away from both events with a renewed sense of community spirit and enjoyed catching up with industry peers and good friends that I don’t see often enough. While the VMUG is generally struggling a little around the world at this point in time, kudos goes to both Sydney and Melbourne chapter leaders and steering committee in being able to bring out a superstar bunch of presenters (see panel below)…there might not be a better VMUG lineup anywhere in the world this year!

There was a heavy automation focus this year…which in truth was the same as last years events however last years messaging was more around the theory of _change or die_ this year there was more around the practical. This was a welcome change because, while it’s all well and good to beat the change messaging into people…actually taking them through real world examples and demo’s tends to get people more excited and keen to dive into automation as they get a sense of how to apply it to their every day jobs.

In the VMware community, there are not better examples of automation excellence than Alan Renouf and William Lam and their closing keynote sessions where they went through and deployed a fully functional SDDC vSphere environment on a single ESXi host from a USB Key was brilliant and hopefully will be repeated at other VMUGs and VMworld. This project was born out of last years VMworld Hackerthon’s and ended up being a really fun and informative presentation that showed off the power of automation along with the benefits of what undertaking an automation project can deliver.

“Its not stealing, its sharing” 

During the presentation Alan Renouf shared this slide which got many laughs and resonated well with myself in that apart from my very early failed uni days, I don’t think I have ever created a bit of code or written a script from scratch. There is somewhat of a stigma attached with “borrowing” or “stealing” code used to modify or create scripts within the IT community. There might also be some shame associated in admitting that a bit of code wasn’t 100% created by someone from scratch…I’ve seen this before and I’ve personally been taken to task when presenting some of the scripts that I’ve modified for purpose during my last few roles.

What Alan is pointing out there is that it’s totally ok to stand on the shoulders of giants and borrow from what’s out there in the public domain…if code is published online via someones personal blog or put up on GitHub then it’s fair game. There is no shame in being efficient…no shame in not having to start from scratch and certainly no shame in claiming success after any mods have been done… Own it!

Conclusion and Event Wrap Up:

Overall the 2017 Sydney and Melbourne UserCons where an excellent event and on a personal note I enjoyed being able to attend with Veeam as the Platinum Sponsor and present session on our vSAN/VVOL/SPBM support and introduce our Windows and Linux Agents to the crowd. The Melbourne crowd was especially engaged and asked lots of great questions around our agent story and where looking forward to the release of Veeam Agent for Windows.

Again the networking with industry peers and customers is invaluable and there was a great sense of community once again. The UserCon events are of a high quality and my thanks goes out to the leaders of both Sydney and Melbourne for working hard to organise these events. And which one was better? …I won’t go there but those that listened to my comment during our Sponsor giveaways at the end of the event knows how I really feel.

Until next year UserCon!

ESXi 6.5 Storage Performance Issues and Fix

[NOTE] : I decided to republish this post with a new heading and skip right to the meat of the issue as I’ve had a lot of people reach out saying that the post helped them with their performance issues on ESXi 6.5. Hopefully people can find the content easier and have a fix in place sooner.

The issue that I came across was to do with storage performance and the native driver that comes bundled with ESXi 6.5. With the release of vSphere 6.5 yesterday, the timing was perfect to install ESXI 6.5 and start to build my management VMs. I first noticed some issues when uploading the Windows 2016 ISO to the datastore with the ISO taking about 30 minutes to upload. From there I created a new VM and installed Windows…this took about two hours to complete which I knew was not as I had expected…especially with the datastore being a decent class SSD.

I created a new VM and kicked off a new install, but this time I opened ESXTOP to see what was going on, and as you can see from the screen shots below, the Kernel and disk write latencies where off the charts topping 2000ms and 700-1000ms respectively…In throuput terms I was getting about 10-20MB/s when I should have been getting 400-500MB/s. 

ESXTOP was showing the VM with even worse write latency.

I thought to myself if I had bought a lemon of a storage controller and checked the Queue Depth of the card. It’s listed with a QD of 31 which isn’t horrible for a homelab so my attention turned to the driver. Again referencing the VMware Compatibility Guide the listed driver for the controller the device driver is listed as ahci version 3.0.22vmw.

I searched for the installed device driver modules and found that the one listed above was present, however there was also a native VMware device drive as well.

I confirmed that the storage controller was using the native VMware driver and went about disabling it as per this VMwareKB (thanks to @fbuechsel who pointed me in the right direction in the vExpert Slack Homelab Channel) as shown below.

After the host rebooted I checked to see if the storage controller was using the device driver listed in the compatibility guide. As you can see below not only was it using that driver, but it was now showing the six HBA ports as opposed to just the one seen in the first snippet above.

I once again created a new VM and installed Windows and this time the install completed in a little under five minutes! Quiet a difference! Upon running a crystal disk mark I was now getting the expected speeds from the SSDs and things are moving along quiet nicely.

Hopefully this post saves anyone else who might by this, or other SuperMicro SuperServers some time and not get caught out by poor storage performance caused by the native VMware driver packaged with ESXi 6.5.


References
:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2044993

VMUG UserCon – Sydney and Melbourne Events are Huge This Year!

Last year I claimed that the Melbourne VMUG Usercon was the “Best Virtualisation Event Outside of VMworld!” …that was a big statement if ever there was one however, fast forward a year and credit goes to the VMUG steering committee’s of both Sydney and Melbourne as it seems they have bettered themselves this time around for the 2017 editions. Both events happen a couple of days apart from each other on the 21st and 23rd of March and both are filled with quality content, quality presenters and a great community feel.

This will be my fourth Melbourne UserCon and my second Sydney UserCon…The last couple of years I have attended the event in Melbourne I have taken away a lot of great technical and non-technical knowledge back home with me and with keynote speakers the likes of no less than Duncan Epping and Amy Lewis together with other industry superstars William LamAlan Renouf, Emad Younis, Josh Atwell and other great local presenters I expect the same for the 2017 editions.

With what is the strongest lineup in the Usercon’s history, it promises to be a worthwhile event to attend…if you haven’t already registered head to the registration pages below and sign up.

Both the Sydney and Melbourne Agenda’s are jam packed with virtualisation goodness and it’s actually hard to attend everything of interest with schedule conflicts happening throughout the day…the agenda’s are not yet 100% completed on the sites but make you check back later in the week to get details on who is presenting what and when.

Veeam is sponsoring both events as Platinum level sponsors and I’ll be presenting a session on Availability made easy for your vSphere infrastructure where I’ll go through some tips and tricks about getting the most out of Veeam and vSphere as well as talk about how we extend availability into the cloud.

If you are in Sydney or Melbourne next week try and get down to The Westin and The Crown Casino respectively to participate, learn and contribute and hopefully we can catch up for a drink.

« Older Entries Recent Entries »