Tag Archives: vSphere

VMware vSphere 6.5 Host Resources Deep Dive – A Must Have!

Just after I joined Zettagrid in June of 2013 I decided to load up vSphere 5.1 Clustering Deepdive by Duncan Epping and Frank Denneman on my iPad to read on my train journey to and from work. Reading that book allowed me to gain a deeper understanding of vSphere through the in depth content that Duncan and Frank had produced. Any VMware administrator worth their salt would be familiar with the book (or the ones that proceeded it) and it’s still a brilliant read.

Fast forward a few versions of vSphere and we finally have follow up:

VMware vSphere 6.5 Host Resources Deep Dive

This time around Frank has been joined by Niels Hagoort and together they have produced another must have virtualization book…though it goes far beyond VMware virtualization. I was lucky enough to review a couple of chapters of the book and I can say without question that this book will make your brain hurt…but in a good way. It’s the deepest of deep dives and it goes beyond the previous books best practice and dives into a lot of the low level compute, storage and networking fundamentals that a lot of us have either forgotten about, never learnt or never bothered to learn about.

This book explains the concepts and mechanisms behind the physical resource components and the VMkernel resource schedulers, which enables you to:

  • Optimize your workload for current and future Non-Uniform Memory Access (NUMA) systems.
  • Discover how vSphere Balanced Power Management takes advantage of the CPU Turbo Boost functionality, and why High Performance does not.
  • How the 3-DIMMs per Channel configuration results in a 10-20% performance drop.
  • How TLB works and why it is bad to disable large pages in virtualized environments.
  • Why 3D XPoint is perfect for the vSAN caching tier.
  • What queues are and where they live inside the end-to-end storage data paths.
  • Tune VMkernel components to optimize performance for VXLAN network traffic and NFV environments.
  • Why Intel’s Data Plane Development Kit significantly boosts packet processing performance.

If any of you have read Frank’s NUMA Deep Dive blog series you will start to get an appreciation of the level of technical detail this book covers, however it is written in a way that allows you absorb the information in a way that is digestible, though some parts may need to be read twice over. Well done to Frank and Niels on getting this book out and again, if you are working in and around anything to do with computers this is a must read so do yourself a favour and grab a copy.

The current Amazon locals that have access to purchase the book can be found below:

Amazon US: http://www.amazon.com/dp/1540873064
Amazon France: https://www.amazon.fr/dp/1540873064
Amazon Germany: https://www.amazon.de/dp/1540873064
Amazon India: http://www.amazon.in/dp/1540873064
Amazon Japan: https://www.amazon.co.jp/dp/1540873064
Amazon Mexico: https://www.amazon.com.mx/dp/1540873064
Amazon Spain: https://www.amazon.es/dp/1540873064
Amazon UK: https://www.amazon.co.uk/dp/1540873064

Released: vCenter and ESXi 6.0 Update 3 – What’s in It for Service Providers

Last month I wrote a blog post on upgrading vCenter 5.5 to 6.0 Update 2 and during the course of writing that blog post I conducted a survey on which version of vSphere most people where seeing out in the wild…overwhelmingly vSphere 6.0 was the most popular version with 5.5 second and 6.5 lagging in adoption for the moment. It’s safe to assume that vCenter 6.0 and ESXi 6.0 will be common deployments for some time in brownfield sites and with the release of Update 3 for vCenter and ESXi I thought it would be good to again highlight some of the best features and enhancements as I see them from a Service Provider point of view.

vCenter 6.0 Update 3 (Build 5112506)

This is actually the eighth build release of vCenter 6.0 and includes updated TLS support for v1.0 1.1 and 1.2 which is worth a look in terms of what it means for other VMware products as it could impact connectivity…I know that vCloud Director SP now expects TLSv 1.1 by default as an example. Other things listed in the What’s New include support for MSSQL 2012 SP3, updated M2VCSA support, timezone updates and some changes to the resource allocation for the platform services controller.

Looking through the Resolved Issue there are a number of networking related fixes in the release plus a few annoying problems relating to vMotion. The ones below are the main ones that could impact on Service Provider operations.

  • Upgrading vCenter Server from version 6.0.0b to 6.0.x might fail. 
    Attempts to upgrade vCenter Server from version 6.0.0b to 6.0.x might fail. This issue occurs while starting service An error message similar to the following is displayed in the run-updateboot-scripts.log file.
    “Installation of component VCSServiceManager failed with error code ‘1603’”
  • Managing legacy ESXi from the vCenter Server with TLSv1.0 disabled is impacted.
    vCenter Server with TLSv1.0 disabled supports management of legacy ESXi versions in 5.5.x and 6.0.x. ESXi 5.5 P08 and ESXi 6.0 P02 onwards is supported for 5.5.x and 6.0.x respectively.
  • x-VC operations involving legacy ESXi 5.5 host succeeds.
    x-VC operations involving legacy ESXi 5.5 host succeeds. Cold relocate and clone have been implicitly allowed for ESXi 5.5 host.
  • Unable to use End Vmware Tools install option using vSphere Client.
    Unable to use End VMware Tools install option while installing VMware Tools using vSphere Client. This issue occurs after upgrading to vCenter Server 6.0 Update 1.
  • Enhanced vMotion fails to move the vApp.VmConfigInfo property to destination vCenter Server.
    Enhanced vMotion fails to move the vApp.VmConfigInfo property to destination vCenter Server although virtual machine migration is successful.
  • Storage vMotion fails if the VM is connected with a CD ISO file.
    If the VM is connected with a CD ISO file, Storage vMotion fails with an error similar to the following:
  • Unregistering an extension does not delete agencies created by a solution plug-in.
    The agencies or agents created by a solution such as NSX, or any other solution which uses EAM is not deleted from the database when the solution is unregistered as an extension in vCenter Server.

ESXi 6.0 Update 3 (Build 5050593)

The what’s new in ESXi is a lot more exciting than what’s new with vCenter highlighted by a new Host Client and fairly significant improvements in vSAN performance along with similar TLS changes that are included in the vCenter update 3. With regards to the Host Client the version is now 1.14.0. and includes bug fixes and brings it closer to the functionality provided by the vSphere Client. It’s also worth mentioning that new versions of the Host Client continue to be released through the VMware Labs Flings site. but, those versions are not officially supported and not recommended for production environments.

For vSAN, multiple fixes have been introduced to optimize I/O path for improved vSAN performance in All Flash and Hybrid configurations and there is a seperate VMwareKB that address the fixes here.

  • More Logs Much less Space vSAN now has efficient log management strategies that allows more logging to be packed per byte of storage. This prevents the log from reaching its assigned limit too fast and too frequently. It also provides enough time for vSAN to process the log entries before it reaches it’s assigned limit thereby avoiding unnecessary I/O operations
  • Pre-emptive de-staging vSAN has built in algorithms that de-stages data on periodic basis. The de-staging operations coupled with efficient log management significantly improves performance for large file deletes including performance for write intensive workloads
  • Checksum  Improvements vSAN has several enhancements that made the checksum code path more efficient. These changes are expected to be extremely beneficial and make a significant impact on all flash configurations, as there is no additional read cache look up. These enhancements are expected to provide significant performance benefits for both sequential and random workloads.

As with vCenter, I’ve gone through and picked out the most significant bug fixes as they relate to Service Providers. The first one listed below is important to think about as it should significantly reduce the number of failures that people have been seeing with ESXi installed on SD-Flash Card and not just for VDI environments as the release notes suggest.

  • High read load of VMware Tools ISO images might cause corruption of flash media  In VDI environment, the high read load of the VMware Tools images can result in corruption of the flash media.
    You can copy all the VMware Tools data into its own ramdisk. As a result, the data can be read from the flash media only once per boot. All other reads will go to the ramdisk. vCenter Server Agent (vpxa) accesses this data through the /vmimages directory which has symlinks that point to productLocker.
  • ESXi 6.x hosts stop responding after running for 85 days
    When this problem occurs, the /var/log/vmkernel log file displays entries similar to the followingARP request packets might drop.
  • ARP request packets between two VMs might be dropped if one VM is configured with guest VLAN tagging and the other VM is configured with virtual switch VLAN tagging, and VLAN offload is turned off on the VMs.
  • Physical switch flooded with RARP packets when using Citrix VDI PXE boot
    When you boot a virtual machine for Citrix VDI, the physical switch is flooded with RARP packets (over 1000) which might cause network connections to drop and a momentary outage. This release provides an advanced option /Net/NetSendRARPOnPortEnablement. You need to set the value for /Net/NetSendRARPOnPortEnablementto 0 to resolve this issue.
  • Snapshot creation task cancellation for Virtual Volumes might result in data loss
    Attempts to cancel snapshot creation for a VM whose VMDKs are on Virtual Volumes datastores might result in virtual disks not getting rolled back properly and consequent data loss. This situation occurs when a VM has multiple VMDKs with the same name and these come from different Virtual Volumes datastores.
  • VMDK does not roll back properly when snapshot creation fails for Virtual Volumes VMs
    When snapshot creation attempts for a Virtual Volumes VM fail, the VMDK is tied to an incorrect data Virtual Volume. The issue occurs only when the VMDK for the Virtual Volumes VM comes from multiple Virtual Volumes datastores.
  • ESXi host fails with a purple diagnostic screen due to path claiming conflicts
    An ESXi host displays a purple diagnostic screen when it encounters a device that is registered, but whose paths are claimed by a two multipath plugins, for example EMC PowerPath and the Native Multipathing Plugin (NMP). This type of conflict occurs when a plugin claim rule fails to claim the path and NMP claims the path by default. NMP tries to register the device but because the device is already registered by the other plugin, a race condition occurs and triggers an ESXi host failure.
  • ESXi host fails with a purple diagnostic screen due to path claiming conflicts
    An ESXi host displays a purple diagnostic screen when it encounters a device that is registered, but whose paths are claimed by a two multipath plugins, for example EMC PowerPath and the Native Multipathing Plugin (NMP). This type of conflict occurs when a plugin claim rule fails to claim the path and NMP claims the path by default. NMP tries to register the device but because the device is already registered by the other plugin, a race condition occurs and triggers an ESXi host failure.
  • ESXi host fails to rejoin VMware Virtual SAN cluster after a reboot
    Attempts to rejoin the VMware Virtual SAN cluster manually after a reboot might fail with the following error:
    Failed to join the host in VSAN cluster (Failed to start vsantraced (return code 2)
  • Virtual SAN Disk Rebalance task halts at 5% for more than 24 hours
    The Virtual SAN Health Service reports Virtual SAN Disk Balance warnings in the vSphere Web Client. When you click Rebalance disks, the task appears to halt at 5% for more than 24 hours.

It’s also worth reading through the Known Issues section as there is a fair bit to be aware of especially if running NFS 4.1 and worth looking through the general storage issues.

Happy upgrading!

References:

http://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-vcenter-server-60u3-release-notes.html

http://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-esxi-60u3-release-notes.html

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2149127

Upgrading Windows vCenter 5.5 to 6.0 In-Place: Issues and Fixes

Yes that’s not a typo…this post is focusing on upgrading Windows vCenter 5.5 to 6.0 via an in-place upgrade. There is the option to use the vSphere 6.0 Update2M build with the included Migrate to VCSA tool to achieve this and move away from Windows, but I thought it was worth documenting my experiences with a mature vCenter that’s at version 5.5 Update 2 and upgrade that to 6.0 Update 2. Eventually this vCenter will need to move off the current Windows 2008 RTM server which will bring into play the VCSA migration however for the moment it’s going to be upgraded to 6.0 on the same server.

With VMware releasing vSphere 6.5 in November there should be an increased desire for IT shops to start seriously thinking about moving on from there existing vSphere versions and upgrading to the latest 6.5 release however many people I know where still running vSphere 5.5, so the jump to 6.5 directly might not be possible due to internal policies or other business reasons. Interestingly in the rough numbers, I’ve got an active Twitter Poll out at the moment which after 100 votes shows that vSphere 5.5 makes up 53% of the most common vCenter version, followed by 6.0 with 44% and 6.5 with only 3%.

Upgrade Options:

You basically have two options to upgrade a Windows based 5.5 vCenter:

My approach for this particular environment (which is a NestedESXi lab environment) was to ensure a smooth upgrade to vSphere 6.0 Update 2 and then look to upgrade again to 6.5 once is thaws outs in the market. That said, I haven’t read too many issues with vSphere 6.5 and VMware have been excellent in ensuring that the 6.5 release was the most stable for years. The cautious approach will still be undertaken by many and a stepped upgrade to 6.5 and migration to the VCSA will be common place. For those that wish to move away from their Windows vCenter, there is nothing stopping you from going down the Migrate2VCSA path, and it is possible to migrate directly from 5.5 to 6.5.

Existing Component Versions:

  • vCenter 5.5 (2001466)
  • ESXi 5.5 (3116895)

SQL Version Requirements:

vCenter 6.0 Update 2 requires at least SQL Server 2008 R2 SP1 or higher, so if you are running anything lower than that you will need to upgrade to a later service pack or upgrade to later versions of SQL Server. For a list of all compatible databases click here.

vCenter Upgrade Pre-Upgrade Checks:

First step is to make sure you have a backup of the vCenter environment meaning VM state (Snapshot) and vCenter database backup. Once that’s done there are a few pre-requisites that need to be met and that will be checked by the upgrade process before the actual upgrade occurs. The first thing the installer will do after asking for the SSO and VC service account password is run the Pre-Upgrade Checker.

vCenter SSL and SSO SSL System Name Mismatch Error:

A common issue that may pop up from the pre-upgrade checker is the warning below talking about an issue with the system name of the vCenter Server certificate and the SSO certificate. As shown below it’s a hard stop and tells you to replace one or the other certificate so that the same system name is used.

If you have a publicly signed SSL Certificate you will need to generate a new cert request and submit that through the public authority of choice. The quickest way to achieve this for me was to generate a new self signed certificate by following the VMwareKB article here. Once that’s been generated you can replace the existing certificate by following a previous post I did using the VMware SSL Certificate Updater Tool.

After all that, in any case I got the warning below saying that the 5.5 SSL Certificates do not meet security requirements, and so new SSL certificates will need to be generated for vCenter Server 6.0.0.

With that, my suggestion would be to generate a temporary self signed certificate for the upgrade and then apply a public certificate after that’s completed.

Ephemeral TCP Port Error:

Once the SSL mismatch error has been sorted you can run the pre-upgrade checker again. Once that completes successfully you move onto the Configure Ports window. I ran into the error shown below that states that the range of port is too large and the system must be reconfigured to use a smaller ephemeral port range before the install can continue.

The fix is presented in the error message so after running netsh.exe int ipv4 set dynamicportrange tcp 49152 16384 you should be ok to hit Next again and continue the upgrade.

Export of 5.x Data:

During the upgrade the 5.5 data is stored in a directory and then migrated to 6.0. You need to ensure that you have enough room on the drive location to cater for your vCenter instance. While I haven’t seen any offical rules around the storage required, I would suggest having enough storage free and the size of your vCenter SQL database data file.

vCenter Upgrade:

Once you have worked through all the upgrade screens you are ready for upgrade. Confirm the settings, take note of the fact that once updated the vCenter will be in evaluation mode, meaning you need to apply a new vCenter 6.x license once completed, check the checkbox that states you have a backup of the vCenter machine and database and you should be good to go.

Depending on the size of you vCenter instance and the speed of your disks the upgrade can take anywhere from 30 to 60 minutes or longer. If at any time the upgrade process fails during the initial export of the 5.5 data a roll back via the installer is possible…however if there is an issue while 6.0 is being installed the likelihood is that you will need to recover from backups.

Post Upgrade Checks:

Apart from making sure that the upgrade has gone through smoothly by ensuring all core vCenter services are up and running, it’s important to check any VMware or third party services that where registered against the vCenter especially given that the SSL Certificate has been replaced a couple of times. Server applications like NSX-v, vCloud Director and vCO explicitly trust SSL certificates so the registration needs to be actioned again. Also if you are running Veeam Backup & Replication you will need to go through the setup process again to accept the new SSL Certificate otherwise your backup jobs will fail.

If everything has gone as expected you will have a functional vCenter 6.0 Update 2 instance and planning can now take place for the 6.5 upgrade and in my case…the migration from Windows to the VCSA.

References:

http://www.vmware.com/resources/compatibility/sim/interop_matrix.php#db&2=998

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1029944

 

vSphere 6.5 – Whats in it for Service Providers Part 1

Last week after an extended period of development and beta testing VMware released vSphere 6.5. This is a lot more than a point release and is a major major upgrade from vSphere 6.0. In fact, there is so much packed into this new release that there is an official whitepaper listing all the features and enhancements that had been linked from the release notes.  I thought I would go through some of the key features and enhancements that are included in the latest versions of vCenter and ESXi and as per usual I’ll go through those improvements that relate back to the Service Providers that use vSphere as the foundation of their Managed or Infrastructure as a Service offerings.

Generally the “whats new” would fit into one post, however having gotten through just the vCenter features it became apparent that this would have to be a multi-post series…this is great news for vCloud Air Network Service Providers out there as it means there is a lot packed in for IaaS and MSPs to take advantage of.

With that, in this post will cover the following:

  • vCenter 6.5 New Features
  • vCD and NSX Compatibility
  • Current Known Issues

vCenter 6.5 New Features:

Without question the enhancements to the VCSA stand out as one of the biggest features of 6.5 and as mentioned in the whitepaper, the installer process has been overhauled and is a much smoother, streamlined experience than with previous versions. It’s also supported across more operating systems and the 6.5 version of vCenter now surpasses the Windows version offering the migration tool, native high availability and built in backup and restore. One interesting sidenote to the new VCSA is that the HTML5 vSphere Client has shipped, though it’s still very much a work in progress as a lot of unsupported functionality mentioned in the release notes…there is lots of work to do to bring it up to parity with the Flex Web Client.

In terms of the inbuilt PostGreSQL database I think it’s time that Service Providers feel confident in making the switch away from MSSQL (which was the norm with Windows based vCenters) as the enhanced VCSA Management Interface (found on port 5480) has a new monitoring screen showing information relating to disk space usage and also provides a way to gracefully start and stop the database engine.

Other vCenter enhancements that Service Providers will make use of is the High availability feature which is something a lot of people have been asking for a long time. For me, I always dealt with the no HA constraint in that vCenter may become unavailable for 5-10 minutes during maintenance or at worse an extended outage while recovering from a VM or OS level failure. Knowing that hosts and VMs are still working and responding with vCenter down leaving only core management functionality unavailable it was a risk myself and others were willing to take. However, in this day of the always on datacenter it’s expected that management functionality be as available at IaaS services…so with that, this HA feature is well welcomed for Service Providers.

This native HA solution is available exclusively for the VCSA and the solution consists of active, passive, and witness nodes that are cloned from the existing vCenter Server instance. The HA cluster can be enabled, disabled, or destroyed at any time. There is also a maintenance mode that prevents planned maintenance from causing an unwanted failover.

The VCSA Migration Tool that was previously released in 6.0 Update 2m is shipped in the VCSA ISO and can be used to migrate from Windows based 5.5 vCenter’s to the 6.5 VCSA. Again this is something that more and more service providers will take advantage of as the reliance on Windows based vCenters and MSSQL becomes more and more something that’s unwanted from a manageability and cost point of view. Throw in the enhanced features that have only been released for the VCSA and this is a migration that all service providers should be planning.

To complete the move away from any Windows based dependencies the vSphere Update Manager has also been fully integrated into the VCSA. VUM is now fully integrated into the Web Client UI and is enabled by default. For larger environments with a large numbers of hosts AutoDeploy is now fully manageable from the VCSA UI and doesn’t require PowerCLI to manage or configure it’s options. There is a new image builder included in the UI that can hit local or public repositories to pull images or drivers and there are performance enhancements during deployments of ESXi images to hosts.

vCD and NSX Compatibility:

Shifting from new features and enhancements to an important subject to talk about when talking service provider platform…VMware product compatibility. For those vCAN Service Providers running a Hybrid Cloud you should be running a combination of vCloud Director SP or/and NSX-v of which, at the moment there is no support for either in vSphere 6.5. No compatible versions of NSX are available for vSphere 6.5. If you attempt to prepare your vSphere 6.5 hosts with NSX 6.2.x, you receive an error message and cannot proceed.

I haven’t tested to see if vCloud Director SP will connect and interact with vCenter 6.5 or ESXi 6.5 however as it’s not supported I wouldn’t suggest upgrading production IaaS platforms until the interoperability matrix’s are updated.

At this stage there is no word on when either product will support vSphere 6.5 but I suspect we will see NSX-v come out with a supported build shortly…though I’m expecting vCloud Director SP to no support 6.5 until the next major version release, which is looking like the new year.

Installation and Upgrade Known Issues:

Having read through the release notes, there are also a number of known issues you should be aware of. I’ve gone through those and pulled the ones I consider the most likely to be impactful to IaaS platforms.

  • After upgrading to vCenter Server 6.5, the ESXi hosts in High Availability clusters appear as Not Ready in the VMware NSX UI
    If your vSphere environment includes NSX and clusters configured with vSphere High Availability, after you upgrade to vCenter Server 6.5, both NSX and vSphere High Availability start installing VIBs on all hosts in the clusters. This might cause installation of NSX VIBs on some hosts to fail, and you see the hosts as Not Ready in the NSX UI.
    Workaround: Use the NSX UI to reinstall the VIBs.
  • Error 400 during attempt to log in to vCenter Server from the vSphere Web Client
    You log in to vCenter Server from the vSphere Web Client and log out. If, after 8 hours or more, you attempt to log in from the same browser tab, the following error results.
    400 An Error occurred from SSO. urn:oasis:names:tc:SAML:2.0:status:Requester, sub status:nullWorkaround: Close the browser or the browser tab and log in again.
  • Using storage rescan in environments with the large number of LUNs might cause unpredictable problems
    Storage rescan is an IO intensive operation. If you run it while performing other datastore management operation, such as creating or extending a datastore, you might experience delays and other problems. Problems are likely to occur in environments with the large number of LUNs, up to 1024, that are supported in the vSphere 6.5 release.Workaround: Typically, storage rescans that your hosts periodically perform are sufficient. You are not required to rescan storage when you perform the general datastore management tasks. Run storage rescans only when absolutely necessary, especially when your deployments include a large set of LUNs.
  • In vSphere 6.5, the name assigned to the iSCSI software adapter is different from the earlier releases
    After you upgrade to the vSphere 6.5 release, the name of the existing software iSCSI adapter, vmhbaXX, changes. This change affects any scripts that use hard-coded values for the name of the adapter. Because VMware does not guarantee that the adapter name remains the same across releases, you should not hard code the name in the scripts. The name change does not affect the behavior of the iSCSI software adapter.Workaround: None.
  • The bnx2x inbox driver that supports the QLogic NetXtreme II Network/iSCSI/FCoE adapter might cause problems in your ESXi environment
    Problems and errors occur when you disable or enable VMkernel ports and change the failover order of NICs for your iSCSI network setup.Workaround: Replace the bnx2x driver with an asynchronous driver. For information, see the VMware Web site.
  • When you use the Dell lsi_mr3 driver version 6.903.85.00-1OEM.600.0.0.2768847, you might encounter errors
    If you use the Dell lsi_mr3 asynchronous driver version 6.903.85.00-1OEM.600.0.0.2768847, the VMkernel logs might display the following message ScsiCore: 1806: Invalid sense buffer.Workaround: Replace the driver with the vSphere 6.5 inbox driver or an asynchronous driver from Broadcom.
  • Storage I/O Control settings are not honored per VMDK
    Storage I/O Control settings are not honored on a per VMDK basis. The VMDK settings are honored at the virtual machine level.Workaround: None.
  • Cannot create or clone a virtual machine on a SDRS-disabled datastore cluster
    This issue occurs when you select a datastore that is part of a SDRS-disabled datastore cluster in any of the New Virtual Machine, Clone Virtual Machine (to virtual machine or to template), or Deploy From Template wizards. When you arrive at the the Ready to Complete page and click Finish, the wizard remains open and nothing appears to occur. The Datastore value status for the virtual machine might display “Getting data…” and does not change.Workaround: Use the vSphere Web Client for placing virtual machines on SDRS-disabled datastore clusters.

These are just a few, that I have singled out…it’s worth reading through all the known issues just in case there are any specific issues that might impact you.

In the next post in this vSphere 6.5 for Service Providers series I will cover, more vCenter features as well as ESXi enhancements and what’s new in Core Storage.

References:

http://pubs.vmware.com/Release_Notes/en/vsphere/65/vsphere-esxi-vcenter-server-65-release-notes.html

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/vsphere/vmw-white-paper-vsphr-whats-new-6-5.pdf

http://pubs.vmware.com/Release_Notes/en/vsphere/65/vsphere-client-65-html5-functionality-support.html

Beta Participation Matters! – vSphere Beta Program

Over the past week there have been a number of posts around the new vSphere Beta which is the first step in testing the next major release from VMware following vSphere 6. As has become custom there is a private beta form that can be accessed here and people that are interested can fill out the form and register their intent to participate.

With all the issues that VMware have experienced over the past 12-18 months it’s massively important (I feel) that this beta is well represented and as many people as possible download the bits and put to the test the new vSphere platform. There is no doubt that this next release will be VMware’s most important when you think about the 5.5 and 6.0 issues as well as the perceived pressure being heaped by Nutanix and…to a less extent Microsoft with Hyper-V.

VMware need to nail .NEXT!

I say this because the one thing that VMware need to combat AHV, Hyper-V and other hypervisors out their is a return to core platform stability and that can be further achieved if there are enough people testing and then reporting back to the VMware beta teams about their issues…the more diverse the beta base is the great the exposure to potential issues and bugs. This isn’t a guarantee that the perception of reduced stability and increased bugs won’t be totally eradicated but it goes some way to helping.

If selected to participate in the beta there are a set of expectations that people need to commit to.

Participant Expectations:

  • Online acceptance of the Master Software Beta Test Agreement will be required prior to visiting the Private Beta Community
  • Install beta software within 3 days of receiving access to the beta product
  • Provide feedback within the first 4 weeks of the beta program
  • Submit Support Requests for bugs, issues and feature requests
  • Complete surveys and beta test assignments
  • Participate in the private beta discussion forum and conference calls

I highlighted the dot point above relating to participants being active when part of a beta program. If you are just wanting to download the bits and install them for a quick look then you are probably not going to get anything substantial out of a beta program. One of the key reasons they exist is to generate feedback and testing on software thats not yet feature set and potentially has undiscovered bugs. Computer game companies have of recent times been putting out open betas (such as the recent one for the new Doom) to put their software through harsh testing at the hands of their potential customer base…this is no different to what betas such as the vSphere Beta.

When I participate in Beta’s I know that I am helping to shape the future of the product…people that know me know that I am a bit of a “beta whore” but that’s only because I understand the benfits of being involved in the programs and understand that it’s not only important for the vendor…but also important for the customer…after all you are getting a look at what’s next and essentially get to contribute in the final release.

vSphere 6 Beta Details:

This program enables participants to help define the direction of the most widely adopted industry-leading virtualization platform. Folks who want to participate in the program can now indicate their interest by filling out this simple form. The vSphere team will grant access to the program to selected candidates in stages. This vSphere Beta Program leverages a private Beta community to download software and share information.

 

We will provide discussion forums, webinars, and service requests to enable you to share your feedback with us.

You can expect to download, install, and test vSphere Beta software in your environment or get invited to try new features in a VMware hosted environment. All testing is free-form and we encourage you to use our software in ways that interest you. This will provide us with valuable insight into how you use vSphere in real-world conditions and with real-world test cases, enabling us to better align our product with your business needs.

So if you want to contribute to the future of vSphere…register for the BETA and be active in your participation!

References:

http://info.vmware.com/content/35853_VMware-vSphere-Beta_Interest

vSphere 6 Update 2 – Whats In It for Service Providers

It’s been just over a week since VMware released vSphere 6 Update 2 and I thought I would go through some of the key features and fixes that are included in the latest versions of vCenter and ESXi. As usual I generally keep an eye out for improvements that relate back to Service Providers who use vSphere as the foundation of their Managed or Infrastructure as as Service offerings.

New Features:

Without question the biggest new feature is the release of VSAN 6.2. I’ve covered this release in previous blog posts and when you upgrade to ESXi 6.0 Update 2 the VSAN 6.2 bits are present within the kernel. Some VSAN services are actually in play regardless if you have it configured or not…which is interesting. With the new pricing for VSAN through the vCAN program, Service Providers now can seriously think about deploying VSAN for their main IaaS platforms.

The addition of support for High Speed Ethernet Links is significant not only because of the addition of 25G and 50G link speeds means increased throughput for converged network cards allowing for more network traffic to flow through hosts and switches for Fault Tolerance, vMotion, Storage vMotions and storage traffic but also because it allows SPs to think about building Edge Clusters for networking services such as NSX and allow the line speeds to take advantage of even higher backends.

From a manageability point of view the Host Client HTML5 user interface is a welcome addition and hopefully paves the way for more HTML5 management goodness from VMware for not only hosts…but also vCenter its self. There is a fair bit of power already in the Host Client and I can bet that admins will start to use it more and more as it continues to evolve.

For vCenter the addition of Two-Factor Authentication using RSA or Smartcard technology is an important feature for SPs to use if they are considering any sort of certification for their services. For example many government based certifications such as IRAP require this to be certified.

Resolved Issues:

There are a bunch of resolved issues in this build and I’ve gone through the rather extensive list to pull out the biggest fixes that relate to my experience in service provider operations.

vCenter:

  • Upgrading vCenter Server from 5.5 Update 3b to 6.0 Update 1b might fail if SSLv3 is disabled on port 7444 of vCenter Server 5.5 Update 3b. An upgrade from vCenter Server 5.5 Update 3b to 6.0 Update 2 works fine if SSLv3 is disabled by default on 7444 port of vCenter Server 5.5 Update 3b.
  • Deploying a vApp on vCloud Director through the vApp template fails with a Profile-Driven storage error. When you refresh the storage policy, an error message similar to the following is displayed: The entity vCenter Server is busy completing an operation.
  • Delta disk names of the source VM are retained in the disk names of the cloned VM. When you create a hot clone of a VM that has one or more snapshots, the delta disk names of the source VM are retained in the cloned VM
  • vCenter Server service (vpxd) might fail during a virtual machine power on operation in a Distributed Resource Scheduler (DRS) cluster.

ESXi:

  • Hostd might stop responding when you execute esxcli commands using PowerCLI resulting in memory leaks and memory consumption exceeding the hard limit.
  • ESXi mClock I/O scheduler does not work as expected. The ESXi mClock I/O scheduler does not limit the I/Os with a lesser load even after you change the IOPS of the VM using the vSphere Client.
  • After you upgrade Virtual SAN environment to ESXi 6.0 Update 1b, the vCenter Server reports a false warning similar to the following in the Summary tab in the vSphere Web Client and the ESXi host shows a notification triangle
  • Attempts to perform vMotion might fail after you upgrade from ESXi 5.0 or 5.1 to 6.0 Update 1 or later releases. An error message similar to the following is written to the vmware.log file.
  • Virtual machine performance metrics are not displayed correctly as the performance counter cpu.system.summation for a VM is always displayed as 0
  • Attempts to perform vMotion with ESXi 6.0 virtual machines that have two 2 TB virtual disks created on ESXi 5.0 fail with an error messages similar to the following logged in the vpxd.log file:2015-09-28T10:00:28.721+09:00 info vpxd[xxxxx] [[email protected] sub=vpxLro opID=xxxxxxxx-xxxxxxxx-xx] [VpxLRO] — BEGIN task-919 — vm-281 — vim.VirtualMachine.relocate — xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

The mClock fix highlighted above is a significant fix for those that where looking to use IOPS limiting. It’s basically been broken since 5.5 Update 2 and also impacts/changes the way in which you would think IOPS are interpreted through the VM to storage stack. For service providers looking to introduce IOPS limited to control the impact noisy neighbors the fix is welcomed.

As usual there are still a lot of known issues and some that have been added or updated to the release notes since release date. Overall the early noise coming out from the community is that this Update 2 release is relatively solid and there have been improvements in network performance and general overall stability. Hopefully we don’t see a repeat of the 5.5 Update 2 issues or the more recent bug problems that have plagued previous released…and hopefully not more CBT issues!

vSphere 6.0 Update 2 has a lot of goodness for Service Providers and continues of offer the number one vitalization platform from which to build managed and hosted services on top of. Go grab it now and put it through it’s paces before pushing to production!

References:

http://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-esxi-60u2-release-notes.html

http://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-vcenter-server-60u2-release-notes.html

http://pubs.vmware.com/Release_Notes/en/vsan/62/vmware-virtual-san-62-release-notes.html

http://pubs.vmware.com/Release_Notes/en/vsphere/60/vmware-host-client-10-release-notes.html

vSphere 5.5 Update 3 Released: Features and Top Fixes

vSphere 5.5 Update 3 was released earlier today and there are a bunch of bug fixes and feature improvements in this update release for both vCenter and ESXi. For most Service Providers updating to vSphere 6.0 is still a while away so it’s good to have continued support and improvement for the 5.5 platform. I’ve scanned through the release notes and picked out what I consider some of the more important bug fixes and resolved issues as they pertain to my deployments of vSphere.

Note: Still appears that there is no resolution to the vMotion errors I reported on earlier in the year or the bugs around the mClock Scheduler and IOPS Limiter on NFS.

ESXi 5.5 Update 3:

  • Status of some disks might be displayed as UNCONFIGURED GOOD instead of ONLINEStatus of some disks on an ESXi 5.5 host might be displayed as UNCONFIGURED GOOD instead of ONLINE. This issue occurs for LSI controller using the LSI CIM provider.
  • Cloning CBT-enabled virtual machine templates from ESXi hosts might failAttempt to clone CBT-enabled virtual machines templates simultaneously from two different ESXi 5.5 hosts might fail. An error message similar to the following is displayed:Failed to open VM_template.vmdk': Could not open/create change tracking file (2108).
  • ESXi hosts with the virtual machines having e1000 or e1000e vNIC driver might fail with a purple screenESXi hosts with the virtual machines having e1000 or e1000e vNIC driver might fail with a purple screen when you enable TCP segmentation Offload (TSO). Error messages similar to the following might be written to the log files:cpu7:nnnnnn)Code start: 0xnnnnnnnnnnnn VMK uptime: 9:21:12:17.991 cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x65b stack: 0xnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x18ab stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0xa2 stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0xae stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x488 stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x60 stack: 0xnnnnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x185 stack: 0xnnnnnnnnnnnn
  • Attempts to reboot Windows 8 and Windows 2012 server on ESXi host virtual machines might failAfter you reboot, the Windows 8 and Windows 2012 Server virtual machines might become unresponsive when the Microsoft Windows boot splash screen appears. For more information refer, Knowledge Base article 2092807.
  • Attempts to install or upgrade VMware Tools on a Solaris 10 Update 3 virtual machine might fail
    Attempts to install or upgrade VMware Tools on a Solaris 10 Update 3 virtual machine might fail with the following error message:Detected X version 6.9
    Could not read /usr/lib/vmware-tools/configurator/XOrg/7.0/vmwlegacy_drv.so Execution aborted.This issue occurs if the vmware-config-tools.pl script copies the vmwlegacy_drv.so file, which should not be used in Xorg 6.9.

In going through the remaining Known Issues you come across a lot of Flash Read Cache related problems…maybe VMware should call it a day with this feature…not sure if anyone has the balls to actually use it in production…be interested to hear? There are also a lot of VSAN issues still being reported as known with workarounds in place…all the more reason to start a VSAN journey with vSphere 6.0.

For a look at what’s new and for the release notes in full…click on the links below:

VMware ESXi™ 5.5 Update 3 | 16 SEP 2015 | Build 3029944

vCenter Server 5.5 Update 3 | 16 SEP 2015 | Build 3000241

NSX Bytes: 6.1.3 Upgrade and vSphere 6.0

Earlier today NSX-v 6.1.3 was released. This update brings vSphere 6.0 Support as well as bug fixes and a couple minor feature enhancements.

https://www.vmware.com/support/nsx/doc/releasenotes_nsx_vsphere_613.html 

There is a warning in the release notes:

NSX vSphere 6.1.3 is compatible with vSphere 6.0. However, the new vSphere features introduced in vSphere 6.0 have not been tested with NSX vSphere. These new vSphere features should not be used in environments where NSX vSphere is installed as they are unsupported. For a list of specific NSX vSphere limitations with respect to vSphere 6.0, see the VMware Knowledge Base article 2110197.

Read through the above KB and make sure you understand what can and can’t be done when running NSX-v on vSphere 6.0…there are some significant constraints in terms of using some of the new shiny bits of vSphere 6.0 …some of which I see as being potential matches made in heaven for NSX-v Hybrid Solutions with vSphere based Platforms.

  • While vSphere 6.0 adds support for vMotion over larger network distances (where the RTT is up to 150ms), this has not been tested in deployments that use NSX-v
  • Multi-vCenter vMotion
  • Deploying NSX-v components from a vSphere Multisite Content Library has not been tested and is not supported.
  • The following improvements introduced in vSphere 6.0 have not been tested with NSX-v: vSphere Fault Tolerance improvements and vSphere Replication improvements

Hopefully the next Major Release of NSX-v will take advantage of the new features above, but for now at least we have the ability to have NSX-v running with vCenter and ESXi 6.0.

With regards to the upgrade process, nothing has changed in regards to the steps listed in my previous NSX Upgrade Posts:

NSX components must be upgraded in the following order:

  1. NSX Manager
  2. NSX controller
  3. Clusters and Logical Switches
  4. NSX Edge and Guest Introspection

Build Numbers shown above and Below for Manager and Controllers

After NSX has been upgraded as above, you can Upgrade vCenter Server to 6.0 (if running the VCSA 5.5 Appliance read here for Upgrade Details) from there you can upgrade your ESXi hosts to 6.0…upon reboot the NSX Manager pushes the ESXi 6.0 VIBs to the hosts. When the hosts show reboot required on the Hosts and Clusters tab in the left hand side of the vSphere Web Client, reboot the hosts a second time and the NSX VIBs for ESXi 6.0 are now enabled.

vSphere 6.0 vCenter Server Appliance: Upgrading from 5.x

Like most VMware Junkies over the past 24 hours I’ve downloaded the vSphere 6.0 bits and had them ready and primed to deploy and discover all the new features. Given the nature of a .0 release (not withstanding Change Control) upgrading production systems won’t be happening any time soon…however I had an opportunity right away to upgrade the Zettagrid Lab VCSA 5.5 Appliance to the 6.0 Appliance.

No word if this upgrade is possible via the old school method shown above, however the basic run through of the upgrade process is that the new VCSA is deployed and then imports data and settings from the existing VCSA while living on temporary network settings. It then shuts down the old VSCA and assumes it’s IP Address.

Working through the Upgrade Process, you need to download VMware-VCSA-all-6.0.0-2562643.iso and mount the VCSA ISO (Thanks @grantorchard) and from there double click on vcsa-setup.htm.

If you haven’t installed the 6.0 Client Integration Plugin located in the \vcsa folder of the ISO close down the browser and run that.

Rerun the vcsa-setup.htm and you should get the Install/Upgrade page…for me in Chrome I have to allow an External Protocol Request.

From there you have the option to Install or Upgrade to VCSA 6.0. This is an Upgrade that was chosen at which point the installed kicks off and gives you some details on what version of the VCSA you can upgrade from.

Clicking on Ok we are now presented with the VCSA Deployment Menu. First step is to enter in a target ESXi Host…not a vCenter like I first did to get the error below!

Moving through the Steps, we enter in a Virtual Machine Name and then specify the Source Appliance details which again is all host based.

Set the Appliance Size

Select the Datastore and then move onto the Temporary Network Settings…the explanation of what’s happening can be seen in the description below

Once all the settings are reviewed click on finish and the installer does all the work. For me this took about 20-30 minutes…bearing in mind that this vCenter was not large so import times will vary. You do have the option to bring across all existing Statistics…checking that option will vary the install time depending on how much data you have.

NOTE: My install progress stalled on Start the vCenter Web Client but that was more likely due to me not following instructions. When you select your target ESXi Host it states that DRS should be switched to Manual during the install process…my assumption is that the the VCSA was brought up on a different host other than the one specified and the installer lost track of the VM and wasn’t able to confirm the completion.

Regardless of that, I was able to log into the Web Client and ensure all Services where in tact and that the upgrade was successful…however I did start to see some strange behaviours and login issues over the weekend. I rolled back and went through the upgrade again, this time ensuring DRS was set to Manual…and had a 100% successful outcome.

I now can enjoy the benefits of the Web Client and start to get a feel for vSphere 6.0.

http://www.vmware.com/files/pdf/vsphere/VMware-vSphere-Platform-Whats-New.pdf

https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html

vSphere 6.0 Launch: What’s in it for Service Providers #vmw28days

Today vSphere 6.0 was officially announced and will be GA in about 6-8 weeks…I’ve had limited time myself to tinker with the BETA in great depth, however I have been keeping a close eye on some of the key features being released in the v6.0 edition. I’ve gone through and pulled out the top new features/enhancements as I think they relate to vCloud Air Network Ecosystem Partners and how SPs can look at enhancing an already strong IaaS platform with vSphere 6.x

Enhanced Fault Tolerance:

This has always been cool, but impracticable given the vCPU limitations placed on previous iterations, however with FT in v6.0 there is now support for VMs up to 4 vCPUs

  • Enhanced virtual disk format support for thin and thick disks
  • Ability to hot configure FT
  • Backup support with snapshots
  • Uses copies of VMDKs for added storage redundancy

There are a couple of benefits for SPs with enhanced FT…first and foremost it offers a way to protect vCenter and other critical management appliances like an NSX Manager Appliance or SQL Server without the need to rely on MS Clustering. There is also a case to look to productize VM FT and offer it as a check box feature for clients…while not completely straightforward (thinking about vCD/CMP Awareness) there is a serious value add to be exploited here.

Virtual Volumes (VVOLS) and Enhanced Storage APIs:

Storage is the number one pain point for any SP with scaleability and reliability all factors in storage pain driven by VM Latency. VVOLS are a logical extension of virtualization into the storage layer and offer policy based management of storage on per VM basis. In theory this will eliminate LUN management as every VM is treated as it’s own object with attached policy based management that leverages enhanced VASA APIs to map storage to policies/capabilities. All major storage vendors are looking at offering VVOL support…for SPs it should be top of the list when looking at any new storage platform from today on wards.

Enhanced vMotion and Hybrid Cloud Support:

vMotion has been given a facelift in v6.0 and can now do Cross vSwitch, Cross vCenter, Long Distance and vMotion across L3 boundaries. This opens the door for SPs to offer greater flexibility for On-Premises to Cloud Migrations, Cloud to Cloud Migrations and SP intra zone movement of VMs…all without any downtime! This is truly amazing tech that we probably take for granted these days, but make no mistake there is something very special about being to move a VM from one location to another on the fly.

vCenter also now has more direct path through to vCloud Air and vCloud Air Network Partners (still to be 100% with vCloud SP 5.6.x) by way of the Hybrid Cloud Plugin. Still to confirm if this will allow for vMotion between on-premises and hosted, but the future is looking promising for Hybrid deployments when you add in the features of NSX Gateway Services into the mix.

vCenter Scalability Enhancements:

  • 64 Hosts per cluster
  • 8000 VMs per cluster
  • Up to 1000 VMs per Host
  • 10,000 VMs per vCenter

I’m not one to push configuration maximums but extending vCenters capability to manage more Host’s per Cluster (up from 32) and more VMs per host and cluster will give Service Providers greater opportunity to drive higher host/VM density and work on larger pools of compute to abstract and offer tenants.

The vCenter Appliance is now pound for pound with the Windows version which means SP’s can now seriously consider replacing/upgrading to the appliance while still having the ability to scale out VC roles and even enable Linked Mode…if required.

There are other vCenter enhancements that lay the foundation for even more scale out functionality with the addition of Platform Services and more complex deployment scenarios.

vCenter Content Library:

The Content Library is a Repository for VM templates, ISOs and vApps. You can allow content to be stored in one location and replicated out to other vCenters. This is handy for Service Providers than run multiple zones with separate vCenters and allows all content to be sync’ed between different hosting locations. Also has the added benefit of keeping vCloud Director Catalog’s up to date and consistent across the board.

Conclusion:

Many people have said that VMware’s time is numbered and that the hypervisor has become irrelevant…those that believe either those points of views in my opinion are greatly mistaken and need to pay close attention to the list of feature enhancements announced today.

There is no doubt in my mind that VMware’s vSphere and ESXi will continue to hold a significant advantage when looking at competing hypervisor platforms…this release proves that VMware are serious about continuing to power Virtual Platforms of all sizes as people transition even further towards the Hybrid Cloud while maintaining operational efficiency and stability though the most mature and proven hypervisor on the market.

A number of vExperts where featured on the VMware Blog Page and there are a number of great articles going through features in depth.

http://blogs.vmware.com/vmtn/2015/02/vsphere-6-0-vexpert-blog-articles.html

FT Ref: http://www.vmwarearena.com/2015/02/vsphere-6-0-whats-new-in-vmware-fault-tolerance-ft.html 

Feature List Ref: http://ecktech.me/new-features-vsphere-6/

Tintri VVOLs: http://www.theregister.co.uk/2015/02/03/tintri_smooching_vmwares_vvols_vio_and_vrops/

vMotions Enhancements: http://www.vmguru.com/2015/02/whats-new-vsphere-6-vmotion-enhancements/

vCenter Roles and Design: http://vinfrastructure.it/2015/02/vmware-vcenter-server-6-design/