Tag Archives: Upgrade

Upgrading Windows vCenter 5.5 to 6.0 In-Place: Issues and Fixes

Yes that’s not a typo…this post is focusing on upgrading Windows vCenter 5.5 to 6.0 via an in-place upgrade. There is the option to use the vSphere 6.0 Update2M build with the included Migrate to VCSA tool to achieve this and move away from Windows, but I thought it was worth documenting my experiences with a mature vCenter that’s at version 5.5 Update 2 and upgrade that to 6.0 Update 2. Eventually this vCenter will need to move off the current Windows 2008 RTM server which will bring into play the VCSA migration however for the moment it’s going to be upgraded to 6.0 on the same server.

With VMware releasing vSphere 6.5 in November there should be an increased desire for IT shops to start seriously thinking about moving on from there existing vSphere versions and upgrading to the latest 6.5 release however many people I know where still running vSphere 5.5, so the jump to 6.5 directly might not be possible due to internal policies or other business reasons. Interestingly in the rough numbers, I’ve got an active Twitter Poll out at the moment which after 100 votes shows that vSphere 5.5 makes up 53% of the most common vCenter version, followed by 6.0 with 44% and 6.5 with only 3%.

Upgrade Options:

You basically have two options to upgrade a Windows based 5.5 vCenter:

My approach for this particular environment (which is a NestedESXi lab environment) was to ensure a smooth upgrade to vSphere 6.0 Update 2 and then look to upgrade again to 6.5 once is thaws outs in the market. That said, I haven’t read too many issues with vSphere 6.5 and VMware have been excellent in ensuring that the 6.5 release was the most stable for years. The cautious approach will still be undertaken by many and a stepped upgrade to 6.5 and migration to the VCSA will be common place. For those that wish to move away from their Windows vCenter, there is nothing stopping you from going down the Migrate2VCSA path, and it is possible to migrate directly from 5.5 to 6.5.

Existing Component Versions:

  • vCenter 5.5 (2001466)
  • ESXi 5.5 (3116895)

SQL Version Requirements:

vCenter 6.0 Update 2 requires at least SQL Server 2008 R2 SP1 or higher, so if you are running anything lower than that you will need to upgrade to a later service pack or upgrade to later versions of SQL Server. For a list of all compatible databases click here.

vCenter Upgrade Pre-Upgrade Checks:

First step is to make sure you have a backup of the vCenter environment meaning VM state (Snapshot) and vCenter database backup. Once that’s done there are a few pre-requisites that need to be met and that will be checked by the upgrade process before the actual upgrade occurs. The first thing the installer will do after asking for the SSO and VC service account password is run the Pre-Upgrade Checker.

vCenter SSL and SSO SSL System Name Mismatch Error:

A common issue that may pop up from the pre-upgrade checker is the warning below talking about an issue with the system name of the vCenter Server certificate and the SSO certificate. As shown below it’s a hard stop and tells you to replace one or the other certificate so that the same system name is used.

If you have a publicly signed SSL Certificate you will need to generate a new cert request and submit that through the public authority of choice. The quickest way to achieve this for me was to generate a new self signed certificate by following the VMwareKB article here. Once that’s been generated you can replace the existing certificate by following a previous post I did using the VMware SSL Certificate Updater Tool.

After all that, in any case I got the warning below saying that the 5.5 SSL Certificates do not meet security requirements, and so new SSL certificates will need to be generated for vCenter Server 6.0.0.

With that, my suggestion would be to generate a temporary self signed certificate for the upgrade and then apply a public certificate after that’s completed.

Ephemeral TCP Port Error:

Once the SSL mismatch error has been sorted you can run the pre-upgrade checker again. Once that completes successfully you move onto the Configure Ports window. I ran into the error shown below that states that the range of port is too large and the system must be reconfigured to use a smaller ephemeral port range before the install can continue.

The fix is presented in the error message so after running netsh.exe int ipv4 set dynamicportrange tcp 49152 16384 you should be ok to hit Next again and continue the upgrade.

Export of 5.x Data:

During the upgrade the 5.5 data is stored in a directory and then migrated to 6.0. You need to ensure that you have enough room on the drive location to cater for your vCenter instance. While I haven’t seen any offical rules around the storage required, I would suggest having enough storage free and the size of your vCenter SQL database data file.

vCenter Upgrade:

Once you have worked through all the upgrade screens you are ready for upgrade. Confirm the settings, take note of the fact that once updated the vCenter will be in evaluation mode, meaning you need to apply a new vCenter 6.x license once completed, check the checkbox that states you have a backup of the vCenter machine and database and you should be good to go.

Depending on the size of you vCenter instance and the speed of your disks the upgrade can take anywhere from 30 to 60 minutes or longer. If at any time the upgrade process fails during the initial export of the 5.5 data a roll back via the installer is possible…however if there is an issue while 6.0 is being installed the likelihood is that you will need to recover from backups.

Post Upgrade Checks:

Apart from making sure that the upgrade has gone through smoothly by ensuring all core vCenter services are up and running, it’s important to check any VMware or third party services that where registered against the vCenter especially given that the SSL Certificate has been replaced a couple of times. Server applications like NSX-v, vCloud Director and vCO explicitly trust SSL certificates so the registration needs to be actioned again. Also if you are running Veeam Backup & Replication you will need to go through the setup process again to accept the new SSL Certificate otherwise your backup jobs will fail.

If everything has gone as expected you will have a functional vCenter 6.0 Update 2 instance and planning can now take place for the 6.5 upgrade and in my case…the migration from Windows to the VCSA.





VSAN Upgrading From 6.1 To 6.2 Hybrid To All Flash – Part 2

When VSAN 6.2 was released earlier this year it came with new and enhanced features and with the price of SSDs continuing to fall and an expanding HCL it seems like All Flash instances are becoming more the norm and for those that have already deployed VSAN in a Hybrid configuration the temptation to upgrade to All Flash is certainly there. Duncan Epping has previously blogged the overview of migrating from Hybrid to All Flash so I wanted to expand on that post and go through the process in a little more detail. This is part two of what is now a three part blog series with the process overview outlined below.

Use the links below to page jump.

In part one I covered upgrading existing hosts, expanding an existing VSAN cluster and upgrading the license and disk format. In this part am going to go through the simple task of extending the cluster by adding new All Flash Disk Groups on the host I added in part one and then go through the actual Hybrid to All Flash migration steps.

The configuration of the VSAN Cluster after the upgrade will be:

  • Four Host Cluster
  • vCenter 6.0.0 Update 2
  • ESXi 6.0.0 Update 2
  • One Disk Groups Per Host
  • 1x 480GB SSD Cache and 2x 1000GB SSD Capacity
  • VSAN Erasure Coding Raid 5 FTT=1
  • DeDuplication and Compression On

As mentioned in part one I added a new host to the cluster in order to give me some breathing room while doing the Hybrid to All Flash upgrade as we need to perform rolling maintenance on each hosts in the cluster in order to get to the All Flash configuration. Each host will be entered into maintenance mode and all data evacuated. Before the process is started on the initial three hosts lets go ahead and create a new All Flash Disk Group on the new hosts.

To create the new Disk Group head to Disk Management under the Virtual SAN section of the Manage Tab whilst the Cluster and click on the Create New Disk Group Button. As you can see below I have the option of selecting any of the flash devices claimed as being ok for VSAN.

After the disk selection is made and the disk group created, you can see below that there is now a mixed mode scenario happening where the All Flash host is participating in the VSAN Cluster and contributing to the capacity.

Upgrade Disk Group from Hybrid to All Flash:

Ok, now that there is some extra headroom the process to migrate the existing Hybrid Hosts over to All Flash can begin. Essentially what the process involves is placing the hosts in maintenance mode with a full data migration, deleting any existing Hybrid disk groups, removing the spinning disk, replacing them with flash and then finally creating new All Flash disk groups.

If you are not already aware about maintenance mode with VSAN then it’s worth reading over this VMware Blog Post to ensure you understand that using the VI Client is a big no no. In this case I wanted to do a full data migration which moves all VSAN components onto remaining hosts active in the cluster.

You can track this process by looking at the Resyncing Components section of the Virtual SAN Monitor Tab to see which objects are being copied to other hosts.

As you can see the new host is actively participating in the Hybrid mixed mode cluster now and taking objects.

Once the copy evacuation has completed we can now delete the existing disk groups on the host by highlights the disk group and clicking on the Remove Disk Group button. A warning appears telling us that data will be deleted and also lets us know how much data is currently on the disks. The previous step has ensured that there should be no data on the disk group and it should be safe to (still) select Full data migration and remove the disk group.

Do this for all existing Hybrid disk groups and once all disk groups have been deleted from the host you are ready to remove the existing spinning disks and replace them with flash disks. The only thing to ensure before attempting to claim the new SSDs is that they don’t have any previous partitions on them…if so you can use the ESXi Embedded Host Client to remove any existing partitions.

Warning: Again it’s worth mentioning that any full data data migration is going to take a fair amount of time depending on the consumed storage of your disk groups and the types of disks being used.

Repeat this process on all remaining hosts in the cluster with Hybrid disk groups until you have a full All Flash cluster as shown above. From here we are now able to take advantage of erasure coding, DeDuplication and compression…I will finish that off in part three of this series.


VSAN Upgrading from 6.1 to 6.2 Hybrid to All Flash – Part 1

When VSAN 6.2 was released earlier this year it came with new and enhanced features and depending on what version you where running you might not have been able to take advantage of them all right away. Across all versions, Software Checksum was added with Advanced and Enterprise versions getting VSANs implementation of Erasure Coding (RAID 5/6) with Deduplication and Compression available for the All Flash version and QOS IOPS Limiting available in Enterprise only.

With the price of SSDs continuing to fall and an expanding HCL it seems like All Flash instances are becoming more the norm and for those that have already deployed VSAN in a Hybrid configuration the temptation to upgrade to All Flash is certainly there. Duncan Epping has previously blogged the overview of migrating from Hybrid to All Flash so I wanted to expand on that post and go through the process in a little more detail. This is a two part blog post with a lot of screen shots to compliment the process which is outlined below.

Use the links below to page jump.

Warning: Before I begin it’s worth mentioning that this is not a short process so make sure you plan this out relative to the existing size of your VSAN cluster. In talking with other people who have gone through the disk format upgrade the average rate seems to be about 10TB of consumed data per day depending on the type of disks being used. I’ll reference some posts at the end that relates to the disk upgrade process as it has been troublesome for some however also worth pointing out that the upgrade process is non disruptive for running workloads.

Existing Configuration:

  • Three Host Cluster
  • vCenter 6.0.0 Update 2
  • ESXi 6.0.0 Update 1
  • Two Disk Groups Per Host
  • 1x 200GB SSD and 2x 600GB HDD
  • VSAN Default Policy FTT=1

Upgrade Existing Hosts to 6.0 Update 2:

At the time of writing ESXi 6.0.0 Update 2 is the latest release and the builds that contain the VSAN 6.2 codebase. From the official VMware Upgrade matrix it seems you can’t upgrade from VSAN versions older than 6.1, so if you are on 5.x or 6.0 releases you will need to take note of this VMwareKB to get to ESXI 6.0.0 Update 2. A great resource for the latest builds as well as links to upgrade from head here:


For a quick upgrade directly from the VMware online host update repository you can do the following on each host in the cluster after putting them into VSAN Maintenance Mode. Note that there are also some advanced settings that are recommended as part of the VSAN Health Checks in 6.2

After rolling through each host in the cluster make sure that you have an updated copy of the VSAN HCL and run a health check to see where you stand. You should see a warning about the disks needing an upgrade and if any hosts didn’t have the above advanced settings applied you will have a warning about that as well.

Expanding VSAN Cluster:

As part of this upgrade I am also adding an additional host to the existing three to expand to a four host cluster. I am doing this for a couple of reasons, not withstanding the accepted design position on four host being better than three from a data availability point of view you also need a minimum of four hosts if you want to enable RAID5 erasure coding (six is required as a minimum for RAID6). The addition of the fourth host also allowed me to roll through the Hybrid to AF upgrade with a lot more headroom.

Before adding the new host to the existing cluster you need to ensure that the build is consistent with the existing hosts in terms of versioning and more importantly networking. Ensure that you have configured an VMkernel Interface for VSAN traffic and marked it as such through the Web Client. If you don’t do this prior to putting the host into the existing cluster I found that the management VMKernel interface was enabled by default for VSAN.

If you notice below this cluster is also NSX enabled, hence the events relating to Virtual NICs being added. Most importantly the host can see other hosts in the cluster and is enabled for HA.

Once in the cluster the host can be used for VM placement with data served from the existing hosts with configured disk groups over the VSAN network.

Upgrade License:

At this point I upgraded the licenses to enable the new features in VSAN 6.2. As a refresher on VSAN licensing there are three editions with the biggest change from previous versions being that to get the Deduplication and Compression, Erasure Coding and QoS features you need to be running All Flash and have an Enterprise license key.

To upgrade the license you need to head to Licensing under the Configuration section of the Manage Tab whilst the Cluster is selected. Apply the new license and you should see the following.

Upgrade Disk Format:

If you have read up around upgrading VSAN you know that there is a disk format upgrade required to get the benefits of the newer versions. Once you have upgraded both vCenter and Hosts to 6.0.0 Update 2 if you check the VSAN Health under the Monitor Tab of the Cluster you should see an failure talking about v2 disks not working with v3 disks as shown below.

You can click on the Upgrade On-Disk Format button here to kick off the process. This can also be triggered from the Disk Management section under the Virtual San menu in the Manage cluster section of the Web Client. Once triggered you will see some events trigger and an update in progress message near the version number.

Borrowing from one of Cormac Hogan’s posts on VSAN 6.2 the following explains what is happening during the disk format upgrade. Also described in the blog post is a way using the Ruby vSphere Client to monitor the progress in more detail.

There are a few sub-steps involved in the on-disk format upgrade. First, there is the realignment of all objects to a 1MB address space. Next, all vsanSparse objects (typically used by snapshots) are aligned to a 4KB boundary. This will bring all objects to version 2.5 (an interim version) and readies them for the on-disk format upgrade to V3. Finally, there is the evacuation of components from a disk groups, then the deletion of said disk group and finally the recreation of the disk group as a V3. This process is then repeated for each disk group in the cluster, until finally all disks are at V3.

As explained above the upgrade can take a significant amount of time depending on the amount of disk groups, data consumed on your VSAN datastore as well as the type of disks being used (SAS based vs SATA/NL-SAS) Once complete you should have a green tick and the On-Disk format version reporting 3.0

With that done we can move ahead to the Hybrid to All Flash conversion. For details on the look out for Part 2 of this series coming soon.


Hybrid vs All-flash VSAN, are we really getting close?

VSAN 6.2 Part 2 – RAID-5 and RAID-6 configurations

VSAN 6.2 Part 12 – VSAN 6.1 to 6.2 Upgrade Steps

vCloud Director 5.5 Upgrade: Database Upgrade Issues

Generally speaking upgrading the vCloud Director binaries is a pretty straight forward task. It’s a two step process where the Cell’s are upgraded first followed by some update scripts run against the vCloud database. Finally database Indexes and Statistics are updated. In the last 6-8 months I’ve worked through 1.5 -> 5.1 and recently 5.1 -> 5.5 upgrades…as well as point releases and BETA builds in the lab…all of those upgrades have ultimately been successful, but as recently as last week I’ve come across a couple of issues when running the update database schema script. The second issue is of more significance as there is little reference to it on Google.

Issue 1:

This one is apparently fairly common, and doesn’t need too much concern.

The fix is straight forward and is explained in this post, but the workaround is to simply log into your SQL server and execute the stored procedure as instructed in the warning message from the upgrade script. Done, nice and easy!

Issue 2:

After the database upgrade tasks have been completed you are asked if you want to rebuild the database indexes and that it may take several minutes. I’ve found in the lab and in production this can take 1-2 minutes, however during our latest upgrade we received the message below

Even though this is listed as an extra step, we all know how critical indexes are to efficient database query execution so I wanted to try and get the rebuild done. The vCD database in question isn’t overly large (about 3-4GB) and suspecting a transient SQL connectivity error I backed up the database again and ran the upgrade database script once again…this is safe to do and it’s smart enough to work out that the database update schema tasks have been committed…however I received the same error. I didn’t want to blanket rebuild all Indexes and suspected the rebuild to be a little more targeted.

Thinking back to my old days of troubleshooting dodgy programmer queries I decided to fire up MSQL Management Studio and ran the Activity Monitor and filtered out for the vCD database. I ran the upgrade script again and was able to see the queries being run via the monitor tab. Right clicking on the suspected query I was able to view what was being run:

Against the vCD Database I executed the specific query as show below and it completed successfully in a tick under 3 minutes. There was a noticeable pause in the output messages about 2/3 the way through so my thinking is that the upgrade script has a timeout that was reached due to inactivity.

From here we where able to fire up the Cell and continue with the upgrade process. Again, this issue seems to be rare, but hopefully if anyone comes across it the fix is as straight forward as the update statistics fix.

vCenter SSO 5.5 Backwards Compatability: Mixed Mode Install

While at a post #vFourmAU event last week a group of us where talking about SSO in vCenter 5.1 and what a disaster it had been (credibility wise) and that VMware had done their best to improve the experience in the 5.5 release. Out of that conversation came word that if people where looking to upgrade from 4.x or 5.0 to 5.1 as an interim step to 5.5, you could install SSO 5.5 in that 5.1 Environment so as not to go through the relative pain that SSO 5.1 brings to the table…that is to say SSO 5.5 was backwards compatible.


That post above goes through a little background and case examples where it might be relevant to run SSO 5.5 mixed in with vCenter5.1.

While I have been doing a bunch of upgrade testing in the ZettaGrid Labs for 5.0 to 5.1/5.5 Upgrades I thought I’d go through the process of installing SSO 5.5 into the environment before upgrading the rest of the vCenter components to 5.1. My main driver here was the opportunity to not have to configure an SSO database and to be ahead of the game when it comes time to jump to 5.5.

vCenter 5.0 to 5.1 Upgrade Mixed Mode SSO 5.5 Install:

Load up the vCenter 5.5 Installer and select vCenter Single Sign On Install. You can see I’m running this installer on a vCenter 5.0 Update 2 instance. The aim here is to have this version upgraded to 5.1 while utilizing the mixed mode capability of SSO 5.5

Run through the installer as you would for a standard SSO 5.5 install

One of the differences to point out here with SSO 5.5 is that the infamous [email protected] account has been replaced with a domain name, in this case I’m rolling with the default vsphere.local domain.

Much like an AD install, you have the ability to choose the default first site name…which does use Default-First-Site is left unchanged.

Once the SSO Service has been installed you can swap out the vSphere 5.5 Media for the 5.1 version and proceed with the upgrade of the Inventory Service and the Upgrade of vCenter it’s self. Both installs ask you for the SSO information relating to the SSO service as shown below. 5.1 will default to [email protected] so you need to modify that to reflect the account details as configured during the SSO 5.5 installer.

Once both the Inventory Service and vCenter have been upgraded, the final step for mixed mode is to install the vCenter 5.5 Web Client as the 5.1 Web Client will not recognise the SSO 5.5 install so you will be unable to modify the Configuration until it’s installed. I did run into a few issues with the install from a 5.0 environment, but that was more due to the fact that I had expired certs in the 5.0 Web Client SSL Folder, which the 5.5 installer checks and than attempts to copy into the new install location. While on the vCenter Single Sign On Information page I got an error saying the SSL cert was invalid, however it was not reflecting the Lookup Service URL as you would assume…but actually doing a check against c:\ProgramData\VMware\vSphere Web Client\ssl\rui.crt which it validates and copies to the new install location.

Once installed it’s business as usual and we have a fully operational mixed mode environment ready for action.