Tag Archives: vCenter

Quick Fix: vCenter 5.5 Update 3x Phone Home Warning and VPXD Service not Starting

This week I’ve been upgrading vCenter in a couple of our labs and came across this issue during and after the upgrade of vCenter from 5.5 Update 2 to 5.5 Update 3a or 3b. During the upgrade of the vCenter the error below is thrown.

It’s an easy one to ignore as it only relates to the Phone Home Service…which to be honest I didn’t think would or was important at the time. When you click ok the installed finished as being successful, however the vCenter Service is not brought up automatically and when you go to start the service you get the following error from the services manager.

Not sure why the Googling for this particular error wasn’t as straight forward to search against but if you search to Error 1053 or Error 1053 + VMware you get referenced to some generic forum issues and this VMware KB which is a red herring in relation to this error. With that I went back to search against the Phone Home Warning 32014 and got a hit against this VMware KB which contains the exact error and reference to the deployPkg.dll that you would see in the Windows Application Event Logs when you try to start the vCenter Service.

The KB title is a little misleading in that it states

Updating vCenter Server 5.5 to Update 3 fails with the error: Warning 32014

However the fix is the right fix and after working through the work around in the KB the upgrades went through without issue and vCenter was at 5.5 Update 3b.

References:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2134141

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2069296

Quick Fix: VCSA Web Client 6.0 Throws Monitoring Errors

This quick fix post is for those out there who are still using vCenter Operations Manager 5.8.x and are or thinking about deploying or upgrading to vCenter 6.0…I came across this annoying situation all of a sudden while working on a new vCenter instance when the Web Client started to report the error shown below.

This can be ignored by clicking no and you will still be able to operate most areas of the Web Client but you will find that Monitoring and Health pages fail to load and give you a generic Error #2036 as shown below.

It took me a while to realize that the error was related specifically to the monitoring modules and it finally clicked in my head that the error started happening when I Registered the vCenter against my lab vCOPs instance. I was still running vCOPs (not vRA) and the instance hadn’t been upgraded to the latest build. Having a look through the VMwareKBs I came across KB 2111224 which explained the cause.

This issue occurs because vRealize Operations Manager versions prior to 5.8.5 are not supported in the vSphere 6.0 environment.

Upgrading the vCOPs Appliances to build 5.8.5-2532416 sorted the issue and I was able to browse through the Web Client without the error and have the integrated Health Monitoring work without issue.

References:

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2111224&sliceId=1&docTypeID=DT_KB_1_1&dialogID=850159080&stateId=0%200%20850161293

vSphere 5.5 Update 3 Released: Features and Top Fixes

vSphere 5.5 Update 3 was released earlier today and there are a bunch of bug fixes and feature improvements in this update release for both vCenter and ESXi. For most Service Providers updating to vSphere 6.0 is still a while away so it’s good to have continued support and improvement for the 5.5 platform. I’ve scanned through the release notes and picked out what I consider some of the more important bug fixes and resolved issues as they pertain to my deployments of vSphere.

Note: Still appears that there is no resolution to the vMotion errors I reported on earlier in the year or the bugs around the mClock Scheduler and IOPS Limiter on NFS.

ESXi 5.5 Update 3:

  • Status of some disks might be displayed as UNCONFIGURED GOOD instead of ONLINEStatus of some disks on an ESXi 5.5 host might be displayed as UNCONFIGURED GOOD instead of ONLINE. This issue occurs for LSI controller using the LSI CIM provider.
  • Cloning CBT-enabled virtual machine templates from ESXi hosts might failAttempt to clone CBT-enabled virtual machines templates simultaneously from two different ESXi 5.5 hosts might fail. An error message similar to the following is displayed:Failed to open VM_template.vmdk': Could not open/create change tracking file (2108).
  • ESXi hosts with the virtual machines having e1000 or e1000e vNIC driver might fail with a purple screenESXi hosts with the virtual machines having e1000 or e1000e vNIC driver might fail with a purple screen when you enable TCP segmentation Offload (TSO). Error messages similar to the following might be written to the log files:cpu7:nnnnnn)Code start: 0xnnnnnnnnnnnn VMK uptime: 9:21:12:17.991 cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x65b stack: 0xnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x18ab stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0xa2 stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0xae stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x488 stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x60 stack: 0xnnnnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x185 stack: 0xnnnnnnnnnnnn
  • Attempts to reboot Windows 8 and Windows 2012 server on ESXi host virtual machines might failAfter you reboot, the Windows 8 and Windows 2012 Server virtual machines might become unresponsive when the Microsoft Windows boot splash screen appears. For more information refer, Knowledge Base article 2092807.
  • Attempts to install or upgrade VMware Tools on a Solaris 10 Update 3 virtual machine might fail
    Attempts to install or upgrade VMware Tools on a Solaris 10 Update 3 virtual machine might fail with the following error message:Detected X version 6.9
    Could not read /usr/lib/vmware-tools/configurator/XOrg/7.0/vmwlegacy_drv.so Execution aborted.This issue occurs if the vmware-config-tools.pl script copies the vmwlegacy_drv.so file, which should not be used in Xorg 6.9.

In going through the remaining Known Issues you come across a lot of Flash Read Cache related problems…maybe VMware should call it a day with this feature…not sure if anyone has the balls to actually use it in production…be interested to hear? There are also a lot of VSAN issues still being reported as known with workarounds in place…all the more reason to start a VSAN journey with vSphere 6.0.

For a look at what’s new and for the release notes in full…click on the links below:

VMware ESXi™ 5.5 Update 3 | 16 SEP 2015 | Build 3029944

vCenter Server 5.5 Update 3 | 16 SEP 2015 | Build 3000241

January vCenter and ESXi 5.5 Patch Releases: Critical CBT Bug Fixed

VMware released new builds for vCenter and ESXi 5.5 today. The builds contain mostly bug fixes, but I wanted to point out one fix that had affected those who use CBT as part of their VM backup strategy. Veeam users where initially affected by the bug…though Veeam released a work around in subsequent Veeam 8 builds this ESXi patch officially fixes the issue.

When you use backup software that uses the Virtual Disk Development Kit (VDDK) API call QueryChangedDiskAreas(), the list of allocated disk sectors returned might be incorrect and incremental backups might appear to be corrupt or missing. A message similar to the following is written to vmware.log:

DISKLIB-CTK: Resized change tracking block size from XXX to YYY

For more information, see KB 2090639.

There are also a couple of other interesting fixes in the build:

  • Storage vMotion of thin provisioned virtual machine disk (VMDK) takes longer than the Lazy Zeroed Thick (LZT) disk.
  • When a quiesced snapshot is created on a Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2, or a Windows Server 2012 virtual machine, duplicate disks might be created in the virtual machine.
  • On an ESXi 5.5 host, the NFS volumes might not restore after the host reboots. This issue occurs if there is a delay in resolving the NFS server host name to IP address.

For those who backup VMs with disks larger that 128GB I would be looking to deploy this patch ASAP.

vCOPS 5.8: Critical Data Collection Bug

UPDATE: VMware Global Support supplied me with vCOPs 5.8.0 Hot Fix 01 Build 1537842 which is available via a support request. This is a complete .pak update so you will need to go through the upgrade process as per usual.

The issue of the missing data has been resolved, however I did need to go through and remove a heap of duplicate entity types in the Custom Dashboard. I have a fully functional vCOPs platform now. Hopefully no more bugs in this build!

Like most…I jumped to upgrade vCOPs from 5.7.x to 5.8 when it went GA mid December. Initially the upgrade completed without issue and the four vCenter’s registered looked to continue collecting data as expected.

# To cut to the guts of the error scroll to the bottom on the post.

Shortly after the upgrade I went through and performed an upgrade of vCenter from 5.0 to 5.1 in one of our sites. Upon completing the upgrade and having a look at the Custom Dashboards we have setup, I noticed that 90% of the hosts in the recently upgraded vCenter where showing as white boxes (no data)

Looking through the Analytic VM collector logs I found these entries that seemed to point to a connectivity/communication issue between vCOPs and the Hosts.

After going through a painful couple of support calls with VMware support where they where insistent the issue was with the vCenter (have you tried turning it off and on) and/or a disk space issue on the Analytic VM. They suggested it was a well known upgrade bug that can be resolved as per below.



Off the bat I knew this wasn’t my issues because initially on upgrade I didn’t have the issue. While I waited for support to try and diagnose the issue via vCOPs support log bundles, I assumed that the issue was in the data…possibly a bad row in the database relative to a host/vCenter.

I removed the affected vCenter from vCOPs Admin Registration Tab and ran the following command on the UI VM that I picked up from @h0bbel‘s Post here:

 

The command ran for about 20-30 minutes and returned as being successful. When I went back into vCOPs the affected vCenter’s hosts had returned and was actively collecting and reporting data…however I now saw a number of other hosts across multiple vCenters showing showing the same problem!

I contacted VMware support again and had the case escalated which resulted in the admission that there was a newly discovered (probably through my persistence in regards to the case) bug in 5.8 and that I was experiencing all the symptoms.

Issue in 5.8 where we have the below symptoms:

  • One or more ESXi/ESX hosts are no longer present in the vCenter Operations Manager inventory.
  • ESXi/ESX hosts are missing from the vCenter Operations Manager inventory.
  • Child objects of missing ESXi/ESX hosts such as virtual machines and datastores are present in the vCenter Operations Manager inventory.
  • This issue occurs when you place the ESXi/ESX hosts into Maintenance Mode in vCenter Server and then take the hosts out of Maintenance Mode.

Review the Below KB:
http://kb.vmware.com/kb/2068303/

So, at the moment there is no resolution or fix and the workarounds are pretty nasty! …basically you will be modding database entries and taking snapshots of the Analytic VM.

I just got off the phone with VMware support in Palo Alto and got told that a hotfix was being worked on and should be available soon. Once released to me, i’ll update this post with the final resolution.

vCenter SSO 5.5: AD Group Membership Gotchya

I’ve been running vCenter SSO 5.5 in mixed mode with vCenter 5.1 for a little while in our lab and production environment and recently had to configure external authentication for access against a subset of VMs we host. I covered how to setup a vCenter 5.1 AD Identity source in a this previous post, and the process hasn’t really changed that much in 5.5.

While there has been a slight tweek in the UI, the setup process is the same except you now have an additional option to utilize AD with Integrated Windows Authentication, which is covered by @chriswahl in this blog post.

After configuring the identity source as shown below.

And after configuring the appropriate permissions in vCenter I was getting the following error while trying to login with a user who was part of a Group with access to the VM location.

There wasn’t a hell of a lot on Google that looked to resolve this issue, and after some investigation I found that the reason for the error was that vCenter SSO will look to interrogate/verify every group of the user attempting to login…even though some may be out of scope and not relevant. The fact the error was returning a GroupSID that couldn’t’ be found told me that during the login process, SSO enumerates all Groups in order to find a match relevant to the vCenter Permission. If that group falls out of the Base DN for Groups, SSO can’t resolve the name to match it.

To resolve the issue I had to modify the Base DN for Group Location to be the domain root…once done the user was able to login without issue. This probably isn’t a bug like previous AD Domain SSO bugs however its still limits security restrictions when designing/allocating a user and group OU structure for AD authentication.

vCenter SSO 5.5 Backwards Compatability: Mixed Mode Install

While at a post #vFourmAU event last week a group of us where talking about SSO in vCenter 5.1 and what a disaster it had been (credibility wise) and that VMware had done their best to improve the experience in the 5.5 release. Out of that conversation came word that if people where looking to upgrade from 4.x or 5.0 to 5.1 as an interim step to 5.5, you could install SSO 5.5 in that 5.1 Environment so as not to go through the relative pain that SSO 5.1 brings to the table…that is to say SSO 5.5 was backwards compatible.

http://blogs.vmware.com/vsphere/2013/10/backwards-compatible.html

That post above goes through a little background and case examples where it might be relevant to run SSO 5.5 mixed in with vCenter5.1.

While I have been doing a bunch of upgrade testing in the ZettaGrid Labs for 5.0 to 5.1/5.5 Upgrades I thought I’d go through the process of installing SSO 5.5 into the environment before upgrading the rest of the vCenter components to 5.1. My main driver here was the opportunity to not have to configure an SSO database and to be ahead of the game when it comes time to jump to 5.5.

vCenter 5.0 to 5.1 Upgrade Mixed Mode SSO 5.5 Install:

Load up the vCenter 5.5 Installer and select vCenter Single Sign On Install. You can see I’m running this installer on a vCenter 5.0 Update 2 instance. The aim here is to have this version upgraded to 5.1 while utilizing the mixed mode capability of SSO 5.5

Run through the installer as you would for a standard SSO 5.5 install

One of the differences to point out here with SSO 5.5 is that the infamous [email protected] account has been replaced with a domain name, in this case I’m rolling with the default vsphere.local domain.

Much like an AD install, you have the ability to choose the default first site name…which does use Default-First-Site is left unchanged.

Once the SSO Service has been installed you can swap out the vSphere 5.5 Media for the 5.1 version and proceed with the upgrade of the Inventory Service and the Upgrade of vCenter it’s self. Both installs ask you for the SSO information relating to the SSO service as shown below. 5.1 will default to [email protected] so you need to modify that to reflect the account details as configured during the SSO 5.5 installer.

Once both the Inventory Service and vCenter have been upgraded, the final step for mixed mode is to install the vCenter 5.5 Web Client as the 5.1 Web Client will not recognise the SSO 5.5 install so you will be unable to modify the Configuration until it’s installed. I did run into a few issues with the install from a 5.0 environment, but that was more due to the fact that I had expired certs in the 5.0 Web Client SSL Folder, which the 5.5 installer checks and than attempts to copy into the new install location. While on the vCenter Single Sign On Information page I got an error saying the SSL cert was invalid, however it was not reflecting the Lookup Service URL as you would assume…but actually doing a check against c:\ProgramData\VMware\vSphere Web Client\ssl\rui.crt which it validates and copies to the new install location.

Once installed it’s business as usual and we have a fully operational mixed mode environment ready for action.


How-To: vCenter 5.1 SSO Adding AD Identity Source


The SSO Component of vCenter 5.1 throws a couple of spanners in the works with regards to a straight forward upgrade of an existing vCenter install. While not overly complicated in terms of understanding what and how the SSO Service fits into the 5.1 puzzle, I found that it did add a couple of additional configuration steps that where not expected during and after the upgrade process. There are a heap of resources out there already on the end to end install of the SSO…be it a Simple Server install or a multi-server HA set-up, but your best bet is to catch up on the official VMware Documentation here.

EDIT: @VMwareKB vSphere Blog SSO Help Page http://t.co/9y20Kk22

In my environment we already employed AD authentication by way of Group Membership that dictated access to the vSphere Datacenters and Clusters. This was well established and working without too much hassle. My first attempt at the vCenter 5.1 upgrade yielded mixed results with the SSO, but lesson learnt was that I made the mistake of being too eager to jump into the upgrade without RTFM!

What I am now calling an exercise in executing a roll-back plan came about because I didn’t bother to understand how the SSO component affects an existing set-up and also from not paying attention during the install. In truth, I thought the first upgrade failed to install SSO correctly as I was getting errors when trying to login and the Web Client wasn’t able to connect to the SSO service. Couple of points here is that I rushed through the “Error 29155 Identity source discovery error” which is referenced by KB 2034374 and I attempted to “fix” the service by messing with the SSO Service Log-on user configuration. In the end I got impatient and rolled back the vCenter SnapShot I had taken before upgrade and started again. (Side note: that actually worked ok even after 5.1 agents where deployed to hosts managed by the vCenter…after rolling back the snap the 5.0 agent’s where re-redeployed without hassle)

So, once 5.1 had been installed and all components have been upgraded, you need to add your AD LDAP profile as an Identity Source via the vCenter Web Client. Without this, your existing AD credentials will not be honoured.

Log into the vCenter Web Client with the credentials provided during setup:

sso_1

Click through Administration -> Sign-On and Discovery -> Configuration and click on the green + Button in the centre window pane.

sso_2

Collect all your relevant AD LDAP information and complete the set-up as shown below.

sso_3

If all the settings are correct you will get a positive Test Connection response.

sso_4

Now that you have your Identity Source configured you can add the new source to the default domains by clicking Add to default domains in the top bar and bump the new source to the top of the list in the bottom pane. This allows you to not have to enter the NETBIOS name of the domain during login. eg DOMAN\username vs username.

Final thing to check now is to ensure that your previous Permissions based on AD groups are still in place relative to the vCenter, Datacenters, Clusters etc. As shown below, from this point forward you can configure access as you would have previously…the only change now is you have the option of selecting the Domain to reference.

sso_5

What this means, is in theory you could pull in external/client LDAP Identity Sources to use as authentication mechanisms on your vCenter…not sure it’s totally useful for vCenter’s, but can see this being extremely useful for management and automation layers like vCloud and vCOPs or even vCO.

Recent Entries »