Generally speaking upgrading the vCloud Director binaries is a pretty straight forward task. It’s a two step process where the Cell’s are upgraded first followed by some update scripts run against the vCloud database. Finally database Indexes and Statistics are updated. In the last 6-8 months I’ve worked through 1.5 -> 5.1 and recently 5.1 -> 5.5 upgrades…as well as point releases and BETA builds in the lab…all of those upgrades have ultimately been successful, but as recently as last week I’ve come across a couple of issues when running the update database schema script. The second issue is of more significance as there is little reference to it on Google.
|
1 |
/opt/vmware/vcloud-director/bin/upgrade |
Issue 1:
This one is apparently fairly common, and doesn’t need too much concern.
|
1 |
Unable to update database statistics. Database user has insufficient privileges to update database statistics. To complete this step manually, run 'EXEC sp_updatestats' as the DBO or a member of the sysadmin fixed server role. |
The fix is straight forward and is explained in this post, but the workaround is to simply log into your SQL server and execute the stored procedure as instructed in the warning message from the upgrade script. Done, nice and easy!
Issue 2:
After the database upgrade tasks have been completed you are asked if you want to rebuild the database indexes and that it may take several minutes. I’ve found in the lab and in production this can take 1-2 minutes, however during our latest upgrade we received the message below
|
1 |
Unable to rebuild database indexes. A timeout has occurred while rebuilding database indexes; this can happen if the vCloud Director database is very large. To complete this step manually, consult your databases's administration guide on how to rebuild indexes. |
Even though this is listed as an extra step, we all know how critical indexes are to efficient database query execution so I wanted to try and get the rebuild done. The vCD database in question isn’t overly large (about 3-4GB) and suspecting a transient SQL connectivity error I backed up the database again and ran the upgrade database script once again…this is safe to do and it’s smart enough to work out that the database update schema tasks have been committed…however I received the same error. I didn’t want to blanket rebuild all Indexes and suspected the rebuild to be a little more targeted.
Thinking back to my old days of troubleshooting dodgy programmer queries I decided to fire up MSQL Management Studio and ran the Activity Monitor and filtered out for the vCD database. I ran the upgrade script again and was able to see the queries being run via the monitor tab. Right clicking on the suspected query I was able to view what was being run:
|
1 |
EXEC sp_MSforeachtable@command1="DBCC DBREINDEX ('?', ' ', 80)" |
Against the vCD Database I executed the specific query as show below and it completed successfully in a tick under 3 minutes. There was a noticeable pause in the output messages about 2/3 the way through so my thinking is that the upgrade script has a timeout that was reached due to inactivity.
From here we where able to fire up the Cell and continue with the upgrade process. Again, this issue seems to be rare, but hopefully if anyone comes across it the fix is as straight forward as the update statistics fix.
