log-jam
This is something to look out for if you are upgrading from vCloud Director 5.5.x to 5.6.x SP and if you have altered the syslog settings of your cells….This little gotcha nearly caused a couple of our production vCloud Director Cells to fail due to insufficient storage on the their file systems. Prior to our alerting warning us about an low disk space alert on the cells I couldn’t remember any issues with disk space and vCD Guest Systems since I first started with vCD in 2011…as is normally the case storage space is generally consumed by logs that don’t rotate or grow unexpectedly…so I went looking in the vCD Logs directory under /opt/vmware/vcloud-director/logs And sure enough I found that the jmx.log had ballooned to 17GB and had not rotated since the date of the upgrade to 5.6.4

After getting a support ticket raised it was found that the vCD container debug logs where not even rolling…in fact there was no debug log activity on the cells (except for the cell.log) since the upgrade.

Looking at the log4j.properties file under /opt/vmware/vcloud-director/etc it looked like everything was in order and that the logs should be rotating after the file hit 20MB however it was obvious that this wasn’t happening.

Working with VMware Support we came across this section of the Install and Upgrade guide where it talks about

(Optional) Update logging properties.

After an upgrade, new logging properties are written to the file /opt/vmware/vcloud-director/etc/log4j.properties.rpmnew

Option

Action

If you did not change existing logging properties

Copy this file to /opt/vmware/vcloud-director/etc/log4j.properties.

If you changed logging properties

Merge /opt/vmware/vcloud-director/etc/log4j.properties.rpmnew file with the existing /opt/vmware/vcloud-director/etc/log4j.properties. Merging these files preserves your changes.

Sure enough our log4j.properties file had been altered to add some custom logging formats and targets and in each of the directories there was the log4j.properties.rpmnew file. That is to say there are new logging properties that come with vCD 5.6.x SP and if you had altered the file prior to upgrade those additions meant the logging broke. As suggested above all that needed to be done was to merge both files, or append the customer entries to the bottom of the .rpmnew version and save that as log4j.properties and restart the cells. Once I did this the debug logs started to flow again and the jmx.log file was rotated allowing me to shrink/delete the file consuming the storage.

 

References:

http://pubs.vmware.com/vcd-56/index.jsp#com.vmware.vcloud.install.doc_56/GUID-CEF834DA-1FF5-4819-9D24-88DE6F005C78.html?resultof=%2522%2572%2570%256d%256e%2565%2577%2522%2520