VMware Capacity Planner – Missing Data

A while ago I was performing a VMware Capacity Plan for a third party company. During the implementation of the planning software I experienced an issue collecting CPU, RAM and disk statistics for two particular servers. I noted that when running an inventory job in the capacity planning software, the following error was being produced:

RetrieveSystemUpTime: PdhVbGetDoubleCounterValue() has bad status. No uptime data 

After a prolonged investigation of the issue I contacted VMware to see if they had experienced this problem. To some surprise, there was no entry in their internal knowledge base for this issue and the problem was passed to their developers in the United States. A week later, and it turned out this error is actually produced if the capacity planning software receives “invalid” data from a server. In my case, rebuilding the Windows Performance Counter Library Values resolved the issue. We could only assume these had become corrupt and were therefore sending incorrect values to the capacity planner. To rebuild your performance counters, please follow the steps detailed in Microsoft KB Article 300956: http://support.microsoft.com/kb/300956

VMware – Thin Provisioning Woes

Recently I found myself in a situation where someone had deleted the contents of a SAN LUN that was used to host a proportion of virtual machines for a VMware vSphere infrastructure. Thankfully, there was a full backup of all virtual machines at my disposal. After restoring a virtual machine to an alternate location and then uploading it to the datastore, I experienced the following error when attempting to start the VM:

“Failed to open disk scsi0:0: Unsupported and/or invalid disk type 7. Did you forget to import the disk first?Unable to create virtual SCSI device for scsi0:0, ‘/vmfs/volumes/4a8075b3-4e4bf1b8-28e40017a48d6112/VMFolder/VirtualMachine.vmdk’ Module DevicePowerOn power on failed.”

After some research, it emerged that this issue was due to the fact all of the virtual machines were thin provisioned when created. When restoring a backed up virtual machine that has been created using a product such as Symantec Backup Exec 2010, a 2gbsparse disk is created that cannot be used with virtual machines hosted in ESX, which causes the error listed above. To resolve this issue we need to essentially convert the thin provisioned disk to a thick provisioned disk. This can be achieved by performing the following:

1. Log into the root of an ESX host using an SSH client

2. Change directory to the location of the restored virtual machine

3. Run the following command: vmkfstools –i “Thindisk.vmdk” -d zeroedthick “Thickdisk.vmdk” . Please note, replace thindisk with your restored vmdk file name, and replace thickdisk with the new name for the vmdk file.

4. The clone operation will now start and a percentage complete indicator will be displayed. Once this is finished the thick provisioning of the vmdk will be complete and you will now be able to start your virtual machine. Please note that you will need sufficient space on your LUN to allow for the thick provisioning.

For reference, here’s VMware’s slightly unclear article on the procedure: http://kb.vmware.com/kb/1005628

Symantec Backup Exec 2010 – VMware Virtual Machine Issue

Since upgrading from version 12.5 of Symantec Backup Exec, my virtual machine backup job had consistently failed. As highlighted by a colleague of mine, Backup Exec 2010 no longer utilises VMware Consolidated Backup (VCB) as it’s method of VM backup, but a new API called vStorage.

Due to this, the methodology of backing up virtual machines through vStorage is very different to that of VCB, and as such generated an error for each of my virtual machines during backup:

V-79-57344-38297

The virtual machine has a physical disk error (V-79-57344-38297), as shown in the above screenshot, was being reported for each virtual machine that has a Raw Device Mapping. In order to resolve this issue each Raw Device Mapping had to be changed from physical mode to virtual mode. The following actions were performed to achieve this:

1. Log into VMware vCenter as an administrative user.

2. Locate an affected virtual machine and safely power it down.

3. Right click on the virtual machine and select “Edit Settings”.

4. Locate the virtual hard disk drive that has been configured as a Raw Device Mapping.

5. Click remove and then when prompted select to delete the file. Please note this does not remove the data stored on your SAN LUN, this is simply a reference file used by the virtual machine itself.

6. Click OK, and wait for the Raw Device Mapping to be removed.

7. Right click on the virtual machine and select “Edit Settings”, select Add, and then select Hard Disk. Following the wizard, re-add your LUN or LUN’s and ensure that you set the compatibility mode to virtual and not physical.

8. Repeat for all affected virtual machines, and your all done!

For reference here’s VMware’s KB article on switching between RDM physical and virtual modes: http://kb.vmware.com/kb/1006599