Veeam SureBackup – Virtual Lab Creation Failure

I was recently in the process of configuring a virtual lab for Veeam Backup & Replication 6.1 Patch 1 to find that should you select not to create a virtual machine folder or resource pool at the Host configuration stage, the lab creation will fail. A valid question would be why would you not want to create the folder or resource pool, however in my case this was a small deployment and the customer only had VMware Essentials Plus licensing which does not offer resource pools as a feature. The error being experienced was the following:

Registering proxy appliance Error: The object has already been deleted or has not been completely created

Failed to register VM, configFile ‘[Datastore Name]  Virtual Lab Name/drv_va.vmx’, name ‘Virtual Lab Name’, isTemplate ‘False’, poolRef ‘resgroup-8’, hostRef ‘host-9’, folderRef ‘ha-folder-vm’

In order to resolve this error, ensure that the creation of the virtual machine folder and resource pool is enabled in the virtual lab configuration. As a result, I have now logged this issue with Veeam as a product bug.

That’s it!

Update: Veeam have now confirmed this is a bug in version 6.1 Patch 1.

URL: http://forums.veeam.com/viewtopic.php?f=24&t=13501

VMware View – Adding & Removing Pool Entitlements Via Scheduled Tasks

I recently had a requirement for a customer to automatically add and remove user entitlements on VMware View 5.0 virtual desktop pools. While this can be achieved through View PowerCLI, the customer needed to query Active Directory to obtain a security group and then apply or remove this group from a pool. The issue with View PowerCLI in 5.0 is that the PowerShell aspect is a snap-in and not a module. Due this, a simple script calling both the Active Directory and View PowerCLI modules is not possible. To work around the limitation I have written the following scripts to add and remove pool entitlements using Active Directory groups.

Add Entitlement PowerShell Script: Download

Remove Entitlement PowerShell Script: Download

There are a two environment specific variables that need changing in each of the .PS1 files, these are the following:

 Get-ADGroup“Users” | Add-PoolEntitlement -pool_id “Test”

 Get-ADGroup“Users” | Remove-PoolEntitlement -pool_id “Test”

Where “Users” is the name of the Active Directory security group you wish to add or remove from a pool, and where “Test” is the name of the pool in which you want to apply or remove an entitlement. To run the scripts as scheduled tasks, place each script on one of your VMware View Connection Servers, in a directory such as C:\ViewScripts. Proceed and create a new basic task in the Windows Task Scheduler and specify the required bat file as available for download below.

Add Entitlement Bat File: Download

Remove Entitlement Bat File: Download

The created bat files essentially call powershell.exe and then execute the required .PS1 file. A specific “-File” parameter in the bat file needs amending for your environment, as detailed below.

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -file “C:\ViewScripts\AddEntitlement.ps1”

The process is now complete.

Veeam Backup & Replication – Exchange 2010 DAG Issue

I recently experienced an issue with a Microsoft Exchange 2010 Database Availability Group (DAG) failing over during a Veeam Backup & Replication job. The issue was occurring due to the snapshot committal process in VMware, which causes a brief pause in virtual machine I/O. This pause was causing the DAG member to lose sight of the file share witness, which in this case was housed on the customer CAS server, and subsequently fail over.

The resolution to this issue was to increase the CrossSubnetThreshold and CrossSubnetDelay of the cluster. The CrossSubnetThreshold specifies how many heartbeats can be skipped before the cluster fails over and the CrossSubnetDelay specifies the heartbeat interval. The threshold you set for both of these properties can depend on many factors, for example the speed of your underlying storage array or the size of the virtual machine that be being snapshot. In my case I needed to set both values to their maximum. This can be performed by carrying out the following:

1. Navigate to Start -> Administrative Tools and launch Windows PowerShell Modules

2. When the Powershell Window opens please enter the following command:
 
$cluster = Get-Cluster; $cluster.CrossSubnetThreshold = 10; $cluster.CrossSubnetDelay = 4000
 
3. Once the command has completed please run the following and ensure that the CrossSubnetDelay and CrossSubnetThreshold are set to 4000 and 10.
 
Get-Cluster | fl *

4. Re-run your Veeam backup job and see if the cluster fails over. If the backup completes correctly you can they reduce the CrossSubnetDelay and CrossSubnetThreshold to find the optimum values.

That’s it, your done.

VMware ThinApp – Microsoft Office 2010 Setup Guide

After successfully being able to create a full Microsoft Office 2010 suite ThinApp, I have written a setup guide to detail the steps that had to be taken in order to get all Office applications working and to allow standard windows users (non local administrators) to activate office when launching the ThinApp for the first time. This article has been written using information for VMware sources and my own experience. The following ThinApp capture has been performed on a fresh installation of Windows XP SP3 with VMware ThinApp 4.6.1.

1. Download and copy the Microsoft .NET Framework 3.5 and Microsoft Office 2010 installation files to the ThinApp capture machine.

2. Ensure you have internet access on the ThinApp capture machine.

3. Install VMware ThinApp on capture machine.

4. Run the Setup Capture wizard, start and complete the pre-scan process.

5. When the pre-scan is complete, on the Install Application page of the Setup Capture wizard, minimize the wizard and install Microsoft .NET Framework 3.5.

6. The Microsoft .NET installation generates the mscorsvw.exe process that continues for an extended period, we need to stop the process with the ngen.exe tool. Open a new command prompt window and type the following command and press return:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\ngen.exe executequeueditems

7. In order for Office 2010 to activate correctly we need to edit the Config.xml file and specify your Multiple Activation Key (MAK). By default, the Microsoft Office 2010 Config.xml file is stored in the core product folder in the Microsoft Office 2010 installation media. Explore the Office 2010 installation media on the capture machine and you should see a folder named similar to “core_product_folder_name.WW folder”. For example, if my Microsoft Office 2010 media was Professional Plus, the folder name would be “ProPlus.WW”.

8. Open the core product folder and locate the Config.xml file. Open the Config.xml with Notepad and add the following lines to the to the file file and save it to your local C:\ drive:

<PIDKEY Value=”Enter 25 Character MAK Here” />
<Setting Id=”USEROPERATIONS” Value=”1″ />

Note: Do not enter any spaces or hyphens when entering the license key in the PIDKEY value, this must be all one string such as ABCDEFGHIJK.

9. To start the Microsoft Office 2010 setup wizard and apply the settings in Config.xml, open a new command prompt window and type navigate to your Office 2010installation media by using for example “cd d:\”. When in the installation media directory enter the following command and press return.

Setup.exe /config c:\config.xml

10. When the installation starts click to accept the terms of the agreement and click next.

11. On the Choose the installation dialog box, click the Customise button.

12. Select Microsoft Office and then select “Run all from My Computer” as the installation option.

13. Click “Install Now” to install Microsoft Office 2010, this will likely take some time.

14. When the installation is complete click “Finish”. Proceed and click the Start menu, select the Run option, and then type services.msc and click ok. When the services snap-in opens scroll down the list of services and locate the “Office Software Protection Platform” service, right click this service and select stop.

15. Maximise the ThinApp Capture wizard that we minimised in step 5 and then click to start the post scan process.

16. Once the post scan process has completed, change the Inventory name to something appropriate for your use, for example Microsoft Office 2010.

17. Before building the ThinApp project , copy the OfficeSoftwareProtectionPlatform folder from C:\Documents and Settings\All Users\Microsoft to the folder %Common AppData%\Microsoft. To get to the” %Common AppData%\Microsoft folder” open the ThinApp Setup Wizard and click the “Open Project Folder” button, in the resulting window you should then see a folder named “%Common AppData%”  double click this folder then double click the “Microsoft” folder inside. Simply copy and paste the “OfficeSoftwareProtectionPlatform” folder into this location.

18. On the capture machine open the registry editor and navigate to the following key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Search\Preferences

In this location, right click in a white space area an create a new Multi-String Value with the name of, without quotes, “isolation_writecopy”. Right click the created “isolation_writecopy” key and select “Modify”. When the modify window opens, in the value area, enter the following information and click ok:

Value=PreventIndexingOutlook
REG_DWORD=#01#00#00#00

19. In the same registry editor window navigate to the following key:

HKEY_LOCAL_MACHINE\Software\Microsoft\OfficeSoftwareProtectionPlatform

In this location, right click in a white space area an create a new Multi-String Value with the name of, without quotes, “isolation_full”. Right click the created “isolation_full” key and select “Modify”. When the modify window opens, in the value area, enter the following information and click ok:

Value=UserOperations
REG_DWORD=#01#00#00#00

20. In the same registry editor window navigate to the following key:

HKEY_CURRENT_USER\Environment

In this location, right click in a white space area an create a new Multi-String Value with the name of, without quotes, “isolation_writecopy”. Right click the created “isolation_writecopy” key and select “Modify”. When the modify window opens, in the value area, enter the following information and click ok and then close the registry editor:

Value=ALLUSERSPROFILE
REG_SZ~%Common AppData%

21. Maximise the ThinApp Setup Wizard and click on the “Edit package.ini” button before starting the build process. In the ini file change the “MSIStreaming” value to 1 if you want this to be streamed ThinApp otherwise this will be a local ThinApp.

22. Click the “Build” button in ThinApp and wait for the build process to complete.

23. The ThinApp creation process is now complete and the applications should work and activate correctly on Windows XP systems. If you intend to use the Office 2010 ThinApp on Windows 7 operating systems, some additional changes need to be made to ensure that the product activation works correctly.

24. For Windows 7 machines only, in order for the ThinApps to launch and activate Office 2010 correctly you will need to disable the “Windows Search Service” and completely disable User Account Control (UAC). Disabling these two properties can be achieved by using Group Policy Objects as detailed here and here.

That’s it, you should now have a fully functionaly Microsoft Office 2010 VMware ThinApp that works on both Windows XP and Windows 7 operating systems.

VMware P2V – Linux Conversion Issue

I recently had to P2V a physical CentOS 5.3 server into a VMware vSphere 4.1 virtual environment. On completing the P2V, which took over four hours, I was presented with an “Error loading operating system” message when powering on the virtual machine. I had initially assumed this was an issue that had occurred during the conversion, however some further investigated revealed this was not that case. When converting the machine from a physical to virtual platform the boot partition location in the GRUB loader was no longer valid, as of course all hardware properties had now changed. To resolve the “Error loading operating system” message the following steps were performed:

1. Download the latest version of CentOS, in my case this was CentOS 5.6 CD 1.

2. Upload the CentOS ISO to a datastore in your virtual environment and attached it to the converted virtual machines CD/DVD drive.

3. Power on the converted virtual machine and press ESC to show the VMware boot device selector. Select and press return on CD/DVD-ROM drive, this will now load the attached CentOS ISO.

4. When CentOS loads, at the boot prompt type without quotations “linux rescue”, and press return.

5. Select to mount all file systems in read-write mode and press return.

6. To re-install GRUB type the following without quotes and press return, “grub-install /dev/sda”.

7. Once the installation has finished type “reboot” and press enter to restart the virtual machine.

That’s it, the GRUB loader should not be repaired and your virtual machine should successfully boot.

Windows Deployment Services – Injecting VMware Drivers

Windows Deployment Services (WDS) is a technology I have been using for a long time, and is by far one of my favourites. I had a recent requirement to deploy a captured WDS image into a new VMware virtual machine. On doing so I was presented with a driver error, as the VMnet drivers that VMware utilises are not included in the Windows 7 boot.wim file as you would expect. To deploy an image to a VMware virtual machine we need to customise the boot.wim file to include the relevant VMnet drivers. To achieve this, perform the following steps.

1. Obtain a copy of, or use your existing boot.wim file. You can obtain a copy of the boot.wim file from the Windows 7 operating system installation media.

2. Place the obtained boot.wim file on the root of your workstations C:\ drive, so for example, C:\boot.wim.

3. On your C:\ drive create a new folder name “mount”. This will be used to open the contents of the WIM so that we can inject the VMware drivers.

4. Download a copy of the VMnet VMware drivers from here. We will be injecting these drivers into our WIM image. Once the download is complete, extract the drivers folder and place it also on the root of your workstations C:\.

5. Download and install the Windows Automated Installation Kit (WAIK) for Windows 7 onto your workstation. This can be downloaded direct from Microsoft here.

6. Once you have downloaded and installed the relevant WAIK, navigate to WAIK start menu entry and launch the “Deployment Tools Command Prompt”.

7. When the command prompt loads, enter the following command to mount the WIM file for modification without quotes:

“dism /mount-wim /wimfile:c:\boot.wim /index:2 /Mountdir:c:\mount”

8. Once the boot.wim file has mounted, we can now inject the VMware drivers. To perform this run the following command without quotes:

“dism /image:c:\mount /add-driver /driver:c:\vmnet\ /recurse”

9. Once both of the VMnet drivers have been successfully injected, we then need to commit the changes and unmount the image. To perform this run the following command without quotes:

“dism /unmount-wim /mountdir:c:\mount\ /commit”

10.Upload the unmouted image to your WDS server  and then add it as boot image via the WDS console.

Your all done, you should now be able to use both WDS deployment and capture images in VMware virtual machines.

VMware Capacity Planner – Collector Access Denied

I recently experienced a strange issue when implementing the VMware Capacity Planner at a customer site. Once the collector software had been installed and I attempted to run the initial inventory of my target servers, the following error message was displayed in the capacity planners task pane.

ERROR: An unexpected error occurred!  Module = VMware Capacity Planner Collector  Function = ProcessJobs  Source = vcpCollector  Error = Permission denied(70:0:1000070)#

On first glance I thought this was an issue when querying a target server though either the remote registry or WMI. Some further research led me to VMware Knowledge Base article KB 1001396 which detailed the error listed above. Unfortunately the article detailed that this error is usually reported when running the collector on a Windows Server 2003 platform that is utilising hardened security methods, I was however using a Windows XP Professional machine as my collector platform. Ironically the suggested solution for this error in the VMware KB article is to install the collector on a Windows XP workstation.

After removing anti-virus applications, GPO’s and several software re-installations later a colleague recommended resetting the local security policy on my Windows XP collector workstation back to default, in the eventuality the security policy applied via GPO had not removed correctly. After running a secedit command to restore the local security policy and re-running the inventory task in the capacity planner everything was now working as expected. To reset the local security policy of a Windows XP workstation, perform the following steps:

1. Click Start and select the Run menu item.

2. In the Run dialog box type CMD and click ok.

3. In the command prompt window type the following entry, without quotations, and press return on your keyboard:

“secedit /configure /cfg %windir%\repair\secsetup.inf /db secsetup.sdb /verbose”

4. Once you have pressed return on your keyboard you should then receive a “Task is completed” message and a warning message that an item could not be completed. You can safely ignore this message.

5. In the Capacity Planner software re-run the initial inventory task, and you should now see that the access denied error is no longer displayed.

For more information on resetting the local security policy of any Windows based operating system, please see the following URL: http://support.microsoft.com/kb/313222

Windows Server 2008 – Disabling Dynamic DNS Updates

I recently experienced an issue with Dynamic DNS updates on Windows Server 2008. Since upgrading VMware tools on a Windows Server 2008 virtual machine, all six network adapters that were assigned to the VM were now registering themselves on my internal DNS servers, despite me having unchecked the “Register the connections address in DNS” checkbox on each adapters properties. This resulted in me having six host A records in my internal DNS for the same server, however I only wanted one of the servers IP addresses to be registered against it’s hostname.

Unfortunately enabling and then disabling the “Register the connections address in DNS” option again did not resolve the issue. I figured this occurred as when upgrading VMware tools the servers network adapters are removed and re-added. To resolve this issue I opted to disable Dynamic DNS updates on the server all together using a registry entry. To disable Dymanic DNS on a Windows Server 2008 or Server 2008 R2 machine, perform the following actions.

1. Login to the server with the issue.

2. Click the Start menu and select Run.

3. In the Run dialog box type the following entry without the quotation marks and then click ok:

“reg add hklm\system\currentcontrolset\services\tcpip\parameters /v DisableDynamicUpdate /t REG_DWORD /d 1 /f”

4. Reboot the server to complete the process.

I would recommend keeping a watch on your internal DNS servers for 24 hours after applying this registry key, to completely ensure the issue is resolved. You can find additional information on methods of disabling Dynamic DNS on Windows Server platforms at the following Microsoft  KB article: http://support.microsoft.com/kb/816592

VMware – Veeam FastSCP

I recently attended the VMware Install, Manage and Configure course for vSphere 4.0. As a part of the course I was introduced to Veeam’s FastSCP client. Veeam FastSCP is a freeware program that allows you to manage your ESX / ESXi host’s datastores. The application provides you with an easy to use Windows Explorer like interface that supports drag and drop functionality between hosts and from your desktop computer. This is functionality is particularly handy when uploaded ISO files or when taking a copy of a VMDK file.

One thing I found before being introduced to Veeam’s FastSCP is how frustrating slow it is when copying files to and from datastores using the Datastore Browser in the vSphere client, I’m sure you’ll agree it’s a pretty painful experience when transferring large files over a 100 megabit network for example. For this reason I thought I would give FastSCP a try, especially when the vendors website claims it is up to six times faster than the traditional datastore browser.

After installing the software and adding my ESXi host, which was achieved by simply hitting the “Add Server” button, I was now able to browse my datastore in full.

Veeam FastSCP Datastore Browing

How much quicker is FastSCP though? Well to test I am going to demonstrate the copy of a 40 gigabyte VMDK file from my datastore to a laptop computer running over a standard 100 megabit connection. This should give a good idea of any real speed advantages. First up I tested the time using the datastore browser to download the VMDK file, here is the estimated download time.

As you can see the total estimated time is 333 minutes, which is the equivalent of 5 hours and 56 minutes. Next up was testing the download of the same VMDK with Veeam’s FastSCP.

Copying a VMDK with Veeam FastSCP

As you can see with FastSCP the estimated time is 2 hours and 2 minutes, which is the equivalent of 122 minutes in total. By comparing both of these times, we can see it will take 211 minutes longer by using the datastore browser to copy the VMDK file. That’s a massive 3 hours and 31 minutes longer compared to FastSCP.

If you looking for a quick, easy and fast utility to use for transferring data to and from datastores or even between hosts Veeam’s FastSCP is certainly worth a try. To save you from having to register to obtain the freeware application, I have uploaded it here. More information can found on FastSCP at the following URL: http://www.veeam.com/vmware-esxi-fastscp.html

VMware – Adding Unallocated Space To Windows Partitions

So, let’s say you have a 40 GB hard disk attached to a virtual machine that is your line of business application role holder. As time has gone by this forty gigabyte basic disk, which holds your operating systems primary partition, is slowly running out of space and gaining any back is become an increasing difficult task. If you don’t want to convert your basic disk to a dynamic configuration, or are not familiar with diskpart.exe, you can use this handy Dell utility to perform this process in seconds. This is particularly useful for virtual machines as you can simply increase your virtual disk size, rescan your Guest OS disks through it’s Device Manager so that the unallocated space is presented, and then run the Dell utility to expand your primary partition. An example output of the utility is shown below:

Expanding primary partitions

Using the utility requires no system downtime and there is no need to move your system page file to a separate hard disk. You can download the utility here and it’s associated read me here.

Please note, this utility was originally created for Dell Disk Array’s and should be used with caution in a physical environment.