VMM Service Crashes Repeatedly

In this case sysadmins were patching with latest firmware and drivers one of the Hyper V cluster hosts, due to unexpected server reboots. HPE support has recommended patching the server DL 380 Gen8 with latest firmware and drivers. So, latest PSP from HPE has been downloaded, and drivers and firmwares were installed.

Few hours later, after the installation of updated drivers and firmwares on the Hyper V host, VMM console of the VMM server which was managing the host has become unavailable. Also, System Center Virtual Machine Manager service on VMM server was terminating unexpectedly and event id 7034 was logged in system event log with information:

The System Center Virtual Machine Manager service terminated unexpectedly.  It has done this 3 time(s).

Since, the event doesn't say much about why the service was crushing, VMM debug logging has to be enabled using the following article https://support.microsoft.com/en-us/help/2913445/how-to-enable-debug-logging-in-virtual-machine-manager .

After enabling debug logging, the log was showing that error was generated whenever VMM was trying to reach the affected Hyper V server using WinRM and querying the WMI, for example (content truncated):

[Microsoft-VirtualMachineManager-Debug]4,4,WsmanAPIWrapper.cs,1913,WinRM: URL: [http:\\affected.hyperv.host]… 

[Microsoft-VirtualMachineManager-Debug]4,1,WsmanAPIWrapper.cs,3148,Retrieving underlying WMI error to throw. Got string ...

So, going back to the "updated" server and checking the update log, I've found that HPE Insight Management WBEM Providers were updated, and checked the following HPE article https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00053606en_us .

If Hyper-V is installed before Downgrade/Upgrade/fresh installation of HPE WBEM Providers, run the following steps after installing HPE WBEM Providers:

• net stop vmms
• mofcomp %SYSTEMROOT%\System32\WindowsVirtualizationUninstall.mof
• mofcomp %SYSTEMROOT%\System32\WindowsVirtualization.V2.mof
• net start vmms

This will restore the Msvm_ classes under root\interop namespace that had been overwritten by HPE WBEM Providers.

After recompiling the "original" mof files, VMM functionality was back, and VMM was able to query the affected Hyper V host.

I hope this article will save some of your "precious" admin time in debugging of this kind of combination of Hyper V, HPE Updates and VMM, and … Happy patching :)

Stuck in WinPE when converting P2V

In this case I got stuck in WinPE when I was converting physical server with installed Windows Server 2003 R2 SP2 operating system using VMM 2012 SP1 P2V method for creation of virtual machines. P2V process has successfully started, and source physical server has started into WinPE, but copying of source hard drive never started because of missing RAID drivers. So, now I was stuck into WinPE. Restarting the physical server was booting again into WinPE because P2V conversion was unable to finish, so modified boot sector was still in place.
Escaping from this (loop) situation was using installation media of Windows Server 2003, and booting into recovery console. Using fixboot has returned the default boot sector to the system partition, and I was able to boot again into source Windows Server 2003 R2 SP2 operating system.

Unable to delete VHD from VMM library

While browsing the VMM 2012 SP1 library, I found an obsolete VHD, and I wanted to delete it from there. But, when I tried to delete this obsolete VHD, VMM was complaining with following error message:

The library object (Name of the VHD) cannot be removed because following objects are dependent on it:

Virtual Hard Disk deployment configuration

In order to delete this VHD from VMM library I have opened table tbl_WLC_VHDConfig from VM database using SQL Management Studio. In this table there is a column named "SourceLocation". In this column we can find the ID of the offending VHD. But how to find the ID of VHD ? Using Get-VirtualHardDisk cmdlet you can list all the VHDs with their IDs:

Get-VirtualHardDisk | ft name,id

Now when we have the ID of the VHD, we can create new query from SQL Management Studio to delete the row containing VHD ID in SourceLocation column:

Delete From dbo.tbl_WLC_VHDConfig Where SourceLocation = 'VHD ID'

Note: Make a backup of the VM database before changing (deleting) rows from database, and this operation is not recommended by Microsoft, so perform at your own risk.

Unable to delete VHD from VMM library

I had VM template associated with VHD from VMM library, and I wanted to delete both of them, since they were used for testing purposes. I deleted the VM template successfully, and after that I wanted to delete the VHD, but VMM was not allowing to delete the VHD because temporary VM template was dependent of it. Here is the error message when I tried to delete the VHD:

And the dependencies from the properties of the VHD file:

Easiest way to delete this temporary VM template is using powershell:

Remove-SCVMTemplate -VMTemplate "Temporary TemplateName"

After deleting the temporary VM template, there were no dependencies of the VHD, and I deleted the file from library successfully.