VMM Service Crashes Repeatedly

In this case sysadmins were patching with latest firmware and drivers one of the Hyper V cluster hosts, due to unexpected server reboots. HPE support has recommended patching the server DL 380 Gen8 with latest firmware and drivers. So, latest PSP from HPE has been downloaded, and drivers and firmwares were installed.
Few hours later, after the installation of updated drivers and firmwares on the Hyper V host, VMM console of the VMM server which was managing the host has become unavailable. Also, System Center Virtual Machine Manager service on VMM server was terminating unexpectedly and event id 7034 was logged in system event log with information:
The System Center Virtual Machine Manager service terminated unexpectedly.  It has done this 3 time(s).
Since, the event doesn't say much about why the service was crushing, VMM debug logging has to be enabled using the following article https://support.microsoft.com/en-us/help/2913445/how-to-enable-debug-logging-in-virtual-machine-manager .
After enabling debug logging, the log was showing that error was generated whenever VMM was trying to reach the affected Hyper V server using WinRM and querying the WMI, for example (content truncated):
[Microsoft-VirtualMachineManager-Debug]4,4,WsmanAPIWrapper.cs,1913,WinRM: URL: [http:\\affected.hyperv.host]… 
[Microsoft-VirtualMachineManager-Debug]4,1,WsmanAPIWrapper.cs,3148,Retrieving underlying WMI error to throw. Got string ...
So, going back to the "updated" server and checking the update log, I've found that HPE Insight Management WBEM Providers were updated, and checked the following HPE article https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00053606en_us .
If Hyper-V is installed before Downgrade/Upgrade/fresh installation of HPE WBEM Providers, run the following steps after installing HPE WBEM Providers:
  • net stop vmms
  • mofcomp %SYSTEMROOT%\System32\WindowsVirtualizationUninstall.mof
  • mofcomp %SYSTEMROOT%\System32\WindowsVirtualization.V2.mof
  • net start vmms
This will restore the Msvm_ classes under root\interop namespace that had been overwritten by HPE WBEM Providers.
After recompiling the "original" mof files, VMM functionality was back, and VMM was able to query the affected Hyper V host.

I hope this article will save some of your "precious" admin time in debugging of this kind of combination of Hyper V, HPE Updates and VMM, and … Happy patching :)

No comments:

Post a Comment

VMM Service Crashes Repeatedly

In this case sysadmins were patching with latest firmware and drivers one of the Hyper V cluster hosts, due to unexpected server reboots. ...