VMM Service Crashes Repeatedly

In this case sysadmins were patching with latest firmware and drivers one of the Hyper V cluster hosts, due to unexpected server reboots. HPE support has recommended patching the server DL 380 Gen8 with latest firmware and drivers. So, latest PSP from HPE has been downloaded, and drivers and firmwares were installed.

Few hours later, after the installation of updated drivers and firmwares on the Hyper V host, VMM console of the VMM server which was managing the host has become unavailable. Also, System Center Virtual Machine Manager service on VMM server was terminating unexpectedly and event id 7034 was logged in system event log with information:

The System Center Virtual Machine Manager service terminated unexpectedly.  It has done this 3 time(s).

Since, the event doesn't say much about why the service was crushing, VMM debug logging has to be enabled using the following article https://support.microsoft.com/en-us/help/2913445/how-to-enable-debug-logging-in-virtual-machine-manager .

After enabling debug logging, the log was showing that error was generated whenever VMM was trying to reach the affected Hyper V server using WinRM and querying the WMI, for example (content truncated):

[Microsoft-VirtualMachineManager-Debug]4,4,WsmanAPIWrapper.cs,1913,WinRM: URL: [http:\\affected.hyperv.host]… 

[Microsoft-VirtualMachineManager-Debug]4,1,WsmanAPIWrapper.cs,3148,Retrieving underlying WMI error to throw. Got string ...

So, going back to the "updated" server and checking the update log, I've found that HPE Insight Management WBEM Providers were updated, and checked the following HPE article https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00053606en_us .

If Hyper-V is installed before Downgrade/Upgrade/fresh installation of HPE WBEM Providers, run the following steps after installing HPE WBEM Providers:

• net stop vmms
• mofcomp %SYSTEMROOT%\System32\WindowsVirtualizationUninstall.mof
• mofcomp %SYSTEMROOT%\System32\WindowsVirtualization.V2.mof
• net start vmms

This will restore the Msvm_ classes under root\interop namespace that had been overwritten by HPE WBEM Providers.

After recompiling the "original" mof files, VMM functionality was back, and VMM was able to query the affected Hyper V host.

I hope this article will save some of your "precious" admin time in debugging of this kind of combination of Hyper V, HPE Updates and VMM, and … Happy patching :)

VM saved state during backup from Hyper V host

In this case, Windows Server 2008 R2 was P2V (physical to virtual) converted, and hosted on Windows Server 2012 Hyper V cluster. This VM was having several vhds attached, and latest integration services available from Hyper V cluster were successfully installed and running.VMs running on this Hyper V cluster were backed up from Hyper V hosts. During the backup schedule this "new" VM, was the only one that was going into saved state during the backup, and services hosted on this VM were unavailable for several minutes. Backup of the VMs is VSS based, and there were no VSS errors.
In this situation, I've checked the backup solution documentation and found that this behavior of saving the state of the VMs is under jurisdiction of Hyper V, and not under control of the backup software.
So,  I've started digging the Hyper V logs and found something interesting: during the backup cycles there was warning event 4098 logged into Hyper V Integration event log:

 This event was logged for the VM that was going into saved state during backup. So, I've checked the VM's scheduled volume shadow copies, and found that shadow copies for volumes were stored on separated disk only for storing shadow copies.
After changing the scheduled volume shadow copies to be stored on same disk as data, the VM was successfully live backed up from the Hyper V host without saving state and without losing the VM's offered services during the backup.

Windows Server 2016 Hyper V requirements

Windows Server 2016 Hyper V has introduced great features (for more info, please checkout the official article What's new in Hyper-V on Windows Server 2016 ), but before jumping in and formatting the disk with previous version of Windows Server operating system with Hyper V installed, please make sure that your hardware has the needed requirements. The easiest way is to run systeminfo.exe from command prompt or Powershell, and checkout the Hyper V requirements part from the command output (for example: this is the output from supported hardware):

In my case, I was having one test box HP DL380 G5 with Windows Server 2012 Hyper V role installed, and the output from systeminfo.exe for Hyper V part, looked like this:

For Windows Server 2016 Hyper V role, Second Level Address Translation (SLAT) is requirement, instead recommendation as it was for Windows Server 2012 Hyper V. So, if you don't check these requirements and you try to install Windows Server 2016 Hyper V role, you might experience the following error message:

Conclusion: some old hardware boxes might not be able to see the "light" of the new Microsoft Windows Server 2016 with Hyper V role installed.

BSOD on VM after installation of Hyper V Integration Services

In this case, Windows Server 2003 R2 physical server was converted (captured) into guest VM on Hyper V cluster based on Windows Server 2012 operating system. Physical disk drives were successfully captured into VHDs. Virtual machine has booted successfully, and installation of hyper v integration services was initiated. Installation of integration services, was requesting reboot of the VM. After the restart, the VM was failing to boot with BSOD, and was complaining about WDFLDR.sys.
In order to fix this situation, I've booted the vm into last known good configuration. Checked the following registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Wdf01000: There was Group (REG_SZ) value and was having Base for the value data.
Changing the data into WdfLoadGroup :

 

 

And, installing the Hyper V Integration Services has fixed the issues. Installation of Integration services requested for restart, and the VM has rebooted successfully (without BSOD).

 

Event ID 5120 Hyper V 2012 Failover Clustering and DPM 2012 Backup Behavior

In this article I would recommend installing the KB 2879635 update for Windows Server 2012 based failover clusters that improves resiliency. This update should prevent the "notorious" event 5120 with description:

Cluster Shared Volume 'Volume1' ('name’) is no longer available on this node because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished.

from happening during backup of VMs from Hyper V host located on CSV volume. Note that after installing this hotfix on Hyper V hosts, you should update integration components on Windows Server 2012 based guest virtual machines running on those hosts.

Also, If you're using DPM 2012 SP1 as you're backup solution I would recommend installing the latest rollup 3 KB2836751 and hotfix KB2886362 .

From my point of view, before installing these updates and rollups I was experiencing strange behavior when I was doing Hyper V host based backup of Windows Server 2012 virtual machines using DPM 2012 SP1. After initial replica creation of vm with installed Windows Server 2012 which is basically transferring the whole vhd(x) file to the DPM 2012 SP1 server, all other scheduled recovery points were transferring again the same size of data to the DPM server. This is an example of backup of one Windows Server 2012 vm:

 

After installing all the rollups and updates, the situation has changed, and the DPM scheduled replica creation task was transferring significantly smaller amount of data, and naturally the backup task was completing much faster.

Also, you will eliminate memory spike on node which is owning the CSV resource, during backup of VMs located on that volume fixed with KB2813630 which is included in KB 2879635.

 

Microsoft virtualization certification exam for free

Sign up for free MVA online training on November 19 & 20 – then receive a voucher to take the new Microsoft virtualization certification exam for free!

For more info please check http://www.virtualizationsquared.com/ .

The Parallel port driver service failed to start

After migrating virtual machines from Virtual Server 2005 to Hyper-V, the following error message was prompted during start-up:

 

 

In the system event log the following event is logged:

 

Event Type: Error

Event Source: Service Control Manager

Event Category: None

Event ID: 7000

Description:
The Parallel port driver service failed to start due to the following error:
The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Same event is logged after P2V migration of servers, in my case it was Windows Server 2003.

In order to prevent this error from showing, I have disabled Parport service from registry, by setting the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Parport\Start to 4.

 

The following values for Start parameter are possible:

 

2	Automatic. This is the default setting.
3	Manual.
4	Disable.

 

Your VM running on Hyper V is not in sync (time)

I have experienced something strange on Hyper V cluster based on Windows Server 2008 R2 SP1 Enterprise and VM guest with same OS but Standard edition, with time synchronization enabled but VM was still out of sync.

I have checked that Windows time service was running and queried the source for synchronization and it was free running system clock !

After restarting the windows time service, the VM started to sync with local CMOS clock instead of synchronizing with host (parent partition).

I'm guessing that VM was not reading the setting for the time synchronization from the VM configuration, so I have disabled the setting and re enabled and restarted the windows time service, and finally within few seconds the VM was synchronizing the time with host :

One more thing : if the server is Windows 2008 R2 and the machine is not domain joined the windows time service will stop automatically. On reboot you can see the following event :

This behaviour is by default and you can find more info on translated Japanese kb :
http://translate.google.com/translate?hl=en&sl=ja&u=http://support.microsoft.com/kb/2385818/ja