Event ID 5120 Hyper V 2012 Failover Clustering and DPM 2012 Backup Behavior

In this article I would recommend installing the KB 2879635 update for Windows Server 2012 based failover clusters that improves resiliency. This update should prevent the "notorious" event 5120 with description:

Cluster Shared Volume 'Volume1' ('name’) is no longer available on this node because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished.

from happening during backup of VMs from Hyper V host located on CSV volume. Note that after installing this hotfix on Hyper V hosts, you should update integration components on Windows Server 2012 based guest virtual machines running on those hosts.

Also, If you're using DPM 2012 SP1 as you're backup solution I would recommend installing the latest rollup 3 KB2836751 and hotfix KB2886362 .

From my point of view, before installing these updates and rollups I was experiencing strange behavior when I was doing Hyper V host based backup of Windows Server 2012 virtual machines using DPM 2012 SP1. After initial replica creation of vm with installed Windows Server 2012 which is basically transferring the whole vhd(x) file to the DPM 2012 SP1 server, all other scheduled recovery points were transferring again the same size of data to the DPM server. This is an example of backup of one Windows Server 2012 vm:

 

After installing all the rollups and updates, the situation has changed, and the DPM scheduled replica creation task was transferring significantly smaller amount of data, and naturally the backup task was completing much faster.

Also, you will eliminate memory spike on node which is owning the CSV resource, during backup of VMs located on that volume fixed with KB2813630 which is included in KB 2879635.

 

DPM encountered a retryable VSS error

In this case I was trying to backup P2V converted VM using DPM 2012 SP1. The VM was hosted on Hyper-V 2012 cluster. DPM Replica Creation Job was failing with following description:

DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer experienced a transient error.  If the backup process is retried, the error may not reoccur. (0x800423F3))

The VM had Windows Server 2003 SP2 operating system installed. I've decided to check the VSS writers, and the vssadmin has returned empty list of VSS writers available on the VM:

According from following Microsoft article http://support.microsoft.com/kb/940184/en-us , I've re-registered the following dlls:

cd /d %windir%\system32

net stop vss

net stop swprv

regsvr32 ole32.dll

regsvr32 oleaut32.dll

regsvr32 /i eventcls.dll

regsvr32 vss_ps.dll

vssvc /register

regsvr32 /i swprv.dll

regsvr32 es.dll

regsvr32 stdprov.dll

regsvr32 vssui.dll

regsvr32 msxml.dll

regsvr32 msxml3.dll

regsvr32 msxml4.dll

Now, running the vssadmin list writers again, has successfully listed all the available VSS writers. I was hoping that DPM will successfully finish Replica Creation Job, but I was wrong. VM was having ISA Server 2006 installed with local MSDE. Someone has changed ISA Logging destination from Database into flat files, and to free some space on disk deleted the mdfs and ldfs files from logging directory. Now, VSS was failing with error event id 6013 with following message:

Error message: Database 'ISALOG_.....' cannot be opened due to inaccessible files or insufficient memory or disk space. See the SQL Server errorlog for details

 

Now I was looking for a way to dismount those old databases with deleted mdf and ldf files. Thankfully, Microsoft has publish a vbscript for dismounting old databases from MSDE, and here is the link from the source of the script http://technet.microsoft.com/en-us/library/cc302448.aspx .
After running the script, the old databases were dismounted, and finally I was able to backup the VM using DPM 2012.

Unable to install DPM 2012 SP1 Agent

I was trying to install DPM 2012 SP1 agent using push method from DPM server and installation job was failing with following error on DPM server:

Agent operation failed. (ID 370)

An error occurred when the agent operation attempted to create the DPM Agent Coordinator service on protected server. (ID 347 Details: The service did not respond to the start or control request in a timely fashion)

On protected server two sidebyside events with event ID 33 were logged in Application Log:

Activation context generation failed for "d:\fbe53baf0ccc85b5405e430f\1033\SetupLaunchScreen.DLL". Dependent Assembly Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.21022.8" could not be found. Please use sxstrace.exe for detailed diagnosis.

Activation context generation failed for "d:\fbe53baf0ccc85b5405e430f\1033\SetupLaunchScreen.DLL". Dependent Assembly Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.21022.8" could not be found. Please use sxstrace.exe for detailed diagnosis.

 Manual installation of DPM agent is failing with following error message and same two SidebySide events were logged on protected server's application event log:

 



To resolve this behavior, first I've installed Microsoft Visual C++ 2008 Redistributable Package on protected server and then re-run the setup for DPM 2012 sp1 agent. After installing the Visual C++ 2008 redistributable package, DPM 2012 SP1 agent has installed successfully.

DPM 2012 SP1 Scheduled jobs disapear without starting

In this case DPM Server 2012 SP1 installed on Windows server 2012 with remote database on SQL Server 2012 SP1 installed on Windows Server 2012, scheduled jobs for protection groups were disappearing without starting job. Dedicated instance of SQL Server was installed just for DPM 2012 as required. This instance of SQL Server was prepared using DPM Remote SQL Prep. Setup of DPM 2012 SP1 has finished successfully without any issues.
Manually triggering backup was successfully finishing, but scheduled jobs were just disappearing without starting the task.
Following event 208 is logged in application log on SQL Server from source SQLAgent$InstanceName :

The job failed. The Job was invoked by Schedule 13 (Schedule 1). The last step to run was step 1 (Default JobStep).

From Job Activity Monitor can be seen that non-sysadmins have been denied permission to run CmdExec job step:

 

 

SQLAgent for DPM instance was running under Local System account, so adding the sysadmin role to the Local System account has solved the problem.