extended VSS copy of hyperv machines is slow

[vn]

I'm copying two hyperv machines from one volume to another and it is super slow, about 426KB/s. These two machines are just single volumes about 36GB and 11GB each. It takes just a couple minutes to generate the shadow copy but then the actual VVP copy takes many hours (current estimate is 30+ hours).

These machines use very little processing time and very low disk I/O as they were just test machines for now. The source drives are 4x15K SAS in RAID5 and the destination volume is also the same configuration. The are different RAID volumes on separate channels. I've even tried to iSCSI target with the same result. Just using copy and past of a 3GB file in Windows Explorer takes about 10 seconds so I know the drives are indeed performing well.

I'd appreciate some assistance in troubleshooting this. Thanks!

[TGRMN Software]

Hello

are you using ViceVersa PRO 2.5 build 2501? Check the Process Priority in the profile settings -> performance, is it set on "Default"?

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[vn]

Yes, I kept playing around with it and found that if I set process priority to 'normal' performance was high, as expected. Thanks! However, I now have a new problem. The host server crashes during snapshot creation. As you can see, the job runs for 27 seconds and then the server crashes and is automatically rebooted. Any suggestions?

2010-09-22 05:22:00 :
2010-09-22 05:22:00 : -
2010-09-22 05:22:00 : START ID: 201092212220-490719-4396
2010-09-22 05:22:00 : ************************************************************
2010-09-22 05:22:00 : Profile Started.
2010-09-22 05:22:00 : ************************************************************
2010-09-22 05:22:00 : Version: ViceVersa PRO 2.5 Build 2501 64-bit (SN229) [Registered][Server][VSS][Kevin-Clar...]
2010-09-22 05:22:00 : Command Line: "c:\program files\viceversa pro 2\viceversa.exe" "e:\it\settings\hyperv.fsf" /hiddenautoexec /wait
2010-09-22 05:22:00 : System: , (64 Bit Edition), (HyperV), (x86-64 Processor) v6.0 Build:6001 Service Pack:1
2010-09-22 05:22:00 : User: PACFIELD.LOCAL\Backup
2010-09-22 05:22:00 : Computer: srv-storage.pacfield.local
2010-09-22 05:22:00 : **** Profile: e:\it\settings\HyperV.fsf
2010-09-22 05:22:00 : **** - Folders -
2010-09-22 05:22:00 : **** Targets: (Archived)
2010-09-22 05:22:00 : **** "vss:\\Microsoft Hyper-V VSS Writer:Backup Using Child Partition Snapshot\srv-tad1" <to> "K:\hyperv\" (Include Subfolders)
2010-09-22 05:22:00 : **** "vss:\\Microsoft Hyper-V VSS Writer:Backup Using Child Partition Snapshot\srv-web1" <to> "K:\hyperv\" (Include Subfolders)
2010-09-22 05:22:00 : **** - Comparison Settings -
2010-09-22 05:22:00 : **** Comparison Type: Size and Timestamp
2010-09-22 05:22:00 : **** - Execution Settings -
2010-09-22 05:22:00 : **** Method: Backup (Mirror Source to Target)
2010-09-22 05:22:00 : **** No overwrite / read-only / error confirmations
2010-09-22 05:22:00 : **** Log File: "C:\ProgramData\ViceVersa PRO 2\log3.log" (max 10000 KB)
2010-09-22 05:22:00 : **** Log only summary and errors
2010-09-22 05:22:00 : **** No archive for deleted/replaced source files
2010-09-22 05:22:00 : **** Archive deleted/replaced target files to: "K:\hypervarchive\"
2010-09-22 05:22:00 : **** Move files to archive folders (instead of copying)
2010-09-22 05:22:00 : **** Archive max. 20 copies per file
2010-09-22 05:22:00 : **** Process Priority: Normal
2010-09-22 05:22:00 : **** Use 'Volume Shadow Copy' to copy open files for source
2010-09-22 05:22:00 : **** Create Volume Shadow Copy before comparing source and target
2010-09-22 05:22:00 : **** Send notification e-mail after execution to vn@pacfield.com from vvp@pacfield.com. Use SMTP server pfssbs on port 25 (Send e-mail on execution success or error)
2010-09-22 05:22:00 : Initializing Volume Shadow Copy ...
2010-09-22 05:22:00 : Creating Volume Shadow Copy for source volume(s)...
2010-09-22 05:22:00 : Gathering writer metadata...
2010-09-22 05:22:21 : Initializing writer metadata...
2010-09-22 05:22:22 : Listing writer metadata ...
2010-09-22 05:22:22 : Selecting VSS components ...
2010-09-22 05:22:22 : * Writer 'ASR Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'Microsoft Hyper-V VSS Writer' is included:
2010-09-22 05:22:22 : - Include component \7619875A-E1DE-4F3F-B32D-D7807C29E64B, caption: 'Backup Using Child Partition Snapshot\srv-tad1'
2010-09-22 05:22:22 : - Include component \A2AD1BDB-17D5-41BA-83C1-A68D9F9467D9, caption: 'Backup Using Child Partition Snapshot\srv-web1'
2010-09-22 05:22:22 : - Exclude component \Initial Store, caption: 'Initial Store'
2010-09-22 05:22:22 : * Writer 'System Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'Shadow Copy Optimization Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'FRS Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'COM+ REGDB Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'WMI Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'Registry Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'BITS Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'TermServLicensing' is excluded.
2010-09-22 05:22:22 : * Writer 'IIS Config Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'IIS Metabase Writer' is excluded.
2010-09-22 05:22:22 : * Writer 'NTDS' is excluded.
2010-09-22 05:22:22 : Creating shadow set {643ccda7-5e6e-4534-8858-6470d579c52f} ...
2010-09-22 05:22:22 : Adding volume \\?\Volume{8856231b-bba0-11df-9aa6-001e4f39a1bf}\ [D:\] to the shadow set...
2010-09-22 05:22:27 : Creating the Volume Shadow Copy...

[TGRMN Software]

Hello,

activate the option Log Level - detailed , it's in profile settings -> advanced settings -> VSS details

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[vn]

Okay, I've enabled detailed logging as specified. It's scheduled to run Monday early morning so I'll post the detailed logs then. Thanks for your support.

[vn]

Here is the result of vssadmin list writers in case it helps . . . probably not as the are all listed as stable with no error.

Microsoft Windows [Version 6.0.6001]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.

C:\Users\administrator.PACFIELD>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {e4e233d7-a723-4144-8518-8e03b772c3a8}
State: [1] Stable
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {a0c73207-722a-486c-bcfb-26d774deaf6e}
State: [1] Stable
Last error: No error

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {b65dee75-5c79-448f-9a7b-e98de3f56e4c}
State: [1] Stable
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {1fb2dcd9-18dd-402f-9ec4-44d1173791a6}
State: [1] Stable
Last error: No error

Writer name: 'TermServLicensing'
Writer Id: {5382579c-98df-47a7-ac6c-98a6d7106e09}
Writer Instance Id: {cc1daa19-0bb2-4536-8c53-4f9dae56aa78}
State: [1] Stable
Last error: No error

Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {1a038caf-d46b-46ce-a49c-d85fa437cbda}
State: [1] Stable
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {65792798-1c9c-4cd5-8bc7-740fce16081e}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {f06c26c0-fe5a-4f8b-8c8f-bf00907cab0a}
State: [1] Stable
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {c09b16bd-cbc5-4f1d-bd3b-e74f79827480}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {296b3338-2da1-4006-924b-c463be6a979f}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {fd94f033-17d2-46d7-8c2b-5d1ef712f145}
State: [1] Stable
Last error: No error

Writer name: 'FRS Writer'
Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
Writer Instance Id: {80a6e843-a29c-440f-b3bf-8f5357dfafa9}
State: [1] Stable
Last error: No error

Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {013c6254-13bb-4a71-8745-82ccae2c47ab}
State: [1] Stable
Last error: No error

[TGRMN Software]

Thanks, we'll check the log, the vssadmin list looks fine.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[vn]

I have the log now. You can download it here:

it.vitalnetworks.com/log3.zip

The server continues to crash and reboot on these backups.

[TGRMN Software]

Hello

It looks like a problem with the hyper-v writer.

Check the following:

1. Hyper-V Integration components are installed and running the latest version

NOTE: (On the Host/Parent partition you can check VMMS.exe = 6.0.6001.22352 (or newer) and in the guest, check vmbus.sys version 6.0.6001.22334 (or newer)

2. No Dynamic disks inside the guest.

3. All volumes are NTFS.

4. All NTFS volumes must be >1GB and have >300MB free space.

5. Shadow copies within the Virtual Machine are on the same volume or are Disabled

6. Virtual Machine is in running state.

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com