Help Please! Long standing Backup of VM unexpectedly failing

[GCK_Graham]

We have been successfully running backup of Server 2012 VM from Server 2012 Hyper V host for 3+years

Unexpectedly over the last week the backup has failed every night with the same error:

2020-08-25 01:01:28 : Copying file \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy9\WIS-FileNPrint\WIS-FileNPrint.vhdx <to> G:\VM Backups\WIS-File-Print_day3\WIS-FileNPrint\WIS-FileNPrint.vhdx (766.19GB) [Can not read data. Incorrect function.] [ERROR].

Having examined a detailed log for last nights run there appears to be no VSS errors or unexpected events.

There are no VSS errors logged on host or VM.

The above error is logged shortly after the shadow copies are completed and following 3 successful file copies - one from the same HarddiskShasowCopy.

The 3 other files are v small by comparison (2 x 40kb + 1x 92MB) and shown in the log as 'Overwriting...OK' so presumably complete

The backup then appears to continue for the next 3hrs 6mins until:

2020-08-25 04:07:34 : -- Done: 3 (92.08MB) Err: 1 (766.19GB) Skipped: 0 (0) Tot: 4 (766.28GB) --
2020-08-25 04:07:34 : -- Average Transfer Rate: 8.50KB/Sec -- Elapsed Time: 3 h 4 min 56 sec
2020-08-25 04:07:34 : Re-comparing ...
2020-08-25 04:07:34 : ---- Copying finished ----
2020-08-25 04:07:34 : ---- End ----

The elapsed time makes it appear that despite the error the backup is attempting to copy data?

The backup job has not been altered.

We are seeing Page File errors logged on the Host server at the time the backup ends (so 04:07 in the above example). The number varies from day to day.

Running Chkdsk on the host and the VM fines no disk errors. There are no hardware errors reported on the host server.

The physical server has been fully rebooted - so VMs shut down, host restarted but with no impact on the problem.

Need advice, please, on where to look next to diagnose this problem.

Thank you!

[GCK_Graham]

Really stuck on this and a week now with no backup of the VHDX file.

Anyone any suggestions on where too look?

Not even sure which file is failing to read...the shadow copy?

Thanks for any help available

[TGRMN Software]

Hi, are there any errors in the Event Viewer related to VSS?
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[GCK_Graham]

Hello!

There are no errors for VSS in either the guest or the host event logs.

We see usual 'surprise remove' - ID 157, preceded by 'Disk signature of disk 1 is equal to disk 0' - ID 58 within the VM. The timing of these match shadow copy timing and don't vary from when job ran successfully.

A detailed log for the VV job from earlier this week listed the writers, showed which had been excluded / included. None reported any error state.

A comparison between normal non detailed log for last success and a fail shows only difference is the sequence of writers listed - and the error line.

Only errors we see are paging errors (error 51) timed to coincide with the VV job ending, The VV Engine log of the failure and a matching error from VMMS noting failure- ID 16010

We have a separate Azure files backup running within the VM (has been for some time prior to this issue). Lately we have seen data errors on 3 files with associated block retry's logged being logged. Excluding these 3 files ensured the Azure backup runs cleanly no errors.

The Azure backup completes 2 hrs prior to the VM backup is scheduled.

As a result of the block retry's We suspect some corruption within VHD but chcdsk finds nothing.

VM so far running fine inc SQL DBs (note SQL instances are schedule stopped prior to shadow copy as originally these caused VSS issues.

We have explicit set permissions on VHDX file in host to account running the VV engine.

Any ideas please?