File verification query

[NickF]

We need to move approx. 40TB of file data from two source Windows file servers to a NAS (source split is roughly 20TB on each existing file server). The plan is to start them synchronising and leave it running (likely several weeks), then once they are in sync we'll cut them over (disable use access to the source and run a final sync then change DNS & DFS to point at the NAS location).

We can't use Robocopy as we need to do file verification but I'm still unclear on how exactly the file verification is done when using Vice Versa.

Ideally we want a hash of each file as it's copied and once complete the source and destination file hash is compared to confirm it's identical. Can Vice Versa do this (automatically during the synchronisation), the FAQ seems to refer to doing post-copy manual verification (this wouldn't be feasible as that would likely take hours/days for each 20TB source and we won't have that amount of downtime available during cut-over).

We'll be using the VVengine as well (to detect changes and resync files that change over the several weeks the initial seeds will take), I assume if Vice Versa can do a per file copy hash check that this ability isn't impacted when it's triggered from VVengine?

[TGRMN Software]

Hi there are some options regarding file verification:

- at comparison time, specifying compariosn method Both (CRC + Size and Timestamp). Files that are already in source and target are compared using hash and verified that they are identical.

- there is the option to use SHA-256 instead of CRC (also in the comparison settings of the profile settings)

- you can also verify CRC as soon as files are copied using Verify copy operation by checking CRC: After each file copy operation is completed ViceVersa verifies CRC of target vs. source. Selecting this option will slow down the copying process, because each file has to be re-opened and re-read to be verified.

Probably the first two options are useful just before cut-off date. Not sure third option is required in your case.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[NickF]

Thanks for the quick reply - I'm still a bit unsure though!

So the first two options you mention are something you initiate manually once you have all the files copied across? If so my concern with that is how long it will take, from my experience of creating file hashes it would take hours/days for 20TB of files (tens of millions of them). Unless VV can create and store the file hashes as it goes and then it just compares the hash sets on request rather than having to generate them from scratch?

The 3rd option sounds like what we need, that's an option on the copy job? How is the verification confirmed (does it just log if there's a mismatch or is it possible to log source and destination checksum for future review)? It would be good, if a user reports file corruption/modification, if we have a log to go back through to confirm at time of copy it was fine.

The intention is to leave the copy/sync running for several weeks until everything has made it over. Obviously over that period of time many files will be changed and need to be re-copied, is it VVengine that's required as well to monitor the source and trigger another copy when a file is changed?

[TGRMN Software]

I think the option Verify copy operation by checking CRC is good to use so every time a file is copied it is verified.

Then you also have snapshots in ViceVersa.

See https://www.tgrmn.com/web/kb/item103.htm

When you are ready to cut-off you can create a snapshot of the source with all the hash info and use that snapshot to verify the target. It will take some time but it can be done.