NAS to NAS copy at the highest speed

[Alice]

Hi

I have to copy approx. 20TB from one NAS device to another ('Source' is full, 'Destination' is empty) over Gigabit Ethernet to complete a hardware upgrade.

Is there any combination of settings I can choose to achieve the max speed in VV (2.5 - 2517) ?

So far I have set :

- two servers up W2008 R2 to copy different sets of data simultaneously
- the 'Process priority' to 'high'
- 'Process Speed (% of Max)' to 'Max'
- 'Copy Buffer Size' to '8388608'
- 'Copy Permissions' as 'Copy file/folder' perms

I wondered whether checking 'Copy directly to destination file (do not use a temp file)' would speed things up a little. Is there a data risk associated with this option ? Bear in mind the 'Destination' NAS is empty and no staff are using it at present.

Tks

[TGRMN Software]

Hi,
Sometimes the option Use Windows native file copy can help.

Definitely copying directly to dest file without using a temp file would help a bit, this option is only useful when overwriting existing files or for sync.

Most improvement you could have by running 3 or 4 profiles simultaneously, e.g. each one dealing with a different set of folders to copy to target.

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[Decibel]

This is exactly how I have set up my NAS to NAS backup.

The option Use Windows native file copy was the biggest improvement in speed.

[Alice]

Thanks both

@ Decibel - I can't engage 'Use Windows Native File Copy' because I'd also like to select 'Preserve last accessed timestamp of files being copied', the two won't work together.

I'll post an update when I finish the first batch of copies, around 800GB of 20TB.

[Alice]

In the end I created 5 Profile Jobs to move 930 User 'Home' folders - firstname.lastname (747GB in total after I removed 'inactive' staff folders ie 'not modified' for > 90 days) to the new NAS.

I created the jobs by exporting the folder structure to Excel, totalling the staff folders by the first letter of each person's firstaname, so users starting with letter 'A' collectively accounted for 'A'= 49GB, 'B'=32GB, 'C'=37GB etc. Then I divided the 747GB total data 'footprint' by 4 to give me a split of 4 Groups in order to create 4 evenly(ish) sized Profiles by GB - I know there’s an argument for using ‘total files’ rather than ‘GB’ but you have to make a judgement on that ! The 5th Group (Profile) would be the remainder of the data left after creating the 4 even(ish) groups (by GB).

Using an Excel formula I ‘summed’ each letter in sequence 'A' , 'B', 'C' then 'D' until the cumulative total reached as close to 186GB (747 divided by 4) as it could, this would be my 'range' for Profile 1 (include '\[a-f]*' subfolder filter), this was marked in the S/S via conditional formatting - then the process continued adding from letter 'G' until the cumulative sum by letter reached as close to 186GB as possible, this became my range for Profile 2 (include ' \[g-j]*' subfolder filter) and so on.

I ended up with 4 Profiles of 164GB, 177GB, 186GB and 185GB, so pretty closely matched, then a 5th Profile of the remainder which was 36GB.

The Profiles were run on two different servers, Profiles 1 and 2 on the first server and 3,4 and 5 on the other server. The VV ‘Comparisons’ were fast as it only compares in the subfilter range, so [a-f], [g-j] etc which speeds things up too 

Finally, although I used VV to replicate the NTFS permissions initially, in order to deal with NTFS changes thereafter (which VV does not update unless there are file changes too) I avoided using ‘robocopy /secfix’ which is mentioned a few times on this forum, I used ‘icacls /save’ and the ‘icacls /restore’ to synchronise the permissions immediately before the new NAS structure went ‘live’ to the Customer.

[jordandevin]

NAS=Network Attached Storage. Servers were inherently designed to be on a network and "serve" so your acronym would be redundant.

What NAS do you have? Or what OS is your NAS running? You should look into rsync. Rsync will sync files between multiple location, copying only the files which are different. If the process fails, the next run of rsync will resume from where it left off as it only does delta copy.

[Alice]

Well this is enterprise data management, large scale by most company standards.

Both 'NAS' devices are IBM - n3240 being replaced by V7000, maybe better to describe them as hybrid storage systems You could visit the IBM site if you’d like further info.

Very happy with the VV solution described, but thank you for your advice.

[TGRMN Software]

That's a great setup Alice!
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com