Methods to speed up high file count transfers

[KCHughesIT]

I've just started a project to copy a massive amount of data between NAS/SANs and I need to come up with any tips/tricks to increase object transfer rates. I'm dealing with over 400 million files, about 100 million folders and over 70 TB of data. I understand that the shear volume of files will definitely slow this down. I have it set to copy across the /copy:dat level permissions, but not NTFS.

When transferring the the tiny files I'm getting sometimes only around 300-400 KB/sec which equates to about 25 objects/sec, whereas when I hit the occasional large files I can hit 50 MB/sec. Accross my data pool I seem to average a little over 2 MB/sec and 20 objects a second.

Any thoughts/suggestions on how to speed up the data transfer/copy?

Oh, I have the buffer set at 2GB. The machine I'm running it on only has 8GB total and I've seen the memory usage creep up and use nearly 6 (I'm guessing from the stored error/skipped log), so I'm hesitant to double that to the 4GB option.

[TGRMN Software]

Try the option "Use Windows Native Copy" in profile settings -> performance. You could also split the job into multiple profiles and run them concurrently, see: http://www.tgrmn.com/web/kb/item104.htm
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com