Buffer Size & Speed Optimization

[Ben]

I have read several posts and the help text regarding the buffer size in ViceVersa Pro which all indicate that it can be changed to help speed up file transfers, but I have not been able to find any suggestions about what it should be changed to. Maybe someone from TGRMN could post a few suggestions here for different situations? I have come up with a list of different types of transfers that might require unique settings:

1) File Share - contains mostly small, highly compressible word documents & excel spreadsheets

2) Music Share - contains mostly 5-10MB, uncompressible MP3 files

3) ISO Share - contains mostly large 300-700MB ISO image files

4) Software Share - contains a mix of small & large files which for the most part will be compressed executables, but could contain just as many or more other files (such as clip art for ms office)

5) Photo Share - very similar to a music share, but the files will be a lot smaller 700kb to 1MB on average

Also, should the buffer size be changed for different types of connections? Some examples would be:

a) Local - disk to disk

b) USB - disk to USB

c) NAS - disk to NAS

d) WAN - disk to disk over WAN connection

e) LAN - disk to disk over LAN connection (similar to NAS)

f) Gigabit LAN - disk to disk over 1,000Mbps LAN

I think that this information would be useful to many users who are trying to configure ViceVersa Pro to be as fast as possible. It would also be nice to know if there are any other big performance enhancers (such as turning off logging or something like that which would allow ViceVersa to perform faster). It sounds like there are a lot of people that are trying to sync up large amounts of data 100GB+ and that type of transfer needs to be as clean as possible in order for it to run smoothly.

[TGRMN Software]

In general the settings should be 'flush the buffer' and small copy buffer size for WAN. Otherwise leave the defaults for LAN or local connections. There are no specific settings by file type; it is the type of connection between source and target that counts.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[Ben]

Unfortunately you simply proved my point. You seem to refuse to give any specific suggestions and I have no idea what "small" is....does any one else? And excuse me if I am incorrect, but isn't a gigabit connection different than a 100Mbps connection and a 10Mbps connection just like they are different from a 1Mbps WAN link? Should the settings really be the same for a local SCSI or SATA hard drive as they are for a LAN connection? What about USB and Firewire? Maybe you could give us a ratio to use (i.e. X Mbps = X Buffer Size) so that we can calculate the buffer size our self since you are unwilling to do so for us?

You also neglected to answer my question about other ways to optimize your product. It seems obvious to me that there is some overhead in the way your program works that could shave a significant amount of time off of large transfers but I have no idea what I as a user can do to help optimize it. People don't have a floppy disk worth of files anymore...they have 40GB music shares and 100GB video shares which need to be replicated as quickly and efficiently as possible.

Please help us use your product....

[TGRMN Software]

You are right. My previous post was not very helpful. Sorry Let me now try to provide more specific suggestions.

a) Use 'Flush the buffer' option only for < 2Mbps WAN links. This option causes VV to flush the cache every write cycle and therefore slows down the overall copy operation. But for a < 2Mbps WAN link this is needed to make the program more responsive.

b) 'Copy directly to destination file'. Using this option may speed things up in the range of 5%. It is not recommended to use this option unless really necessary. It is always better to use a temp file to copy files. The reason is that if the connection is lost during a copy the temp file will be deleted and the original destination file is still be available. When overwriting the target this is not the case.

c) Most important option! The buffer size. This can really make a difference. Use a very large buffer size (> 200.000 bytes) for 'local' connections eg USB drives, SCSI, EIDE, disk to disk etc. Leave the default 32768 for LAN and WAN (100Mbps , 10Mbps , 1Mbps, etc). Different values can be tested to adapt to specific LAN, WAN connections, but never a value above 100000.

d) Set speed to the max

Hope this helps

By the way - In the new release of ViceVersa we are planning to add a 'self adapting' copy buffer size.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[javierdc]

I want to copy 50gb from one USB 2.0 drive to a second USB 2.0 drive. What should be a good transfer rate? I know depends per system but in mine seems slow so I was wondering what other people is getting.

Thanks - Javier

[CoreyPlover]

javierdc,

I regularly copy from local hard drive to USB hard drive and my observations are as follows:

If you have a good USB 2.0 connection then your transfer rates will be limited to your USB hard drive write speed. I have not seen any USB hard drives that can write at more than 20 Mb per second, even in benchmarking tests and reviews online.

I can regularly get 5 Mb per second on my workstation (10 Mb per second for large files), but was commonly getting upwards of 15 Mb per second when plugging the same USB hard drive into my server (SATA mirrored RAID drives). I suspect that either SATA or RAID handled the small files better, resulting in a faster average speed.

If you regularly get under 5 Mb per second, you could try updating your USB port. I have seen ports rated as "USB 2.0" but they are not that fast. Proper onboard USB 2.0 "enhanced" ports should be able to support the 15 Mb per second that I regularly get. Also, a lot of older USB hard drives and almost all USB keys may only have a maximum 8 Mb per second write speed anyway.

Hence, USB 2.0 bus speeds, even network speeds are not the limiting factor. The hard drive write speed usually determined your average speed. For USB to USB I would expect sustained speeds of at least 5 Mb per second. You should average, maybe, 8 Mb per second.

SIDE NOTE: I currently use a 2.5" IDE laptop USB drive, although I have used 3.5" IDE drives as well. I have always used USB caddies and inserted my own standard IDE drive and have just recently moved to using USB powered 2.5" hard drive caddies. They are remarkably portable and PERFECT for file synchronisation with ViceVersa.

[TGRMN Software]

When copying multiple files, the size of individual files makes a huge difference. Copying many small files (e.g. 10000 files of 10K each) takes much longer than a few large files (e.g. 100 of 1000K each), even if the total file size is the same.
This is due to the fact that creating a new file, giving it a name, timestamp it, allocationg file space on disk, closing it, all of this is the slowest part of the copy process and it is repeated many times for small files. So the fewer the files to copy the faster the copy process will be.

Beside this, there is also a HUGE difference in performance between USB 2.0 drives.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com