Multi-thread comparing

[Guest]

Start the 'indexing process' for the target and the source at the same time, probably using multiple threads. Now it takes too long to fetch the file structure of the first hard disk, and only when it's finished from the second.

[TGRMN Software]

This is a very good point. We'll check feasibility for one of the future releases, thanks for the input!
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[Adequat]

This espescially makes sense if source and targets are on different physical drives, then we can expect double speed.

[Guest]

The option for multi-threaded file-copy operations would also be of much use.

Especially when copying large files to a target network share over CIFS over gigabit ethernet for instance.

Doing a single-threaded copy usually results in speeds up between 200-300Mbit while the disks at the serving and receiving end can do alot more.

Doing multiple copy threads simultaneously usually gives greater overall network utilization resulting in a smaller timeframe needed to restore large backups.

[Guest]

It's been a while since I proposed this feature. I am still a grateful customer of your product, but every time I click the compare button it annoys me that it could have finished in half the time.

Are you making any headway on this?

[malega]

I've also been a long time user and like the product, but I am now getting frustrated with the failure to address this speed issue. It's been years since a major update of the software.

My files have increased significantly in size and it's getting to the point that VV2 is too slow in comparing. They have been promising for at least 2 years to address this issue, but they have not done so.

I am beginning to look for similar software that does not have this speed issue.

[TGRMN Software]

Hi, are you comparing files over the internet / VPN ? Is that causing the delay or actually copying the file? thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[tedge]

I understand what is meant by the time it take to compare. I use ViceVersa to move large amounts of data across a network as my storage capacity increases. As we speak, I am moving a 760gb directory between to NAS, it takes several minutes to index the source, then several to index the target if it is similar sized when it could be doing them at the same time...todays processors have multiple cores and handle multiple threads easily, a single threaded function comparing multiple targets is not staying with the times. I understand this issue but it is not the largest for me, as I said, I am moving very large directories regularly, when you have 760gb worth of data that must be moved, and moved correctly which means using CRC to check the data. Doing a CRC check on a 8gb file does not take much processor power but it does take a lot of time, stopping the transfer after each file to do the CRC check adds TONS of time to the transfer. There is plenty of processor to spare to continue the transfer while the CRC check completes and your software could re-que the transfer if the CRC checked invalid. That would greatly improve transfer times.

[gleep]

It is not so much about comparing as it is about the indexing that is done before the actual comparing begins. Either on the network or on a hard disk it takes time to retrieve the directory structure and file attributes, which are fed to the comparing algorithm. When you press 'Compare' it starts doing this, i.e. when it says 'Collecting File Information...'

If it would collect the file information from the two sources at the same time, this should cut the time needed for the whole process almost in half, as the file collection usually takes the most time when the files are small in contrast to comparing the attributes which is efficiently programmed. Processor speed would not be a problem, fetching the data is more a physical bottleneck of the source and the target device.

I don't quite understand why the first reply by the developers (second post) deems this as a 'very good point', only to give a new reply a year and a half later on Jun 12, 2009 which gives away that they don't really get the problem. Of course it must be difficult to fix, as I proposed this more than two years ago.

[onlyasking]

i wonder why they wont address this?

[TGRMN Software]

Hello

yes, collecting source and target info in parallel could be very beneficial in some cases. It would not really help to speed up collecting remote data over VPN.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[JuggalotusHeat]

So is this going to be worked in or not? If multi-connections/multi-threading was implemented it would speed up everything. Right now it takes forever and a day for my email backups to run because it does it on a per-file basis. If it gave me the option to have 10 connections at once I could max my pipe and get them pushed ASAP. Will this ever be implemented?

[wmfoster2001]

Comparing two 2Tb of small files on shares on different SANs is just far too slow.

If the ability to scan both trees at the same time had been included we would have purchased the software as the visual comparison feature is quite useful too us.

Regards,
Michael.

[Guest]

This still seems like a good idea so I thought I'd give it a nudge now there's some work going into version 3.

If the source and target are on different drives, please can you start the source and target scans in different threads? Sometimes a full CRC compare of a pair of drives can take me more than a day, so it would be nice to have a 2x speed improvement reading both drives at the same time rather than alternating between them. The directory read is fine for me, it's the CRC that takes the time.

Thanks!

[TGRMN Software]

Hi, thank you for the feedback, we are looking into this, to see if the target can be prefetched and if CRC comparison can be done at the same time for multiple files during the comparison phase.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com