ViceVersa stuck "preparing display..."

ASPweb · Joined: 30 Jul 2015 Posts: 4

We are trying to backup 16.1 million files containing 645GB of data from one server to another. It took just over 3 hours to scan the source disk and almost another hour to compare 16.1 million files on the source to 21k files on the target, then it was stuck for just over an hour with some message about preparing a display, with instruction to switch to a flat panel display that couldn't be followed because the menu items did not exist.

We really need this product to work for us, but I must say, getting started has been a frustrating experience, particularly when popups like "preparing display" are hung for more than an hour. Had we been impatient, one might have aborted prematurely.

For your help, here are the logs while we wait yet again for the execution of the file copy to begin.

2015-07-30 16:01:38 :
2015-07-30 16:01:38 : -
2015-07-30 16:01:38 : ************************************************************
2015-07-30 16:01:38 : Profile Started.
2015-07-30 16:01:38 : ************************************************************
2015-07-30 16:01:38 : Version: ViceVersa PRO 2.5 Build 2515 64-bit (SN392)
2015-07-30 16:01:38 : Command Line: "C:\Program Files\ViceVersa Pro\ViceVersa.exe"
2015-07-30 16:01:38 : System: Windows Server 2003, (Standard Edition), (64 Bit Edition), (x86-64 Processor) v5.2 Build:3790 Service Pack:2
2015-07-30 16:01:38 : User: xxxx
2015-07-30 16:01:38 : Computer: xxxx
2015-07-30 16:01:38 : **** Profile: d:\viceversa\files.fsf
2015-07-30 16:01:38 : **** - Folders -
2015-07-30 16:01:38 : **** Source: "D:\files\"
2015-07-30 16:01:38 : **** Target: "\\xxxx\D$\Files\"
2015-07-30 16:01:38 : **** Include Subfolders: Yes
2015-07-30 16:01:38 : **** Create sources/targets that do not exist
2015-07-30 16:01:38 : **** - Comparison Settings -
2015-07-30 16:01:38 : **** Comparison Type: Size and Timestamp
2015-07-30 16:01:38 : **** - Execution Settings -
2015-07-30 16:01:38 : **** Method: Backup (Mirror Source to Target)
2015-07-30 16:01:38 : **** No overwrite / read-only / error confirmations
2015-07-30 16:01:38 : **** Tracking Database: "D:\ViceVersa\vvdb1.tdb"
2015-07-30 16:01:38 : **** Do not rescan source
2015-07-30 16:01:38 : **** Log File: "D:\ViceVersa\log1.log" (max 10000 KB) (empty log file each run)
2015-07-30 16:01:38 : **** Log only summary and errors
2015-07-30 16:01:38 : **** No archive for deleted/replaced source files
2015-07-30 16:01:38 : **** No archive for deleted/replaced target files
2015-07-30 16:01:38 : **** Process Priority: High
2015-07-30 16:01:38 : **** Use Windows native file copy
2015-07-30 16:01:38 : **** Network friendly folder scanning
2015-07-30 16:01:38 : Using network friendly folder scanning.
2015-07-30 16:01:38 : Checking source D:\files\ ...
2015-07-30 16:01:38 : [INFO] Rescanning needed for source folder D:\files\. Tracking database does not contain no-rescan info yet.
2015-07-30 19:09:51 : Total found: 16,389,181
2015-07-30 19:16:34 : Checking target \\xxxx\D$\Files\ ...
2015-07-30 19:16:35 : Total found: 21,982
2015-07-30 19:16:36 : Comparing ...
2015-07-30 19:57:31 : Comparing done.
2015-07-30 21:01:49 : - Status Summary -
2015-07-30 21:01:49 : - Total Files - Source: 16,156,212 (645.13GB) - Target: 21,473 (4.61GB)
2015-07-30 21:01:49 : - Excluded Files - Source: 0 (0) - Target: 0 (0)
2015-07-30 21:01:49 : - Matched Files - Source: 21,456 (4.49GB) - Target: 21,456 (4.49GB)
2015-07-30 21:01:49 : - Single Files - Source: 16,134,739 (640.52GB) - Target: 0 (0)
2015-07-30 21:01:49 : - Newer Files - Source: 17 (120.96MB) - Target: 0 (0)
2015-07-30 21:01:49 : - Older Files - Source: 0 (0) - Target: 17 (116.84MB)
2015-07-30 21:01:49 : - Total Subfolders - Source: 232,969 - Target: 509
2015-07-30 21:01:49 : - Excluded Subfolders - Source: 0 - Target: 0
2015-07-30 21:01:49 : - Matched Subfolders - Source: 509 - Target: 509
2015-07-30 21:01:49 : - Single Subfolders - Source: 232,460 - Target: 0
2015-07-30 21:01:49 : Saving tracking database...
2015-07-30 21:02:43 : Tracking database saved.

Just as this topic was being created, the process of preparing the display ended, but there is still some explanation needed to understand why these processes take so long.

The server has two Intel E4750 2.0Ghz cores and 8GB of RAM with a 48 disk RAID10 SAN with over 6,000 IOPS available. While the processors were reach 70-80% use, there was still plenty of performance available.

Please advise.

ASPweb · Joined: 30 Jul 2015 Posts: 4

Since I made my last post, with the source and target panes being displayed and having click execute, the system has been churning away at the CPU, all of the 8GB of system RAM and not a single file has been copied to the target server.

I really don't understand what could be happening. We already scanned the source file system of 16.1 million files, compared to the target server, created the vvdb database, so why is this program using CPU and RAM and not copying a single file?

2015-07-30 19:57:31 : Comparing done.
2015-07-30 21:01:49 : - Status Summary -
2015-07-30 21:01:49 : - Total Files - Source: 16,156,212 (645.13GB) - Target: 21,473 (4.61GB)
2015-07-30 21:01:49 : - Excluded Files - Source: 0 (0) - Target: 0 (0)
2015-07-30 21:01:49 : - Matched Files - Source: 21,456 (4.49GB) - Target: 21,456 (4.49GB)
2015-07-30 21:01:49 : - Single Files - Source: 16,134,739 (640.52GB) - Target: 0 (0)
2015-07-30 21:01:49 : - Newer Files - Source: 17 (120.96MB) - Target: 0 (0)
2015-07-30 21:01:49 : - Older Files - Source: 0 (0) - Target: 17 (116.84MB)
2015-07-30 21:01:49 : - Total Subfolders - Source: 232,969 - Target: 509
2015-07-30 21:01:49 : - Excluded Subfolders - Source: 0 - Target: 0
2015-07-30 21:01:49 : - Matched Subfolders - Source: 509 - Target: 509
2015-07-30 21:01:49 : - Single Subfolders - Source: 232,460 - Target: 0
2015-07-30 21:01:49 : Saving tracking database...
2015-07-30 21:02:43 : Tracking database saved.
2015-07-30 21:12:48 :
2015-07-30 21:12:48 : ---- Start Execution ----
2015-07-30 21:12:48 : ---- Backup (Mirror Source to Target) (Source: D:\files\ Target: \\xxxx\D$\Files\) ----
2015-07-30 21:12:48 : ---- Start copying ----

It's now 22:54 server time... seriously, what is this software doing? Rescanning? Is so, why?

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8763

Hello,
I would recommend that you split the job into multiple profiles, see:

http://www.tgrmn.com/web/kb/item104.htm

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

ASPweb · Joined: 30 Jul 2015 Posts: 4

Thanks for the recommendation, however, this doesn't really make sense to me.

Why would I want multiple profiles to run when one profile is already using up too much processing and RAM?

It's not practical to do this because my client has an images directory that contains tens of thousands of customerID folders with their images. I can't easily split them up into jobs/profiles, even using a RegEx style parameter as you show in your example. I would have to create hundreds of jobs/profiles.

I also don't understand why your program can't keep track of the files with one profile using your database. I have tried repeatedly to get the database populated and use it with the norescan option and it simply can't handle this many files. If there is some secret setting to make it work, I would appreciate knowing about it.

Fortunately, I am able to copy all 16.1m files with RoboCopy, which is using minimal RAM and CPU resources. While this is a great tool, I need your software to be able to keep the files in real-time sync without the need to rescan every time.

Once the initial RoboCopy backup is done, I'll give your program one more shot to see if it can handle the 16.1 million files and half a million folders and keep them in sync.

By the way, you never really explained this whole long delay with the preparing display issue.

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8763

Hi, ViceVersa and Robocopy do work differently because Robocopy does not have a user interface so it does not need to keep all files in memory and create the graphical comparison. That's where the time is spent for the message you are seeing. If you split the job into multiple profiles and run them sequentially the problem is solved and you still have a visual feedback that source and target are really matching. thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

ASPweb · Joined: 30 Jul 2015 Posts: 4

I really don't care about a graphical display.

I only want to scan the file system once, sync the files, then keep them in sync in real time. I really don't mind an occasional rescan, particularly if I can say when and how often, but it is taking 3 hours to scan, plus the time to build the graphical display that I don't need.

Can I not do this without a graphical display? If so, how?

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8763

Hi, the only way is to split the job into multiple profiles and run them sequentially. This also helps that if a full rescan is needed it can be limited to one profile and can be much quicker. thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com