VVPro not working when many profiles start at the same time

jb3643 · Joined: 16 Jun 2010 Posts: 13

I am using VVPro and VVEngine (current versions, just acquired a few weeks ago). VVengine is setup to run as a system service; therefore any profile jobs are running under the system account as well. I have many profiles that I would like to start at the same time on the weekend. It appears VVEngine starts a process for each profile but VVPro becomes very unreliable when a few profiles get started at the same time. Most of the time I get the windows dialog "VVPro has encountered an error and needs to be shut down"; "Would you like to report this error to Microsoft?". Other times it appears it was completed but the data does not get copied to the target path. Another error I see often is "stopped by user action".
Each profile connects to a UNC path "\\servername\c$\backupfolder" (administrative share) with specified user credentials and then copies the data to the local computer. The next profile connects to a different server and copes the data to the local computer ..... and so on.
The computer running VV is windows XP SP2 ..... and most of the sources are XP as well, with a few windows server 2003 mixed in.

Any suggestions or help? I think this must be a software bug in the current version .....

Thanks

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8759

Hello

do you have a profile log set for each profile? It should have more info on each profile running.

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

jb3643 · Joined: 16 Jun 2010 Posts: 13

I do activate the log file for each profile .... here is what I see:

Some profiles have no log file but they are indicated in VVengine as having been attempted (but indicate an error "Execution canceled by user"). This happens when the profile has never run successfully a single time.

Other profiles have an entry in their log file for the particular time (denoted by a session ID). Of these some look like they loaded the profile, connected to the source, did the compare (but no copy) and then exited. This is not plausible because I create a new target directory (that includes the current date in part of the name) for each backup I make; therefore it must always copy the data (unless I were to start the task 2 times on the same day). It looks that the compare doesnt show any difference for some reason because the log file goes from "comparing done" straight to "disconnected from servername".
Others that have entries worked without problems (showing the copied files and everything).

Just as an FYI - many times the error reported is exit code 4.

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8759

Hello

can you please send the following files to support@tgrmn.com :

- VVEngine.log , VVEngine.xml, VVEngine.db (please search hard disk for these files)
- All the profiles (.fsf) files and the corresponding logs.

thanks!
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

jb3643 · Joined: 16 Jun 2010 Posts: 13

I also made another test/experience with this product that I wanted to notify your development about .....

I tried using the VVScheduler to automate many profiles. Just as FYI, some of my profiles take >10 minutes to finish because I have a lot of files to copy. I use multiple profiles because I want to create log files and folder structures that are independent (backup from one machine is completely seperate from another machine).

At first I tried doing them with only the "/hiddenautoexec" switch and starting say 30 profiles at the same time ..... when doing it this way the product would crash when more than a few (maybe 8-9) profiles are started at the same time.
Next I tried doing 30 profiles using the "/hiddenautoexec /wait" switch. This did greatly improve the reliability but I still have errors when too many tasks are running. I have the impression that once ~25 instances are running, then if additional profiles are started then it gets flaky; it doesnt actually crash but the windows task scheduler task exits with a return code of 0xC0000142 (and of course it doesnt copy my files). Also when this happens I do not get any entry in my log files; so I assume it is exiting before starting to write entries into the log file.

This problem should be very easily reproducable (given this information) and correctable. For now I will try to work around the problem by scheduling my profiles to run with more time between them (to give more time for additional profiles to finish); if this doesnt improve my reliability enough then I will be forced to look for another product if development cannot correct the problem.

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8759

Are you dealing with many (millions) files in each profile? Those could be out-of-memory crashes...
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

jb3643 · Joined: 16 Jun 2010 Posts: 13

I am copying quite a bit of files with my profiles ..... anything from a dozen zip files up to thousands of individual files. I dont think I have anything approaching millions of files.
It really seems to get upset with having many instances of the viceversa process (running profiles) running at the same time. It does tolerate it much better with the "/wait" switch appended; but still goes out to lunch when 30 or so (please treat this as a rough estimate as I havent definitively determined the actual number) are started at the same time.

I havent tried it with Windows Server yet .... in the next weeks I will purchase a server license and test it on that platform as well. However usually XP and server 2003 act very close to the same way.
I just need a reliable, maintenance free way to pull files from many machines on a weekly basis.

What does the return code of 0xC0000142 mean? or is it not intended to mean anything?
Any info/tips/support are appreciated as I still believe this product can be what I need.

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8759

Looking at that code, it is an "unknown" code which is probably an indication of ViceVersa crashing. Do you see the files ViceVersa.dmp and ViceVersa.rpt in the installation folder? What date/time do they have?
thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8759

I have tried to run 21 profiles "together" in VVEngine on a Win 2008 server, but it all seems to work OK. If you send to support :

- VVEngine.log , VVEngine.xml, VVEngine.db (please search hard disk for these files).
- All the profiles (.fsf) files and the corresponding logs.

I can try to reproduce the exact scenario.

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

jb3643 · Joined: 16 Jun 2010 Posts: 13

I updated my software (VV Pro 2.5) to the recently released version (build 2502); yet still have a problem with this software.

Here's my current setup:
XP SP2 x64
VV Pro 2.5 Build 2502
VVScheduler (for scheduling)

Here's my application setup:
63 Profiles (copy files from remote source to local destination)
63 task scheduler entries for above profiles (created by vbs script I wrote)
All 63 task scheduler entries are setup to run under the SYSTEM account (so they will run if no one is logged in)

In the script I wrote to create the VVScheduler entries (via task scheduler), I have the ability to use the “/hiddenautoexec” and/or the “/wait” command line switches. For purposes of this discussion, let’s assume my script schedules all profiles to start at exactly the same time AND the /hiddenautoexec switch is used. Note the way my script creates these scheduled entries is functionally the same as those created manually in the VV Scheduler GUI (running with schedstub.exe).

IF I use the /wait switch, the profiles run sequentially without a problem (and I have a lot of history to prove this is reliable).
IF I omit the /wait switch, the profiles attempt to run simultaneously; here’s where my problem begins.

When the 63 profiles attempt to start simultaneously, I get immediate “Visual C++ Runtime Library Errors”; a few profiles exit this way. Most profiles do not run successfully, and have exit code 4. A smaller percentage error out with error code 3. A few profiles hang infinitely. A few profiles run through and complete properly.
In trying to help your development understand what might be going on I did a little digging. I noticed in the windows event log I have Application Errors “Faulting module viceversa.exe version 2.5.0.2 fault address 0x0000000000346e4e”. I also noticed an error that occurred just before the first “Visual C++ Runtime Error”; that error was “Source:Win32k EventID:243 Description: A desktop heap allocation failed”.
The latter error seems pretty interesting …. Maybe it is pointing to part of the problem?

I haven’t tested yet on Server 2008 …. It may not present this behavior. Nonetheless I want some feedback about my current situation as it may help another customer deal with this pesky issue.

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8759

Hi,

when ViceVersa crashes, it creates 2 files, ViceVersa.dmp and ViceVersa.rpt Can you send those files to support@tgrmn.com? Also it would be useful to get the profile logs for all the profiles running simultaneously.

thank you
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

jb3643 · Joined: 16 Jun 2010 Posts: 13

I have sent these dump files along with my profile log files. Please confirm receipt of them.

One additional thing I felt important to add .... This problem seems to be related to the fact that too many instances of VVPro are running at one time. In other words there are multiple ways to reproduce it:
1. Start many profiles at the same time
2. Start a few profiles, start a few more, start a few more (whereas the stackup of running profiles exceed some "unknown limitation").

As a workaround I have scheduled my profiles to start with staggered times so a few can start AND complete, then a few more start AND complete, etc.
But I think fundamentally you want your product to perform with unlimited parallelism.....

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8759

Hi, investigating a similar issue with another customer we found that the issue was only happening when running in a virtual machine. Can you confirm you are running on a physical server and not in a virtual machine?
thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

jb3643 · Joined: 16 Jun 2010 Posts: 13

In my current situation, everything is running on physical servers.

It could very well be something specific to my operating system configuration but so far I havent been able to find it. Based on this fault (in the windows event log about the heap) I get, this points in the direction of windows memory settings. In my case these settings are per Microsoft's recommendation; which is a lot of memory due to the fact I am running on a 64 bit platform. Microsoft even warns customers to NOT change these on a 64 bit platform.

jb3643 · Joined: 16 Jun 2010 Posts: 13

Its been about a month now .... just checking back in for an update on the status of this issue?