Files are not deleted in target (skipped)

[Ego]

I have setup a profile with backup method. Norescan option is enabled for the target. When running the profile, I get al lot of messages, that files deleted from source are not deleted from target.

Message in the logfile is:

File Deletion Errors:
Deleting file K:\ego\AppData\Local\Mozilla\Firefox\Profiles\2jzwn30s.default\thumbnails\705f2d83ffccff491f9e0feb34919d0e.png [File changed] [SKIPPED].
...

Why are the files skipped? They were definitely not changed in the target folder, after VV copied it there. And even if they were, how would VV know if norescan is enabled? How can I setup VV to delete every file from the target, that is not present in source anymore?

[TGRMN Software]

Hi, this usually happens if a file is found to have changed just before being deleted compared to what was in the no-rescan database. Try the option "Copy files even if timestamp changes after initial comparison" in profile settings -> advanced settings. thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[Ego]

I was using the the option "Copy files even if timestamp changes after initial comparison", as well as creating a shadow copy at the beginning of the compare.


I don't know why viceversa thought, that the files were changes in the target folder. But I did a manual full rescan, and the problem was gone. It's running fine, since the rescan.

Are there any other file attributes (besides CRC, size and timestamp), that are taken into account to decide weather two files are equal?

[TGRMN Software]

Hi, only those attributes are used to compare files, CRC, size and/or timestamp plus file name and file path. thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[Ego]

Thanks for clarification. I don't understand, how any of these attributes got changed in the destination folder.

However, this didn't happen again since the full rescan. So I'm fine with it, and bought my license.

[TGRMN Software]

Thanks, should it happen again please let us know.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[iRalph]

In our company, we use ViceVersa Pro 2.5 build 2514 64-bit version in conjuction with VVEngine 2.1 Server Premium build 2105. I recently upgraded from VV Pro 2.5.2513 and VVEngine 2.1 Server Premium build 2104.

I've noticed since many many months (probably years...) that some replication jobs using the "Backup (Mirror Source to Target)" method do not properly delete folders and files on the target side when those folders and files no longer exist on the source side.

I get "Skipped Files", "Errors (Deleting)" and "Skipped Subfolders" counters warnings in red under the "More Info..." view for a given job in VVengine.

In the job log file, I see plenty of entries like this:
...
2015-03-20 19:22:31 : Deleting file \\?\UNC\cifs-srv1\Netsoft\SoftwareABC\PROD\ABC-desktop-client-4.18.12-r39316\lib\ABC-reporting-swing-4.18.12.jar [File changed] [SKIPPED].
2015-03-20 19:22:31 : Deleting file \\?\UNC\cifs-srv1\Netsoft\SoftwareABC\PROD\ABC-desktop-client-4.18.12-r39316\lib\ABC-risk-swing-4.18.12.jar [File changed] [SKIPPED].
2015-03-20 19:22:31 : Deleting file \\?\UNC\cifs-srv1\Netsoft\SoftwareABC\PROD\ABC-desktop-client-4.18.12-r39316\lib\ABC-security-swing-4.18.12.jar [File changed] [SKIPPED].
2015-03-20 19:22:31 : Deleting file \\?\UNC\cifs-srv1\Netsoft\SoftwareABC\PROD\ABC-desktop-client-4.18.12-r39316\lib\ABC-swing-4.18.12.jar [File changed] [SKIPPED].
...

Those source files are gone since months (including the last 2 folder levels) but the errors still show up.
With the "Backup (Mirror Source to Target)" method, it doesn't matter if files are changed or not at destination side, they should be deleted in ANY case since they no longer exist at source side !

More surprisingly, the target files did not even change ! The only file metadata which could have changed is the archive bit because we backup files at source and destination side. But the file content is 100% the same, as well as the size and the various dates.

But anyway, even if size and/or the modified date change, the file should be deleted when using the Backup method.

For me, there is a bug somewhere when running a VV job with VVEngine.
Not sure it's only a VV bug because when I run the exact same fsf job with VV using the [Compare] button and then [Execute] the results, the extra files and folders do get deleted on the target !

We have plenty of jobs which exhibit the exact same problem.
Note: source and target folders are always UNC fileshares.
In the Profile Settings, the "Copy files even if timestamp changes after initial comparison" is set, and also the "Replace newer target files with older source files (backup method)".
The main issue we face with this problem is that the target folders keep growing compared to the source folders.
Running a manual Compare/Execute in VV solves temporarily the problem, but as soon as some folders and files are deleted on the source side, the problem shows up again !

Help please...

[TGRMN Software]

Hi, this is quite strange to have the error when running from VVEngine but not when running directly in ViceVersa... I will check the source code around the [FILE CHANGED] error and let you know.
thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[TGRMN Software]

OK, I checked the code and the files are skipped if the attributes, size or modified time is changed between the comparison phase and the actual deletion phase.

Regarding these files:

[iRalph]

Hi TGRMN Software,

>> Are these files still present in the target?

Yes, these files (among plenty of others) are still present in the target, this is the main reason why I'm complaining

>> And are they skipped each time the profile runs?

Yes, these files are skipped every time. They are only purged if I run the profile directly in ViceVersa with [Compare] / [Execute].
If I let the VVengine job do the... job (it runs once a day, every day), these files are always skipped.

Maybe this information will help you searching in the code: we use a tracking database which refreshes every 7 days, here the parameters as seen by VVengine:

Comparison
- Comparison Type: Size and Timestamp
Execution
- Method: Backup (Mirror Source to Target)
- No overwrite / read-only / error confirmations
- Tracking Database: "E:\Jobs\Netsoft\Netsoft.tdb"
- Do not rescan target (rescan if condition 'elapsed_days > 7' is true)
- Log File: "E:\Logs\Netsoft\Netsoft.log" (max 10000 KB)
- Log only summary and errors
- No archive for deleted/replaced source files
- No archive for deleted/replaced target files
- Network friendly folder scanning
- Copy files even if timestamp changes after initial comparison
- Try to copy files that are in use by other applications
- Retain folder created and modified timestamps when copying
- Replace newer target files with older source files (backup only)

Of course, I've already tried to reset/delete the tracking database, and also forced a rescan, but the orphan files at target are still not deleted.

Thank you for digging into the code ...

[Jimboa]

Hi iRalph

I think you indicated that the target file deletions correctly occur when you run the profile directly in VV but not using VVEngine.

The different behavior might be related to user account permissions. Have you tried setting the VV engine service to use the same username/password as when you run VV directly? (configure in VV Engine service properties)

I realize the messages you get don't appear to be related to permissions, but changing the login name would be worth a try to see if the error messages stay the same...

[TGRMN Software]

Hi, does this happen only with .jar files? Or every file type that is deleted from source?

What type of system / drive is the target?

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[iRalph]

Hi TGRMN Software,

It happens with different file types. Beside .jar files, the error message also shows up with .dll and .exe file types.
It's exactly the same error message: [File changed] [SKIPPED]

Target systems are EMC VNX2 NAS appliances. Storage is configured as storage pool with multiple RAID5 groups aggregated together in "slices" to maximize the performance. Disks are 900GB 10K SAS, no thin provisioning, but with 100GB SSD cache (mirrored).

CIFS "servers" hosted on the NAS appliances are part of the same AD domain as the VV server.
The VVengine and ViceVersa server runs Windows 2008 R2 SP1.
Negociated protocol is SMB 2.1 (the VNX appliance is configured to support SMB 1, 2 (2.1) and 3 protocols.

We have more than 2200 concurrent users connected to the NAS appliances. We haven't had any access or performance issues so far with these NAS appliances, no complains whatsoever, so I'm pretty sure they do their job as expected with CIFS/SMB fileshares.

@Jimboa,

I ran the ViceVersa process with the same credentials as the VVengine service and guess what, I got plenty of the same error messages as well ! Please note that the target fileshare was not rescanned due to the /norescan option (it rescans the target only once every week to minimize the network traffic).

I've checked the target file/folder security and the account used by VVengine/ViceVersa has Full control over the files (as I expected, but I wanted to be 100% sure), so it's not a security problem.

For me, it looks like an issue with the tracking database. Maybe the tracking database has a different time stamp of the target file, so it refuses to delete it. It should use UTC as the timestamp, but maybe it uses another (local) timestamp...

I think the code of ViceVersa should be changed for correctly handling the "Backup (Mirror Source to Target)" replication method. It shouldn't matter if the target file has a different timestamp or not (stored correctly or not in the tracking database), if the size is not the same as registered in the tracking database or if any of the file attribute has changed, it must be deleted on target side since it no longer exists on the source side.

Hope this helps to troubleshoot further...

[TGRMN Software]

Hi, yes, it looks like a tracking db inconsistency. But it is strange that the profile works fine under ViceVersa and the issue appears only under VVEngine. Somehow the attributes or timestamps of the issue target files are reported differently than what is stored in the tracking db when running from VVEngine and ViceVersa then thinks the files have changed and does not delete them. No you are suggesting that the files should be deleted anyway, but if files change compared to what is in the tracking db, ViceVersa wants to rescan to be sure. I think we will have to create a special version to debug this issue, we would like ViceVersa to log more info on what attributes / timestamps have changed.

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[iRalph]

Hello,

>> Hi, yes, it looks like a tracking db inconsistency.

I've just checked another VV job which doesn't use a tracking database and I get the same kind of errors, here the summary:

Run Preview
Source Target
Total Size Total Size
Files to Add 0 0 2 46.11MB
Files to Update 0 0 8 46.56MB
Files to Delete 0 0 268 609.96MB
Subfolders to Add 0 - 0 -
Subfolders to Delete 0 - 8 -

Run Summary
Source Target
Total Size Total Size
Added Files 0 0 2 46.11MB
Updated Files 0 0 8 46.56MB
Deleted Files 0 0 4 1.89MB
Skipped Files 0 0 264 608.07MB
Errors 0 0 0 0
Added Subfolders 0 - 0 -
Deleted Subfolders 0 - 0 -
Skipped Subfolders 0 - 8 -
Errors 0 - 0 -
Result: OK.

Final Status
Source and Target are different.

Source Target
Total % Size % Total % Size %
Files 82'923 100% 72.27GB 100% 83'187 100% 72.87GB 100%
Excluded Files 0 0% 0 0% 0 0% 0 0%
Matched Files 82'923 100% 72.27GB 100% 82'923 100% 72.27GB 99%
Single Files 0 0% 0 0% 264 0% 608.07MB 1%
Newer Files (Changed) 0 0% 0 0% 0 0% 0 0%
Older Files (Changed) 0 0% 0 0% 0 0% 0 0%
Subfolders 7'075 100% - - 7'083 100% - -
Excluded Subfolders 0 0% - 0% 0 0% - -
Matched Subfolders 7'075 100% - - 7'075 100% - -
Single Subfolders 0 0% - - 8 0% - -

And here the profile settings:

- "\\?\UNC\cifs-srv-src\NetSoft\SoftwareD\" <to> "\\?\UNC\cifs-srv-dst\NetSoft\SoftwareD\" (Include Subfolders)
- Create sources/targets that do not exist
Comparison
- Comparison Type: Size and Timestamp
Execution
- Method: Backup (Mirror Source to Target)
- Log File: "E:\Logs\NetSoft.log" (max 100000 KB)
- No archive for deleted/replaced source files
- No archive for deleted/replaced target files
- Network friendly folder scanning
- Copy files even if timestamp changes after initial comparison
- Try to copy files that are in use by other applications
- No empty source/target check
- Replace newer target files with older source files (backup only)

I did a test with an interactive VV process (started with the same domain account as the VVengine service) and it could delete only 6 out of the 268 extra files still available on target:

Result: OK.

Source Target Total Size Total Size
Added Files 0 0 0 0
Updated Files 0 0 0 0
Deleted Files 0 0 6 8.60MB
Skipped Files 0 0 262 645.94MB
Errors 0 0 0 0
Added Subfolders 0 - 0 -
Deleted Subfolders 0 - 0 -
Skipped Subfolders 0 - 8 -
Errors 0 - 0 -

The account has "change" access over all files and folders, so it should be able to delete the orphan files on target.

Given this last test, I still don't understand why ViceVersa doesn't delete orphan target files when using the "Backup (Mirror Source to Target)" method. Without the tracking database, there is no information left about the previously available source files which were deleted some days/weeks/months ago, so the error message is not correct.

I did a second test using my own admin account as ViceVersa credentials (I have "full control" over all files and folders) and the purge of orphan files works !!! With my admin account I'm Administrator and Backup Operator of the CIFS server. Does ViceVersa handle file deletion differently when the user account has full control/admin access to the files/CIFS server compared to a normal account which has only "modify" access ??

At the end, even though it doesn't seem to be a security issue, the results are different when using an admin account with "full" access compared to a normal service account which has only "modify" access...

If you could provide a ViceVersa build with extra trace logging, I'm pretty sure it would help to pinpoint the problem in the source code.