sfsdfd
Joined: 02 Sep 2011
Posts: 3
|
 Posted: Fri Sep 02, 2011 1:14 pm Post subject: Overaggressive folder rescan |
 |
|
Next topic: Folder rescanning while setting up a sync.
Of course, it's critically important that ViceVersa accurately reflect the results of a sync. Therefore, whenever the properties of the sync profile change, ViceVersa initiates a rescan of both folders with the new profile properties.
However, sometimes many incremental changes to the profile are necessary - particularly for filters. For example, in order to create an accurate set of filters for a particular sync operation, it's sometimes necessary to adjust the filters repeatedly (i.e., create some filters, check out the results, notice that some files are accidentally being included, adjust the filters, check out the results, notice that the filter isn't correct and has had no effect, adjust the filter again, check out the results, notice that the filter is now overinclusive... etc.) This can take several tries to get right. And this situation is much more complex with regular expressions, which were recently added to VV, and which can be quite difficult to get right.
The problem here is that rescans take a while. For example, I'm using VV to sync an image of a 3-terabyte archive with about a half-million files over a network... rescanning the archive can take 20 minutes or more. Worse, scenarios like this (lots of files, deep folder structure, and a complex sync methodology) are exactly the sorts of situations where filters (particularly specified with regexes) are likely to be applied!
It seems to me that the rescan trigger could be considerably reduced. Instead of performing a live scan of each folder and applying the current sync logic to it EVERY TIME, how about performing the live scan, keeping the results, and applying (and reapplying) the profile to the scan? Filter adjustments in this solution probably require a few seconds, rather than the better part of an hour.
There is the danger here that if the contents of either folder are modified after the first scan, the profile won't accurately reflect the results in view of the modifications. However, I don't see this as a problem for three reasons:
(a) This can *always* happen, unless you actually lock the file system between the comparison scan and the sync execution, which, of course, isn't done because it's majorly problematic. Conversely, if you perform a scan and then wait an hour before applying it, you're likely to run into the same inconsistency - and yet, VV doesn't spontaneously trigger a rescan to maintain fresh results, right? So the user already has to expect some degree of inconsistency based on the non-atomic nature of the sync operation.
(b) VV already handles inconsistency very well: it will only take actions that were indicated in the comparison, and reports inconsistencies (changed file stamps, attempts to delete files that are now missing, copying over files that have materialized in the target structure in the interim, etc.) as errors for user attention. This capability should fully alleviate concerns about inconsistent results due to slightly stale comparison information caused by not performing a rescan upon every profile change.
(c) The Comparison dialog has a Refresh button. 
As an additional nuance: I imagine that the folder scan is performed in a manner that skips the scanning of folders that are excluded by a filter, and that changing the filter to remove this exclusion therefore needs a rescan. However, it really just needs a rescan *of the previously excluded folders* (and subfolders), not of the entire source or target file/folder set. Thus, the folder-scan caching logic could easily keep a list of the folders that were filtered out during the last scan, and, if a change to the filters alters the logic to include those folders, scan the newly included folders and add them to the cached results.
Thus - I'd like to suggest the following changes:
(1) A caching of the results of a folder scan, and an application of changed profile settings to the cached results.
(2) A modest restraint of the triggering logic for rescanning the source and target folders during a comparison. Of course, there are some identifiable actions that NEED to trigger a full rescan - e.g., changing a source or target folder (...but even then, maybe only rescan the folder having the changed target?) But changes to the filters, in the absence of identifiable changes to the folder contents, probably don't need to trigger a rescan every time, and can be applied to the results of the last scan.
(3) (Maybe?) Backgrounding the rescan process. When profile settings change, VV could apply the profile settings to the last scan and show the results... while also performing a rescan, and then updating the profile to reflect any changes in the folder structure. (Could simply label the changed list "Rescanning" while the rescan is occurring, just to let the user know that changes may occur.) |
|
ViceVersa HOME 
Forum FAQ
Search Forum
Register
Profile
Log in
