Differential Backup

[albedoConsulting]

Is there a way to use VV Pro for a daily differential backup? I want to push Daily Difs to a network hard drive, then push a Full Friday to a different drive and use NT Backup to put just the Full Friday on tape for offsite storage. At nearly 200G of data, a full backup every day is just not an option, but I do need a way to recover any given file from last week. Anything older and I am OK with just the Full Friday.

Crossed fingers that I am just missing something obvious.

Thanks,
Gordon

[CoreyPlover]

I used to also think the same way that you seem to think about backups - that you do an incremental of differential backup often and a full backup less often. However ViceVersa can completely change this and effectively do a full backup as quick as a differential!

My method for backing up is to use ViceVersa with 'Archiving' turned on. Let's say you have D:\ that you wish backed up to X:\. When you choose the "Replication mirror" from Source "D:\" to Target "X:\Current" ViceVersa will first scan all 200 Gb on D:\, then the 200 Gb under "X:\Archives" (assuming that it has been sync'ed once already). In my company, we have about 60,000 files and ViceVersa handles the comparison in about 5-10 minutes (I reckon number of files, not size of files matter for this comparison stage).

It will then do a comparison and sorting of file (very quick, < 1 minute) and give a list of those files that differ between source and target. When you complete the synchronisation with Archiving enabled all modified files that are about to be overwritten by the sync or those files that are about to be deleted will be copied to an Archive location (i.e. X:\Archive), and then the new files are copied into place (i.e. X:\Current). Sync speed will depend on hardware but 15 Mb per second to an external USB 2.0 hard drive is achievable.

The result of this synchronisation is that it took the same time as a differential, but the result is that X:\Current contains the FULL mirrored copy of D:\. With Archiving enabled, your differential backups are actually stored in reverse. By this I mean that under a traditional backup, let's say by Wednesday you have a Full Friday backup + Monday's incremental + Tuesday's incremental. Under ViceVersa you actually have X:\Current containing Tuesday night's full mirror + all files changed between Monday & Tuesday, and Friday & Monday are stored in X:\Archives (Hint, turn on 'Append path to Archive folder" to ensure that yuor archived files retain their path).

I like to think of this as a "reverse incremental" backup. You can actually repeat a full synchronisation with archiving every day and you can recover all files from any day going back an unlimited time (we have daily backups going back over a year). You are limited only by the X:\Archive directory and the capacity of your target media.

FINAL NOTE: In your case, you can actually achieve daily backups that last indefinitely by using 2 x 200 Gb external USB drives. Sync your data to external drive 1 and tell archiving to move old files from external drive 1 to external drive 2. That way you have 200 Gb of target capacity + 200 Gb for historical archives. Also you can change the archive drive for a new one when it gets full for unlimited backups (as long as your data you are backing up remains less than the capacity of you external media)

[Guest]

Corey,
First off, it is not a speed, or even a capacity issue, that drives my daily backup desires. We simply have a genuine need to go back to "last tuesday's version of file X", and this with 40 changes and saves in between, so volume shadow copy doesn't necessarily have the version they need. If I loose that ability, I am in real trouble with my users.
It sounds like I need to look at options for once a day shadows, and the archive approach with VV.

Thanks,
Gordon

[TGRMN Software]

VV does not support the daily differential backup you describe, but
instead it supports a different method, like Coley describes, that is you always get a mirror of your most recent data plus archived version of old files.

It works like this: suppose you setup an A->B mirroring. The first time you run VV it will copy all files and folders from A to B. A and B are now identical. Then files are changed in A, some files are added and other files are deleted. The second time you run VV, it will copy only changed files, delete the files that were deleted in A from B and add files from A to B. Files that have not changed are left alone. So you have again an exact copy of A in B.
If you enable archiving in ViceVersa, instead of simply deleting or overwriting files in B during execution (that is what a standard mirror does) Viceversa can archive (=move) files to a third location. VV will mark the files with the timestamp of when they are moved. You can customize how many copies to keep. For instance after a certain amount of days you can remove very old copies. So you can go back to any old copy. There is an illustration here http://www.tgrmn.com/web/file_backup.htm that kind of shows it visually. Of course the files in the archive can be kept compressed to save space.
VV provides a tool called 'archive viewer' that you can use to browse your archive.
We are also thinking to add a point-in-time restore to one of the next releases which could restore automatically at a certain point in time. And if you have any other suggestion let us know!

Thank you for opening this thread, backup strategy discussions are always good!
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[CoreyPlover]

I think you should definitely be thinking of Volume Shadowing in conjunction with whichever backup strategy you choose to adopt. Volume Shadow would be less resource intensive (? unsure of this), MUCH more transparent (Right click, user restores file themselves) and frequent (if you are wanting to backup regular changes, both Volume Shadow and ViceVersa will miss some, but Volume Shadow has the capabilities to increase snapshot frequency)

I am considering moving to hourly Volume Shadows, or probably more realistically, 4 shadows per day. In effect Volume Shadow becomes a sort of automatic version control which sounds like what you require out of a daily schedule. I am also hoping that with minor tweaking Microsoft will refine the Volume Shadow service to be a true version control service and overlay snapshots of files based low level modifcations rather than folders based on predefined times.


For the weekly / offsite backups I would highly recommend ViceVersa over traditional tape backup for the following reasons:

* Less expensive - Consider the total cost of at least one backup tape unit plus the licensing cost of Veritas or similar software vs external USB drive + ViceVersa

* Speed - I expect that USB 2.0 transfer rates will surpass tape write speeds. I am currently getting 15Mb / sec transfer rates.

* Transparency - To me this is the big item. File synchronisation is visible to the user both when backups are occurring as well as in the restore process. Tapes are streaming media that require third party utilities to restore a specific file. USB sync'ed drives are visible in ALL computers through ALL file management software (Windows Explorer)

Just a few disadvantages to even things up a bit:
* Portability - USB external drives are not as portable as tape drives, but the new generation of USB powered 2.5" IDE caddies may change this in the near future

* Cost per Gb storage - Ignoring compression, Tape drives are slightly cheaper than USB drives per Gb. However you have to recover the cost of a $2,000 tape unit or two plus tape backup software (if applicable), so USB drives are probably better in total cost

* Reliability - Hard drive manaufacturers have been reducing their warranties for a number of years and are currently between 1 and 3 year warranties. Expected life could reach 5+ years but tape drives generally have slightly longer lifetimes. With regular rotation, reliability of media should never be a major issue.

[Guest]

My main worry with HDs for offsite backup is three fold.

1: Anything but a laptop HD is not really designed to move. The process of removing a drive and taking it to the Safe Deposit box can actually kill the drive. We have lost one this way already.
2: No hard drive is designed to sit dead for long periods of time, so end of year backup can't really be done to HD, and at 200G of data, DVD isn't an option either.
3: At a few hundred bucks for a swappable caddy that isn't junk, the prices start to converge anyway.

So we are looking at VV to mirror all data to permanent HDs on a backup machine. NT Backup to push BKFs of server System States to that machine, and Veritas (which we already own) to push daily Exchange to a BKF on the same machine. This last because Veritas/NT Backup will do an online backup and log file cleanup with Exchange. To use VV to mirror the databases would require a script to offline Exchange, which isn't what we want.
Anyway, that gives us good backup to HD in house, with weekly then going to tape and off site, but the tape is on that backup machine, so slow tape processes can happen during business hours with no impact on network bandwidth or server performance.

All in all a good system. I do wish VV could do an online Exchange backup using VSC like Veritas can. Maybe they should look at an add-on for VV to do just that.

Best,
Gordon

[CoreyPlover]

Sounds like the system you have is quite robust (volume shadow, daily sync, weekly offsite, mix of tape and hard drive, etc)

I am used to smaller business solutions. I have implemented them in the business that I have worked for, not large client solutions.

I would say that USB caddies (not swappable trays), even the 3.5" IDE ones seem fairly realiable for transport and storage. At work, we cart them in and out in a briefcase each week, no problems over last 18 months, also have a personal USB caddy at home which spends 99% of time unplugged and sitting on shelf. Again, no problems in last 2 years. However, I can understand your concerns.

Just another side solution that I have found works well for file syncronisation backups is to script ExMerge to perform a .PST export of Exchange mailboxes every night and synchronise these PST files. Using ExMerge is naturally incremental, so is actually VERY quick after the first run and I suspect it can even grab the users Calendar and Task items. Also you can turn the Archive filter for .PST files off in VV because ExMerge never deletes emails, it just exports the new messages into the PST file. But I stress, this is a small business solution and primarily useful to avoid large licensing costs of Veritas and the like and bring VV into a more complete backup strategy.

[TGRMN Software]

BTW, VV can use VSS / VSC to copy open files.
http://www.tgrmn.com/web/kb/item33.htm
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

[robertpri]

I'm reading this thread and trying to understand VSS/VSC but the users appear to be saving across a network of some kind. But the link to the No 33 explanation says:

"The file being copied must be on a local disk, not on a network disk. "

So, how is this being done across the net?

[TGRMN Software]

Hello - the volume for which a shadow copy is requested needs to be local, not the volume where the files will be copied to.

A (VSS) --> B

A needs to be a local volume, but B can be over the network.
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com