ViceVersa 3 copy speed to AWS vs S3Express

Alice · Joined: 29 Jan 2015 Posts: 282

We are new customers to AWS and the first task I have is to copy quite a few million files there from a local IBM Storwize V7000 Unified device.

The initial data set to be 'archived' (ie copied to AWS then deleted at source) is 22,000,000 files (11TB), this is a mix of 'user generated' files so .docs / .pdfs / .msgs all quite small in size.

I've set up lots of VV Profiles to copy sections of the data at approx. 250,000 files per Profile to AWS, but as you would expect its taking ages. So measure this process in weeks not days.

For some other unrelated Enterprise Vault data I was able to use .zip (Store option) so no compression for speed at Source, then I VV'ed the very large files across to AWS, then unzipped them. But I can't do that with the 22,000,000 as I don't really have the disk space needed locally to .zip them or the time !

Would your S3Express product 'push' the data more quickly from Source to AWS than VV 3.0 ? If so by a factor of…..roughly ?

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8769

Hi, do you have a SMB connection to AWS which you are using ViceVersa PRO on? Is it to S3 strorage or a server on AWS? thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

Alice · Joined: 29 Jan 2015 Posts: 282

We have an S3 Storage Gateway VM Appliance with 1TB cache and two Buckets. I am using VV to copy data to one of its Shares... For medium/large files it’s ok speed, for smaller files it’s dire in performance terms. Only 3,000,000 files copied in an entire week !

I know we can use a local Snowball, but that's not available at the mo.

S3express could improve the transfer speeds compared to VV ?

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8769

S3Express can upload very very fast (max speed achievable on S3 with multiple threads and part/restartable uploads) ... but it uses only the S3 protocol over https, no shares or SMB.

Instead ViceVersa only works over network shares and SMB.
If you have not done this already, try increasing the copy buffer to max size in the ViceVersa profile settings -> performance. It should help a bit.

thanks
_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com

Alice · Joined: 29 Jan 2015 Posts: 282

Can S3Express be used to push data to a local Snowball device ?

TGRMN Software · Site Admin Joined: 10 Jan 2005 Posts: 8769

If it supports the S3 protocol, then yes... never tried sorry Sad

_________________
--
TGRMN Software Support
http://www.tgrmn.com
http://www.compareandmerge.com