Reduce Cache usage for SQL Server backups

What is the problem you are having with rclone?

We are using rclone to backup our SQL Server databases to an AWS S3 bucket. We are able to backup the databases without any issues. What we found was that when we use the cache option with rclone, rclone copies the backups to the cache first and then moves it to the S3 bucket. That means that our cache on the local disk has to be the size of all the databases backups combined which runs into GB's for a single days worth of backups. We have a database backup that is nearly 400GB in size and that would force us to have a cache on the local storage for that equal amount of space. The reason we are migrating to AWS is to reduce storage costs. But the Cache storage will still require us to provide that to rclone. Is there a way to perhaps reduce the no of files that are cached to reduce the cache space? What are any best practises that you can recommend in this scenario. Fyi. all the backups are sequential.

Thank you very much for your time and help.

What is your rclone version (output from rclone version)

< V 1.52.3
-->

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Microsoft Windows 2013, 64 bit

Which cloud storage system are you using? (eg Google Drive)

AWS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here
``` start E:\rclone\rclone mount test:crico-sql-prod z: --vfs-cache-mode writes --allow-other --config=c:\temp\rclone\rclone.conf --log-file e:\rclone.log --log-level DEBUG


#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

Paste config here

type = s3
provider = AWS
env_auth = false
access_key_id = ####################
secret_access_key = ##########################################
region = us-east-1
acl = private
storage_class = STANDARD



#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

Paste log here

You should update rclone as that's an old version.

You can remove --vfs-cache-mode writes if you don't need it.

For copying large files, it's better/faster/more reliable to rclone copy rather than a mount albeit you do have to keep a local copy first.

If you do use the mount, you need enough storage for the full file anyway if you use writes or any cache mode so I'd punt on the mount and just use rclone copy.

hi,
this post might be helpful for ideas
Sql Server backup to S3 (vfs-cache-mode) - #6 by asdffdsa

We tried removing the cache and it failed. What we are thinking is using a network drive as a cache rather than a local drive. We then delete all the files in the cache once the backup is done. Please let me know if anyone had success using a network drive as a cache.

As always thank you very much for your help.

in that same space used by the cache, you should be able to run the sql backup.
then rclone move it to the cloud.

is that a SMB share?
have you tried ISCSI?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.