rClone cache accumulation control? Or change cache directory

[naruto]

What is the problem you are having with rclone?

Hi y'all,

We have encountered while using rclone for data archiving to a cold S3 storage. As a newcomer to rclone, I am facing challenges with the accumulation of cache files on my system's C:\ drive, which is causing it to fill up rapidly.

Here's a brief overview of the situation:

We are utilizing a Windows 2016 Server for our data archiving tasks.
We have approximately 16TB of data spread across four disks that need to be copied to the S3 storage.
During the data transfer process, the available space on the C:\ drive is quickly depleted. This led to a critical situation where only 10GB of space remained before the cache files were automatically deleted, temporarily alleviating the problem.
We attempted to address the issue by adjusting the rclone configuration with the --vfs-cache-max-size 50G parameter. However, this did not resolve the underlying problem.
Any help on how to change the cache path to a different directory to prevent further depletion of the C:\ drive.
I had set up the code to DEBUG and unfortunately the log file went over 6GB and could not be opened. Also, I hope logs will not be necessary for changing the cache path?

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

• os/version: Microsoft Windows Server 2016 Standard 1607 (64 bit)
• os/kernel: 10.0.14393.6800 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.22.1
• go/linking: static
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

S3 storage(HCP)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

mount archieve:archieve/ S: --vfs-cache-mode full --no-check-certificate --config "C:\HCP-Drive-Mount\hcp-ac.conf" --log-file "C:\HCP-Drive-Mount\logs\mount.log" --log-level INFO

mount archieve:archieve/ S: --vfs-cache-mode full --vfs-cache-max-size 50G --no-check-certificate --config "C:\HCP-Drive-Mount\hcp-ac.conf" --log-file "C:\HCP-Drive-Mount\logs\mount.log" --log-level DEBUG
 

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[backup]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = xxx
acl = public-read
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

I used DEBUG to fix the issue and now the log file is over 6GB and cannot open the file.

[kapitainsky]

Cache limit is “soft”. If set for example to 50G but you copy 100G to such mount then all data will be cached temporarily.

I would suggest not to use mount for initial transfer. Use rclone copy/move/sync.

[naruto]

@kapitainsky,

What we are doing here is, mounted the S3 as a disk and the have rsync to copy the data from other drives to S3 drive.
Will we be able to copy the data to s3 without mounting s3 as a drive?
Additionally, if cache limit is soft, is there any way in changing the cache path to somewhere else? if so, then we can provide another NAS allocated to the cache directory alone?
We did see a command --cache-dir , but is this legit? Can we use them?

[kapitainsky]

Yes you can specify cache location using exactly this flag. But if you have a lot of data do not use mount. It is not designed for it. And will be always much slower than e.g. rclone copy.

Simply run:

rclone copy /path/to/local/data archieve:archieve

You can always stop it and then run again. Already copied data won’t be copied again.

[naruto]

That archieve:archieve is the S3 that I have mounted. That command is what is on my regedit.
So, if I am copying, willl I be able to go with the flags such as:

rclone copy <source/path> archive:archive/ S: --no-check-certificate --config "C:\HCP-Drive-Mount\hcp-ac.conf" --log-file "C:\HCP-Drive-Mount\logs\mount.log" --log-level INFO

Will this work? Is there any other flags?

[kapitainsky]

You do not need S:

Start with:

rclone copy <source/path> archive:archive --no-check-certificate --config "C:\HCP-Drive-Mount\hcp-ac.conf" --log-file "C:\HCP-Drive-Mount\logs\mount.log" --log-level INFO

let's see how fast it is. If much slower than your uplink Internet connection we can try to optimise it.

You can try to run it with --dry-run flag first - to see if all OK. You can also use --log-level DEBUG to see what is going on in details. Then if all looks fine you can start your copy job:)

Add --progress flag to see in real time how it performs.

[naruto]

gotcha @kapitainsky
thanks a lott!!!
Lemme check and will get back