Sequential server-side copy for S3 mount

What is the problem you are having with rclone?

I'm using S3 as a mount for rclone on my Windows system. Specifically, I am using it for Steam games. However, when a new game is installed, it takes a very long time, as it does a sequential server-side copy one file at a time from one folder in the S3 bucket to another. You can see this in the log where only a single file is being copied from 19:49:29 to 19:50:44, and all other processes stop.

Is it possible to continue doing other server-side copies in parallel (or to disable server-side copy entirely)? If so what do I need to do to change my command?

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: Microsoft Windows Server 2022 Datacenter 21H2 (64 bit)
- os/kernel: 10.0.20348.587 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount "gameaway-steam-games:gameaway-steam-games/!email!" "C:\Program Files (x86)\Steam\steamapps" --allow-other --vfs-cache-mode full --vfs-cache-max-age 0s --fuse-flag --FileSecurity="D:P(A;;FA;;;WD)" --transfers 1000 --fast-list --buffer-size 1G --checkers 32 --vfs-read-chunk-size 16M --vfs-read-chunk-size-limit off --bwlimit 100G --vfs-read-ahead 128M --no-checksum --no-modtime -v --log-file="C:\Startup Scripts\rclone.log"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gameaway-steam-games]
type = s3
provider = AWS
env_auth = true
region = ap-south-1
location_constraint = ap-south-1
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2023/09/17 19:49:28 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsGeometryLODs1.sga: Copied (server-side copy) to: common/Age of Empires IV/cardinal/archives/Gen7AssetsGeometryLODs1.sga
2023/09/17 19:49:28 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsGeometryLODs1.sga: Deleted
2023/09/17 19:49:29 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsGeometryLODs2.sga: Copied (server-side copy) to: common/Age of Empires IV/cardinal/archives/Gen7AssetsGeometryLODs2.sga
2023/09/17 19:49:29 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsGeometryLODs2.sga: Deleted
2023/09/17 19:49:29 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsGeometryLODs3.sga: Copied (server-side copy) to: common/Age of Empires IV/cardinal/archives/Gen7AssetsGeometryLODs3.sga
2023/09/17 19:49:29 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsGeometryLODs3.sga: Deleted
2023/09/17 19:50:44 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsMaterials.sga: Copied (server-side copy) to: common/Age of Empires IV/cardinal/archives/Gen7AssetsMaterials.sga
2023/09/17 19:50:44 INFO  : downloading/1466860/cardinal/archives/Gen7AssetsMaterials.sga: Deleted
2023/09/17 19:50:45 INFO  : downloading/1466860/cardinal/archives/Gen7ShadersAndEDFs.sga: Copied (server-side copy) to: common/Age of Empires IV/cardinal/archives/Gen7ShadersAndEDFs.sga
2023/09/17 19:50:45 INFO  : downloading/1466860/cardinal/archives/Gen7ShadersAndEDFs.sga: Deleted

--checkers - you can remove it. it does nothing for mount
--vfs-cache-max-age 0s - what is the purpose of the cache which is never valid? Remove it or set to some sensible value.
--max-read-ahead - does not work on Windows. you can remove it
--buffer-size - will be probably faster without it
--bwlimit 100G - 100 GiB per second limit? What do you need it for?
--transfers 1000 - this is definitely excessive - change it to maybe 32

add

--vfs-write-back duration   Time to writeback files after last use when using cache (default 5s)

set this to the value which is longer than time it takes to install a new game - 10m maybe.

Ended up making a PR for rclone to do what I want - vfs: add exclusions for vfs write back - fixes #2109 by utkarshdalal · Pull Request #7319 · rclone/rclone · GitHub

2 Likes

This is the best way:)!

welcome to the forum,

maybe try --disable=copy