Is Rclone’s large file upload to S3 atomic?

What is the problem you are having with rclone?

My Rclone setup works well, this is more of a question out of curiosity. :smiley:

We use a cron job with Rclone to upload a large file to S3. There is another downstream cron job that expects the file to be fully uploaded before it begins processing. Occasionally, the upload doesn’t finish in time, causing the downstream job to fail (though this may not be the root cause, I am still investigating).

I am particularly curious about how Rclone handles large file uploads to S3. I understand that S3 operations are atomic, but does Rclone upload the file in large chunks? If so, does that mean the process isn’t atomic during the upload phase? Thanks!

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.3
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-79-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --contimeout=10s --progress --ignore-case "my-machine:/C:/my_data/" "my-s3:/my-bucket/my_data/" --filter="- **/*" --filter="+ *.tar.zst" --filter="- *"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[my-s3]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-west-2
storage_class = STANDARD

[my-machine]
type = sftp
shell_type = powershell
host = XXX
port = 22
user = XXX
pass = XXX
md5sum_command = &{param($Path);Get-FileHash -Algorithm MD5 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{"$($_.ToLower())  ${Path}"}}
sha1sum_command = &{param($Path);Get-FileHash -Algorithm SHA1 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{"$($_.ToLower())  ${Path}"}}

A log from the command that you were trying to run with the -vv flag

sudo rclone copy --contimeout=10s --progress --ignore-case "my-machine:/C:/my_data/" "my-s3:/my-bucket/my_data/" --filter="- **/*" --filter="+ *.tar.zst" --filter="- *" -vv
2025/06/09 12:39:29 DEBUG : rclone: Version "v1.69.3" starting with parameters ["rclone" "copy" "--contimeout=10s" "--progress" "--ignore-case" "my-machine:/C:/my_data/" "my-s3:/my-bucket/my_data/" "--filter=- **/*" "--filter=+ *.tar.zst" "--filter=- *" "-vv"]
2025/06/09 12:39:29 DEBUG : Creating backend with remote "my-machine:/C:/my_data/"
2025/06/09 12:39:29 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2025/06/09 12:39:29 DEBUG : sftp://XXX@172.20.26.210:22//C:/my_data/: New connection 10.11.163.90:52142->172.20.26.210:22 to "SSH-2.0-OpenSSH_for_Windows_8.6"
2025/06/09 12:39:29 DEBUG : sftp://XXX@172.20.26.210:22//C:/my_data/: Shell type "powershell" from config
2025/06/09 12:39:29 DEBUG : sftp://XXX@172.20.26.210:22//C:/my_data/: Using root directory "/C:/my_data/"
2025/06/09 12:39:29 DEBUG : Creating backend with remote "my-s3:/my-bucket/my_data/"
2025/06/09 12:39:29 DEBUG : fs cache: renaming cache item "my-s3:/my-bucket/my_data/" to be canonical "my-s3:my-bucket/my_data"
2025/06/09 12:39:29 DEBUG : 0036_A: Excluded
2025/06/09 12:39:29 DEBUG : S3 bucket my-bucket path my_data: Waiting for checks to finish
2025/06/09 12:39:29 DEBUG : 0034_A.tar.zst: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/06/09 12:39:29 DEBUG : 0034_A.tar.zst: Unchanged skipping
2025/06/09 12:39:29 DEBUG : 0036_B.tar.zst: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/06/09 12:39:29 DEBUG : 0036_B.tar.zst: Unchanged skipping
2025/06/09 12:39:29 DEBUG : 0035_B.tar.zst: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/06/09 12:39:29 DEBUG : 0035_B.tar.zst: Unchanged skipping
2025/06/09 12:39:29 DEBUG : 0035_A.tar.zst: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/06/09 12:39:29 DEBUG : 0035_A.tar.zst: Unchanged skipping
2025/06/09 12:39:29 DEBUG : 0037_B.tar.zst: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/06/09 12:39:29 DEBUG : 0037_B.tar.zst: Unchanged skipping
2025/06/09 12:39:29 DEBUG : 0036_A.tar.zst: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/06/09 12:39:29 DEBUG : 0036_A.tar.zst: Unchanged skipping
2025/06/09 12:39:29 DEBUG : 0034_B.tar.zst: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/06/09 12:39:29 DEBUG : 0034_B.tar.zst: Unchanged skipping
2025/06/09 12:39:29 DEBUG : S3 bucket my-bucket path my_data: Waiting for transfers to finish
2025/06/09 12:39:29 INFO  : There was nothing to transfer
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 8 / 8, 100%
Elapsed time:         0.2s
2025/06/09 12:39:29 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 8 / 8, 100%
Elapsed time:         0.2s

2025/06/09 12:39:29 DEBUG : 29 go routines active
2025/06/09 12:39:29 DEBUG : sftp://XXX@172.20.26.210:22//C:/my_data/: Closing 1 unused connections

sorry, not sure what you are asking?

the end result is the same, chunked or not chunked.
after a file is fully uploaded, only then, it will be available for download.

Is your machine saturating your upload bandwidth? If not, there are some flags that can increase the amount of streams/chunks uploaded at a time. You can also increase the size of chunks, especially if it is a big file. It makes a big difference uploading a 50gb file with 50mb chunks instead of 5mb. More data at once, and less api calls, leading to faster uploads so the you dont get delimitted by the api.

1 Like

Yes rclone does use chunks, but the new file (or any chunks) aren't visible until it is committed making multipart uploads effectively atomic

1 Like