Are transfers atomic?

I was answering this reddit question and it got me to thinking, are rclone transfers atomic at the file level? Does it depend on the remotes? File sizes (multi-part uploads?).

Meaning that if I am uploading to a cloud remote and say another person uploads the same file, what happens? Atomic I always thought referred more to a file system and how file locking worked if multiple things were trying to write to the same file handle.

I made two test files and started 1 upload and a few seconds later, started a second to the same file name:

felix@gemini:~$ fallocate -l 1G test.img
felix@gemini:~$ rclone copy test.img GD: -P
Transferred:            1G / 1 GBytes, 100%, 34.531 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        29.8s
felix@gemini:~$ rclone lsl GD:test.img
1073741824 2021-01-15 16:48:22.751000000 test.img

GD is a bit unique as I'll get double files since I can have duplicates..

felix@gemini:/data$ rclone copyto test2.img GD:test.img -P
Transferred:            2G / 2 GBytes, 100%, 38.931 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        57.9s
felix@gemini:/data$ rclone lsl GD:test.img
2147483648 2021-01-15 16:48:50.167000000 test.img
1073741824 2021-01-15 16:48:22.751000000 test.img

There is no file locking on the remote.

Atomic meaning the file is either there fully or it is not. For example, if you are transferring a file (let's say a large one but the idea remains regardless) and your internet cuts out, will it show up at all?

Also, to say I am jealous of your upload speed is an understatement!

1 Like

There is a feature out there for resume uploads but it is not implemented yet so if it fails, you have to redo the whole file currently.

Great question...

For just about all the providers the answer is yes. The transfer is atomic and you will never see a partially uploaded file.

The exceptions are the local disk. FTP and SFTP.

At some point I would like rclone to upload to a temporary file name then rename it over the existing file like rsync does when used without the flag --inplace.

Thanks. Makes sense. Given the fact that rclone doesn’t do rsync style block transfers, it makes sense that you could do a transfer and move.

1 Like