Are transfers atomic?

Feature
1

I was answering this reddit question and it got me to thinking, are rclone transfers atomic at the file level? Does it depend on the remotes? File sizes (multi-part uploads?).

1 Like
2

Meaning that if I am uploading to a cloud remote and say another person uploads the same file, what happens? Atomic I always thought referred more to a file system and how file locking worked if multiple things were trying to write to the same file handle.

I made two test files and started 1 upload and a few seconds later, started a second to the same file name:

felix@gemini:~$ fallocate -l 1G test.img
felix@gemini:~$ rclone copy test.img GD: -P
Transferred:            1G / 1 GBytes, 100%, 34.531 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        29.8s
felix@gemini:~$ rclone lsl GD:test.img
1073741824 2021-01-15 16:48:22.751000000 test.img

GD is a bit unique as I'll get double files since I can have duplicates..

felix@gemini:/data$ rclone copyto test2.img GD:test.img -P
Transferred:            2G / 2 GBytes, 100%, 38.931 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        57.9s
felix@gemini:/data$ rclone lsl GD:test.img
2147483648 2021-01-15 16:48:50.167000000 test.img
1073741824 2021-01-15 16:48:22.751000000 test.img

There is no file locking on the remote.

3

Atomic meaning the file is either there fully or it is not. For example, if you are transferring a file (let's say a large one but the idea remains regardless) and your internet cuts out, will it show up at all?

4

Also, to say I am jealous of your upload speed is an understatement!

1 Like
5

There is a feature out there for resume uploads but it is not implemented yet so if it fails, you have to redo the whole file currently.

6

Great question...

For just about all the providers the answer is yes. The transfer is atomic and you will never see a partially uploaded file.

The exceptions are the local disk. FTP and SFTP.

At some point I would like rclone to upload to a temporary file name then rename it over the existing file like rsync does when used without the flag --inplace.

7

Thanks. Makes sense. Given the fact that rclone doesn’t do rsync style block transfers, it makes sense that you could do a transfer and move.

1 Like
8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.