I was answering this reddit question and it got me to thinking, are rclone transfers atomic at the file level? Does it depend on the remotes? File sizes (multi-part uploads?).
Meaning that if I am uploading to a cloud remote and say another person uploads the same file, what happens? Atomic I always thought referred more to a file system and how file locking worked if multiple things were trying to write to the same file handle.
I made two test files and started 1 upload and a few seconds later, started a second to the same file name:
felix@gemini:~$ fallocate -l 1G test.img felix@gemini:~$ rclone copy test.img GD: -P Transferred: 1G / 1 GBytes, 100%, 34.531 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 29.8s felix@gemini:~$ rclone lsl GD:test.img 1073741824 2021-01-15 16:48:22.751000000 test.img
GD is a bit unique as I'll get double files since I can have duplicates..
felix@gemini:/data$ rclone copyto test2.img GD:test.img -P Transferred: 2G / 2 GBytes, 100%, 38.931 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 57.9s felix@gemini:/data$ rclone lsl GD:test.img 2147483648 2021-01-15 16:48:50.167000000 test.img 1073741824 2021-01-15 16:48:22.751000000 test.img
There is no file locking on the remote.
Atomic meaning the file is either there fully or it is not. For example, if you are transferring a file (let's say a large one but the idea remains regardless) and your internet cuts out, will it show up at all?
Also, to say I am jealous of your upload speed is an understatement!
There is a feature out there for resume uploads but it is not implemented yet so if it fails, you have to redo the whole file currently.
For just about all the providers the answer is yes. The transfer is atomic and you will never see a partially uploaded file.
The exceptions are the local disk. FTP and SFTP.
At some point I would like rclone to upload to a temporary file name then rename it over the existing file like rsync does when used without the flag
Thanks. Makes sense. Given the fact that rclone doesn’t do rsync style block transfers, it makes sense that you could do a transfer and move.