Does rsync and rclone interfere?

Help and Support
1

What is the problem you are having with rclone?

I'm trying to transfer data onto a NAS. Because I have so many files to transfer, I put them in a temporary folder (using rclone on the PC used to download the data), from where I transfer them to their final destination (using rsync on the NAS).

Moving the files from the temporary folder to the final folder has been done overnight using a command similar to this one:

rsync --remove-source-files --recursive tmpobjects/ objects/ &

The ampersand at the end makes it, so the process frees the command line. I've chosen this method because the SSH connection to the NAS would close every once in a while, and therefore interrupt the rsync command.

While the command runs, I normally continue to put more files onto the NAS using rclone, but into a different folder. Let's call it tmpobjects2. After the files are done moving, I normally repeat the rsync command with the tmpobjects2 folder.

Yesterday I realized that one rsync command apparently didn't shut down properly, and I don't know for how long it's been running. I'm uncertain if I continued putting more files into the tmp folder, which the rsync command tried to continue emptying.

My question: would the two commands somehow interfere and cause data loss because of unfinished/partial data files? How can I figure out if data has been overwritten incorrectly?

I'd appreciate any help here, since I'd have to, painfully, roll back about 5 days of work. Thank you.

(I am aware that I could write to objects directly, but there are just too many files in there and rclone would take too long to even start. And since I have to work in batches, because of several other reasons, I chose this procedure.)

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1
- os/version: darwin 13.4.1 (64 bit)
- os/kernel: 22.5.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.6
- go/linking: dynamic
- go/tags: none

NAS:

$ rsync --version
rsync  version 3.1.2  protocol version 31

Which cloud storage system are you using? (eg Google Drive)

Local drive to Synology NAS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Move files onto NAS:

rclone move /Volumes/extSSD/tmpobjects/ ds220j:"/home/tmpobjects/" --progress --exclude ".*"

Move files from tmpobjects to final destination:

rsync --remove-source-files --recursive tmpobjects/ objects/ &

The rclone config contents with secrets removed.

The relevant config is the one for the DS220j, I suppose. It's configured as SMB remote.

Editing existing "ds220j" remote with options:
- type: smb
- host: DS220j

A log from the command with the -vv flag

I don't know if this helps us solve this particular question. If necessary, I will add the output.

2

"The ampersand" method easily can lead to situations like yours. Install and learn how to use tmux

Difficult to say but when you let two different programs operate on the same data results can be random. With your staged method and using rclone move you can not verify anything as you do not have source to compare anymore.

1 Like
3

And the suggestion would be to repeat the work of the last 5 days?

4

Unless you can verify your data you will never know if things are correct or not.

5

Thank you very much for the quick help here, and for confirming how to resolve this issue. I also think it makes the most sense to unroll the last x days, and just repeat it. I make sure to have a read through the tmux docs, too.

6

Other thing is why to use rclone at all? Wouldn't be easier to connect disk directly to your NAS and rsync to objects?

7

rclone's docs say it copies source to dest, then deletes the source, "if no errors on copy". Copying directly doesn't work as flawlessly as I'd like it to.

8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.