Is there a better way to sync?

What is the problem you are having with rclone?

I'm trying to get away for using Syncthing for syncronizing a large folder (1.2TB) with thousands of files, basically because Syncthing doesn't really work well all the time, machines don't get connected, etc.

My user case is I need to mirror this folder I have in a remote machine with a local machine. Ideally, with the remote folder being the main folder that should be mirrored. I would like to be able to create a cron job that runs rclone periodically to be sure those folders are in sync.

I did it using rclone sync "remote:media/" /mnt/4tb/media/ --checkers=1 with no issue, but it took 4 hours to finish.

I need to know if this is the more efficient way of doing what I need, considering that is not scalable to wait for 4 hours every time I need to pull some files from the remote machine. Maybe I'm misunderstanding something, I would really appreciate some guidance.

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.18

Which cloud storage system are you using? (eg Google Drive)

I'm not using a cloud storage but a remote machine SSH connected.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync "remote:media/" /mnt/4tb/media/ --checkers=1

The rclone config contents with secrets removed.

[whatbox]
type = sftp
host = 
user = user
port = 22
pass = 
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command with the -vv flag

NA

That's a super old version so you'd want to update.

I'd remove that and retest with the new version. 1 checker would make things very slow. Best to try with the defaults first.

Thanks, it is faster indeed.

I'm receiving some errors that are forcing rclone to "retry", but this –again- slow the process a lot. I cancelled it after 27 minutes while it was retrying for the third time.

Maybe a little bit off topic (should I create another thread for this?) but I'm not sure how to fix the error messages I'm receiving. I run rclone with -vv --log-file=mylogfile.txt and the log text is not really helpful to find how to fix it to be honest.

cla@x220 ~> grep ERROR mylogfile.txt | less
2023/01/05 10:38:56 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 10:42:30 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 10:51:40 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 10:51:40 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 10:51:40 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 10:51:44 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 11:00:52 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 11:00:52 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 11:00:52 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 11:00:55 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 11:10:02 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 11:10:02 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 11:10:02 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
(END)

Looking it more in detail

126720   │ 2023/01/05 10:51:40 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
126721   │ 2023/01/05 10:51:40 NOTICE: media: Can't follow symlink without -L/--copy-links
126722   │ 2023/01/05 10:51:44 DEBUG : .DS_Store: Size and modification time the same (differ by 0s, within tolerance 1s)
126723   │ 2023/01/05 10:51:44 DEBUG : .DS_Store: Unchanged skipping
126724   │ 2023/01/05 10:51:44 DEBUG : .stignore: Size and modification time the same (differ by 0s, within tolerance 1s)
126725   │ 2023/01/05 10:51:44 DEBUG : .stignore: Unchanged skipping
126726   │ 2023/01/05 10:51:44 DEBUG : .com.apple.timemachine.supported: Size and modification time the same (differ by 0s, within tolerance 1s)
126727   │ 2023/01/05 10:51:44 DEBUG : .com.apple.timemachine.supported: Unchanged skipping
126728   │ 2023/01/05 10:51:44 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist

This will likely be files that change during the sync.

The ERROR message should tell you the name of the file that rclone couldn't copy - in this case media.

This is the error

So either add -L to look through the symlink and copy where it points to, -l to copy it as a symlink or add a filter to ignore it.

Thanks!

I run it with --skip-links, no luck:

cla@x220 ~> grep ERROR mylogfile.txt | less
2023/01/05 11:56:39 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 12:05:44 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 12:05:44 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 12:05:44 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 12:05:48 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 12:14:52 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 12:14:52 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 12:14:52 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 12:14:56 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist

The ERROR message should tell you the name of the file that rclone couldn't copy - in this case media.

yeah, but media is the main folder I'm trying to get in sync, it's not a file.

What is it? Can you ls -l media and post the result? also stat media would be useful.

Rclone thinks it is a file or maybe a symlink, so its certainly confused!

Is it possible rclone doesn't have permissions to read media?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.