I'm trying to get away for using Syncthing for syncronizing a large folder (1.2TB) with thousands of files, basically because Syncthing doesn't really work well all the time, machines don't get connected, etc.
My user case is I need to mirror this folder I have in a remote machine with a local machine. Ideally, with the remote folder being the main folder that should be mirrored. I would like to be able to create a cron job that runs rclone periodically to be sure those folders are in sync.
I did it using rclone sync "remote:media/" /mnt/4tb/media/ --checkers=1 with no issue, but it took 4 hours to finish.
I need to know if this is the more efficient way of doing what I need, considering that is not scalable to wait for 4 hours every time I need to pull some files from the remote machine. Maybe I'm misunderstanding something, I would really appreciate some guidance.
Run the command 'rclone version' and share the full output of the command.
rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.18
Which cloud storage system are you using? (eg Google Drive)
I'm not using a cloud storage but a remote machine SSH connected.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
I'm receiving some errors that are forcing rclone to "retry", but this –again- slow the process a lot. I cancelled it after 27 minutes while it was retrying for the third time.
Maybe a little bit off topic (should I create another thread for this?) but I'm not sure how to fix the error messages I'm receiving. I run rclone with -vv --log-file=mylogfile.txt and the log text is not really helpful to find how to fix it to be honest.
cla@x220 ~> grep ERROR mylogfile.txt | less
2023/01/05 10:38:56 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 10:42:30 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 10:51:40 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 10:51:40 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 10:51:40 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 10:51:44 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 11:00:52 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 11:00:52 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 11:00:52 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 11:00:55 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 11:10:02 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 11:10:02 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 11:10:02 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
(END)
Looking it more in detail
126720 │ 2023/01/05 10:51:40 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
126721 │ 2023/01/05 10:51:40 NOTICE: media: Can't follow symlink without -L/--copy-links
126722 │ 2023/01/05 10:51:44 DEBUG : .DS_Store: Size and modification time the same (differ by 0s, within tolerance 1s)
126723 │ 2023/01/05 10:51:44 DEBUG : .DS_Store: Unchanged skipping
126724 │ 2023/01/05 10:51:44 DEBUG : .stignore: Size and modification time the same (differ by 0s, within tolerance 1s)
126725 │ 2023/01/05 10:51:44 DEBUG : .stignore: Unchanged skipping
126726 │ 2023/01/05 10:51:44 DEBUG : .com.apple.timemachine.supported: Size and modification time the same (differ by 0s, within tolerance 1s)
126727 │ 2023/01/05 10:51:44 DEBUG : .com.apple.timemachine.supported: Unchanged skipping
126728 │ 2023/01/05 10:51:44 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
cla@x220 ~> grep ERROR mylogfile.txt | less
2023/01/05 11:56:39 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 12:05:44 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 12:05:44 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 12:05:44 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 12:05:48 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
2023/01/05 12:14:52 ERROR : Local file system at /mnt/4tb/media: not deleting files as there were IO errors
2023/01/05 12:14:52 ERROR : Local file system at /mnt/4tb/media: not deleting directories as there were IO errors
2023/01/05 12:14:52 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: Open failed: file does not exist
2023/01/05 12:14:56 ERROR : media: Failed to copy: failed to open source object: Open failed: file does not exist
The ERROR message should tell you the name of the file that rclone couldn't copy - in this case media.
yeah, but media is the main folder I'm trying to get in sync, it's not a file.