Automatic stop or restart sync


#1

Hey guys. I am very new to linux and rclone but I’ve managed to get my synology run scheduled (once a day) tasks to sync some of my folders to the Gteamdrive. Most of em doing great but when I am syncing my plex folder it is getting errors very often and then it’s a neverending loop of:

2019/03/12 07:28:54 INFO :
Transferred: 183.231M / 183.231 MBytes, 100%, 2.750 kBytes/s, ETA 0s
Errors: 5 (retrying may help)
Checks: 44641 / 44641, 100%
Transferred: 1371 / 1371, 100%
Elapsed time: 18h57m2.1s

Yeah I can putty in and kill the process to restart it manually but I rly want this to be automatic so can anyone pls help me modify my script so the sync would stop or restart itself when it’s done and won’t loop till the end of time? (rclone 1.45, running on synology NAS):

/usr/bin/rclone --exclude “#recycle/" --exclude "@eaDir/” --exclude “@eaDir/” sync -v --config="/root/.config/rclone/rclone.conf" /volume1/Plex GTdrive:Plex --log-file=/volume1/homes/admin/RcloneLogs/syncplex.log


#2

What are the errors you are getting in the logs? I’d probably look to see if those fixable before re-running it personally.


#3

There are always gonna be some errors. So I rly need a way to stop or rerun via script.


#4

Can you share examples from the logs of what those errors are?


#5

Sure:
error reading destination directory: couldn’t list directory: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded

Failed to copy: can’t copy - source file is being updated (size changed from 554671 to 555214)

Failed to copy: googleapi: Error 500: Internal Error, internalError

Doesn’t look like those can be solved :).


#6

Usually rclone will retry such an error.

Are you using your own client_id - that will help

That means that the file changed during the upload

Rclone will retry that error too.

Do you see some lines below that - if so then rclone is working on transferring those files.


#7

Well it’s not transfering files but since it’s Plex folder (with metadata) it’s always changing rclone keeps updating modification time I guess. This is how it looks like now:

2019/03/13 10:48:39 INFO : Library/Application Support/Plex Media Server/Media/localhost/f/8182607ed9a400aab66fddf27a0d32d13aecf0c.bundle/Contents/Subtitles.xml: Updated modification time in destination
2019/03/13 10:49:11 INFO :
Transferred: 113.479M / 113.479 MBytes, 100%, 3.287 kBytes/s, ETA 0s
Errors: 7 (retrying may help)
Checks: 30555 / 30555, 100%
Transferred: 52 / 52, 100%
Elapsed time: 9h49m9.6s

2019/03/13 10:50:11 INFO :
Transferred: 113.479M / 113.479 MBytes, 100%, 3.281 kBytes/s, ETA 0s
Errors: 7 (retrying may help)
Checks: 30610 / 30610, 100%
Transferred: 52 / 52, 100%
Elapsed time: 9h50m9.6s

2019/03/13 10:51:11 INFO :
Transferred: 113.479M / 113.479 MBytes, 100%, 3.275 kBytes/s, ETA 0s
Errors: 7 (retrying may help)
Checks: 30722 / 30722, 100%
Transferred: 52 / 52, 100%
Elapsed time: 9h51m9.6s

2019/03/13 10:52:11 INFO :
Transferred: 113.479M / 113.479 MBytes, 100%, 3.271 kBytes/s, ETA 0s
Errors: 7 (retrying may help)
Checks: 30786 / 30787, 100%
Transferred: 52 / 52, 100%
Elapsed time: 9h52m9.6s
Checking:

  • Library/Application Support/Plex Media Server/Media/localhost/9/875b120971f316d2e7067a4cb499ede6964c870.bundle/Contents/Subtitles.xml

2019/03/13 10:52:12 INFO : Library/Application Support/Plex Media Server/Media/localhost/9/875b120971f316d2e7067a4cb499ede6964c870.bundle/Contents/Subtitles.xml: Updated modification time in destination
2019/03/13 10:52:13 INFO : Library/Application Support/Plex Media Server/Media/localhost/b/b6f545cfcade1279a44061fed32ad5bf39ea657.bundle/Contents/Subtitles.xml: Updated modification time in destination
2019/03/13 10:53:05 INFO : Library/Application Support/Plex Media Server/Media/localhost/8/a6ab5b3a5fb75959b510565a076b0953225185b.bundle/Contents/Subtitles.xml: Updated modification time in destination
2019/03/13 10:53:05 INFO : Library/Application Support/Plex Media Server/Media/localhost/4/ae7a6c9f12a720c91f94b0f72148a372747fec5.bundle/Contents/Subtitles.xml: Updated modification time in destination


#8

So it is still running through, checking files need to be uploaded or not. You can see this by the incrementing checks. The updating modification time may be to do with where it is served from - you could try setting --modify-window 1s.

Are you using your own client_id - that will help with the speed, as will --fast-list.


#9

Trying to sync your Plex directory is probably one of the worst use case solutions for rclone and GD as it is a ton of little files and you can only do 3 files uploaded per second.

You’d get much better throughput / use and restorability if you tar’ed up the directory and upload the tarball instead if that’s your goal.

My plex directory is 1,243,196 files.


#10

Hm. Ofc you are right. Ur way is way more efficient. I am struggling though how to implement it to run on schedule via script. Any advice?


#11

I wonder if some folks on the forum have a better way, but my way is I backup locally to another disk and really just keep that as a mirrored backup.

My worse case is I sync to trakt and I can always get my viewing status back if I had a real problem.

You could something like:

cd /var/library/plexmediaserver
sudo tar cf /your-home-directory/PlexLibraryBackup.tar  ./Library
rclone copy /your-home-directory/PlexLibraryBackup.tar

And put some dates or something in it. I haven’t given it too much thought but I can take a look to see what else I can find.