How to make Dropbox <> SFTP Backup reliable?

[rcuser]

Hello

First I’d like to say thanks for developing rclone because I think it’s a great tool once you’ve figured out the right parameters

Actually I want to achieve a reliable backup of my Dropbox to an SFTP server. I’m running two cron jobs. One a week I run a copy job:
rclone copy dropbox:my-files sftpserver:my-files -v --log-file=copy-$DATE.log

The other days I run a sync job:
rclone sync dropbox:my-files sftpserver:my-files --backup-dir sftpserver:backup-$DATE -v --log-file=sync-$DATE.log

In general rclone works as expected but I’m concerned about the reliability. The log of the copy job has several messages like this: error reading source directory: too_many_requests/

The summary says:
Transferred: 23.572 GBytes (804.368 kBytes/s)
Errors: 32
Checks: 189909
Transferred: 118911
Elapsed time: 8h32m8.6s

So it seems 32 files could not be transferred due to errors above. I would expect the sync job running the other days would at least copy the 32 missing files. But the summary says:
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 356723
Transferred: 0
Elapsed time: 1h22m11.2s

How can I achieve a reliable copy/sync job so I can be sure all files of my Dropbox are safely backed up? Are there any further rclone parameters needed?

[ncw (Nick Craig-Wood)]

@rcuser

First I’d like to say thanks for developing rclone because I think it’s a great tool once you’ve figured out the right parameters

Thanks!

@rcuser

How can I achieve a reliable copy/sync job so I can be sure all files of my Dropbox are safely backed up? Are there any further rclone parameters needed?

In general cloud stuff can be unreliable - the Internet is not 100% reliable...

rclone provides two flags to help you with that --retries and --low-level-retries - you can increase those to make it more reliable.

In general rclone will pick up anything that was left out the next run, but...

@rcuser

So it seems 32 files could not be transferred due to errors above. I would expect the sync job running the other days would at least copy the 32 missing files.

I would too but it depends on whether those files are still there - maybe they were temporary files?

If you check the log looking for ERROR messages maybe you'll get an idea of what happened.

[rcuser]

@ncw

In general cloud stuff can be unreliable - the Internet is not 100% reliable…

rclone provides two flags to help you with that --retries and --low-level-retries - you can increase those to make it more reliable.

In general rclone will pick up anything that was left out the next run, but…

In the log of the last copy job I see those two different error messages:

• Failed to copy: failed to open source object: unexpected end of JSON input
• error reading source directory: too_many_requests/.

Are these known errors when working with Dropbox? Do you think increasing --retries and --low-level-retries could help here?

@ncw

So it seems 32 files could not be transferred due to errors above. I would expect the sync job running the other days would at least copy the 32 missing files.

I would too but it depends on whether those files are still there - maybe they were temporary files?

If you check the log looking for ERROR messages maybe you’ll get an idea of what happened.

I don't have temporary files in my Dropbox folder. But to be sure I will retry this behavior. I ran a copy job last night which resulted in 43 errors. Today I didn't put/delete any files in my Dropbox. This night a sync job will run so rclone must pick them up. I will post the result tomorrow.

[ncw (Nick Craig-Wood)]

@rcuser

In the log of the last copy job I see those two different error messages:

Failed to copy: failed to open source object: unexpected end of JSON input
error reading source directory: too_many_requests/.

I'm not sure what causes the first error - sounds like either a networking problem or a dropbox problem.

The second error is caused by dropbox rate limiting. rclone should backoff and retry at that point. However rclone will only try --low-level-retries times before it gives up which it sounds like it is.

@rcuser

Are these known errors when working with Dropbox? Do you think increasing --retries and --low-level-retries could help here?

It might do...

One thing that might help is to set --tpslimit to try to avoid the dropbox rate limiting. I'd try --tpslimit 5 to start with and see what happens.

[rcuser]

@ncw

I’m not sure what causes the first error - sounds like either a networking problem or a dropbox problem.

The second error is caused by dropbox rate limiting. rclone should backoff and retry at that point. However rclone will only try --low-level-retries times before it gives up which it sounds like it is.

In the log I see 3 attempts before rclone skipping the file. Do you suggest increasing --low-level-retries to 10?

@ncw

One thing that might help is to set --tpslimit to try to avoid the dropbox rate limiting. I’d try --tpslimit 5 to start with and see what happens.

I will try this option for the next copy job.

@rcuser

I don’t have temporary files in my Dropbox folder. But to be sure I will retry this behavior. I ran a copy job last night which resulted in 43 errors. Today I didn’t put/delete any files in my Dropbox. This night a sync job will run so rclone must pick them up. I will post the result tomorrow.

Todays sync log summary says:
Transferred: 75.386 kBytes (43 Bytes/s)
Errors: 0
Checks: 127889
Transferred: 3
Elapsed time: 29m44.1s

It seems 40 files where missing according to the 43 reported errors of the last copy job. Or am I misunderstanding something here?

[ncw (Nick Craig-Wood)]

@rcuser

In the log I see 3 attempts before rclone skipping the file. Do you suggest increasing --low-level-retries to 10?

Yes. The default is10, so I think that is a good number.

@rcuser

It seems 40 files where missing according to the 43 reported errors of the last copy job. Or am I misunderstanding something here?

The way to find out exactly what is happening is to run rclone check that will tell you exactly what is where.

[rcuser]

@ncw

Yes. The default is10, so I think that is a good number.

The way to find out exactly what is happening is to run rclone check that will tell you exactly what is where.

Ok thanks so far. I will run a copy + check job tonight with --tpslimit 10 and --low-level-retries 10. I will report the result tomorrow.

[rcuser]

@rcuser

Ok thanks so far. I will run a copy + check job tonight with --tpslimit 10 and --low-level-retries 10. I will report the result tomorrow.

Unfortunately the check job didn't run this night. There is the notice 127889 hashes could not be checked. Do you know what could be wrong here?

[ncw (Nick Craig-Wood)]

@rcuser

Unfortunately the check job didn’t run this night. There is the notice 127889 hashes could not be checked. Do you know what could be wrong here?

The SFTP server and dropbox dont have a common hash so the hashes can't be checked. rclone will check file existence and size though which is a reasonable check.

[rcuser]

@ncw

The SFTP server and dropbox dont have a common hash so the hashes can’t be checked. rclone will check file existence and size though which is a reasonable check.

Ok I have set up a check job directly after the copy and sync jobs so I can watch the logs the next days if everything runs smoothly.

[rcuser]

The rclone backup job seems to run smoothly now since the last weeks. The parameters --tpslimit 10 and --low-level-retries 10 seem to be perfect for dropbox.

The subsequent check reports:
0 differences found
138911 hashes could not be checked

I understand the hashes were different between SFTP server and dropbox. So can I be sure the transfer was successful due to 0 differences found message?

[ncw (Nick Craig-Wood)]

@rcuser

The rclone backup job seems to run smoothly now since the last weeks. The parameters --tpslimit 10 and --low-level-retries 10 seem to be perfect for dropbox.

Great!

@rcuser

I understand the hashes were different between SFTP server and dropbox. So can I be sure the transfer was successful due to 0 differences found message?

You can be sure that all the files are present and they have their correct sizes. Unfortunately because the two don't have a compatible hash you can't know anything about the contents of the files.