Advice & Questions re. Backing up approx 1Tb of small files to Dropbox,

Hi I’m new to the group , been using rclone from my raspberryPi for a few weeks and generally working well. My RPI is an rsync server for various cmputers in my house. They back up once an hour, then every night the RPI backs up the backup to a second mirror drive, then once a week I backup the mirror to dropbox using rclone.

My command line is

/usr/local/bin/rclone sync -q --exclude-from /etc/rclone_exclude.txt /Archive Dropbox:Archive >> /var/tmp/log/rclone.log 2>&1
A few questions

  1. It seems to take approx 3 days for rclone to finish, I get a lot of lock and other errors which I understand is dropbox timing out . What seems odd is that each week it takes approximately the same time (3 days). I’d imagined that once the first sync had been done, subsequent syncs would be much shorter as with rsync ? Perhaps because it completes with errors each time its re-syncing everything ?

  2. I’m using the sync flag but I dont see any obvious evidence of deletions . Would I expect to see them in the dropbox deleted folder or is it an absolute delete i.e removed and not recoverable ?

  3. if the rclone is very long running, I would quite like to limit it to run in shorter chunkc out of hours i.e nighttime, when no other use on my network and also my datacentre (and disks) are coolest. Is there a smart way to break up the transfer ? I guess I could simply kill it from cron and restart the next evening, but I wonder if there is a smarter way

  4. The errors I’m getting are typically lots and lots of

  • The server has either erred or is incapable of performing the requested operation
  • - Failed to grab locks for 20653169: lock held by connection 21018547. Please re-issue the request.
  • - Failed to copy: upload failed: Internal Server Error

Would throttling the transfer rate provide a slower but more reliable session ?

many thanks

Jonathan

OK - I think the abundance of Lock errors I’m getting is due to the “rats nest” of hard links that Apple created when it migrated from iPhoto to Photo. it seems like Photo has access to all iPhoto images via a system of hard links, and if I exclude these my ERROR rate drops dramatically.

It might do…

However if you aren’t using the latest beta then have a go with it as it uses the v2 dropbox API which is better in a lot of ways.

Thanks, I will. The hard links definitely seem to be related to the number of “failed to grab locks” errors. I will try the beta, thanks for the suggestion

well my weekly cron ran with the beta on monday and finsihed around 80 hours later. I still had 16 errors including and 3 goes ( retries=3)

Failed to copy: upload failed:
Failed to copy: upload failed: json: cannot unmarshal object into Go struct field UploadAPIError.error of type string
Failed to copy: upload failed: unexpected end of JSON input

I ran it again and it seemed to still want to upload the same files again, it doesnt seem to know what its already uploaded ?

This is the dropbox server returning an HTML error most likely.

It should upload the files that was missed the first time not all of them.

Thanks, I ran with a -vv flag and yes, I could see it skipping a lot of files, but I could also see it recopying many files that had previously been copied and I can see at both source and target destinations ?

the recopies are more than the handfull or ERRORS (17) I got last time.

I’m trying to figure out what the problem is. I tried --size-only but it didnt make any difference. I have a pdf file inn my home directory which I dont update, yet it seems to need to recopy it each time, despite the timestamp not changing ?

I’m backing up from a raspberry Pi/ ext4 filesystem, but the files have been rsync’d from a mac, I wonder if some file update is being performed by the rsyncd server which causes the files to appear to be new

You might be having filename normalization problems… Try adding --local-no-unicode-normalization and see if that helps.

Thanks, I did a test run adding the --local-no-unicode-normalisation and I still see the same files being copies, apparently unnecessarily. There are a lot of skipped, and possibly that has improved things, I have added the clause to my weekly crontab entry and I’ll see. Thanks for your support. rclone is working in that all files seem to being backed up now, its just some should not need to be recopied and some should be deleted. I accept that the home directories of mac users, at a OSX filesystem level, are a little odd compared to most Linux/Unix systems. Thanks again, its doing its job

Could you paste the lined from the log (with -vv) which mention one of those files which get copied unnecessarily. Save the log to a file and use grep to find all the lines. I’d like to work out the reason why it is happening.

Sure,

So I ran this command from the shell

/usr/local/bin/rclone sync --exclude-from /etc/rclone_exclude.txt --local-no-unicode-normalization -vv /Archive/rsync Dropbox:Pi/rsync >> /var/tmp/log/test1

let it run for 5 mins , then I ran it a second time

/usr/local/bin/rclone sync --exclude-from /etc/rclone_exclude.txt --local-no-unicode-normalization -vv /Archive/rsync Dropbox:Pi/rsync >> /var/tmp/log/test2

grep Daisy test*

test1:2017/07/21 12:45:26 INFO : Jonathan/Family/DaisyDentwedcert.jpg: Copied (new)
test1:2017/07/21 12:45:32 INFO : Jonathan/Family/DaisyMaryDentweddcert.jpg: Copied (new)
test2:2017/07/21 12:50:17 INFO : Jonathan/Family/DaisyDentwedcert.jpg: Copied (new)
test2: * Jonathan/Family/DaisyMaryDentweddcert.jpg: 15% done, 0 Bytes/s, ETA: -
test2:2017/07/21 12:50:25 INFO : Jonathan/Family/DaisyMaryDentweddcert.jpg: Copied (new)

#ls -al /Archive/rsync/Jonathan/Family/Daisy*
-rw-r–r-- 1 833762 Jan 9 2003 /Archive/rsync/Jonathan/Family/DaisyDentwedcert.jpg
-rw-r–r-- 1 2149796 Jan 7 2003 /Archive/rsync/Jonathan/Family/DaisyMaryDentweddcert.jpg

/usr/local/bin/rclone ls -vv Dropbox:Pi/rsync/Jonathan/Family|grep Daisy

2017/07/21 13:17:32 DEBUG : rclone: Version “v1.36-241-g92294a4aβ” starting with parameters ["/usr/local/bin/rclone" “ls” “-vv” “Dropbox:Pi/rsync/Jonathan/Family”]
2017/07/21 13:17:33 INFO : Dropbox root ‘Pi/rsync/Jonathan/Family’: Modify window is 1s
833762 DaisyDentwedcert.jpg
2149796 DaisyMaryDentweddcert.jpg
2017/07/21 13:17:34 DEBUG : Go routines at exit 8
2017/07/21 13:17:34 DEBUG : rclone: Version “v1.36-241-g92294a4aβ” finishing with parameters ["/usr/local/bin/rclone" “ls” “-vv” “Dropbox:Pi/rsync/Jonathan/Family”]

if I login via web to dropbox

Can you make please make a new issue on github about this please?

If you could run two complete syncs with -vv and attach the full logs that would be really helpful - thanks! (zip them first before attaching).

I’m not sure what is going on, but it will be in the logs somewhere!

Thanks, I will do that.

A full rclone sync takes approx 80 hours. I have run one and this has given me some considerable insight:

I am using rclone to cloudbackup up a directory which contains 3 sub directories, one for each computer I have. Looking through the full log with -vv set I realised that only one of the three subfolder is having problems. two of them behave correctly, skipping file copies as I’d expect. the third one NEVER skips, and copies EVERY file, no matter what I do.

Apparently the 3 directories are the same AFAIK, their unix ownerships are different , but I dont think that should be an issue ?

I’m starting to think that the offending directory has some problem at the destination ? I think I pre-created its folder name via the dropbox UI when I first started. I have removed the entire directory from dropbox, and I wanted to also remove it from the Deleted folder . This is proving hard as some of the subfolders are huge (apple iPhoto) , so I may need to wait for 30 days for dropbox to purge out all of the deleted stuff.

I wonder also if this strange case semi-sensitivity that Drpbox has is an issue. The offending folder originally had been created All lower case, but locally the first character was upper. A quick run after removing let rclone recreate it with upper , but still it seemed on a second pass to recopy ?

if I exclude the offending folder, rclone runs fine, completes on the second pass and even deleted some files (correctly) from the server

Dropbox is a bit funny about capitalisation of things. Sometimes it lowercases paths, all except for the leaf.

This issue is possibly related: https://github.com/ncw/rclone/issues/1558

I’m concentrating on getting clear runs on the two folders that seem to be OK, while I wait for Dropbox to clear out the failed folder from Deleted Items .

I noticed this abnormality which might be of interest. Looks like Dropbox also doesn’t like folders with a trailing space character. I have one and it always fails this way

2017/07/27 04:04:11 ERROR : Edward/Final drafting/March 1st/March 3rd/March 5th /chapter 2 March 5th.docx: Failed to copy: upload failed: json: cannot unmarshal object into Go struct field UploadAPIError.error of type string

If I rename the ./March 5th / to ./March 5th/ then it copies fine. I think the folder was named like that by accident, but it should be legal ?

I can’t find the dropbox docs at the moment, but I know other providers don’t allow this (Windows doesn’t). I suggest you rename it.