I'm using Rclone inside a shell script to send a Centos7 backup folder to Google drive. The purpose is to keep a remote clone of my server backups. The backup is running, however the source folder is 48GB and so far the backup is on 102GB and still going!
I'm using the drive.file scope in config for google drive.
The source backups folder being copied contains tar's, sql dumps and (probably more relevant) a folder with rsync backups of complete system (using laurent22/rsync-time-backup). There is a great deal of sym/hard links in this folder.
First run I used the --copy-links switch and it resulted in a gigantic folder. I misunderstood the command and appears the sym/hard links were followed and copied as files /folders.
I'm now using the -l switch assuming the sym/hard links are copied as links (not followed). The log file shows thousands of .rclonelink files so it appears to work as intended.
the backup has been running for over 17 hours. is there a chance that some of the files that are to be backed-up are being modified while rclone is running.
there are over 300 errors, why so many, what are they?
I'm also surprised with the time it's taken so far! This source folder would only grow by a few MB's from one day to another. It's only accessed briefly, once a day when the backup(s) take place. Looking at the errors, seems mainly down to not excluding certain folders/ files which I'm adding to the exclusions as they come up.
normally, you would start a new post using the question template.
if you had, you would be ask to answer the following questions.
please answer them, thanks
What is the problem you are having with rclone?
What is your rclone version (output from rclone version)
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
Evil, but I'd kill it and restart and specify -P and target ONLY a single small-ish top-level directory to start with. Keep going bigger until it fails.
Using -P will talk to you (it's like rsync -P) and needs to be pointed at a terminal. (Will generate a bunch of junk if redirected to a file.)
You might also use "ncdu" for a local vs remote manual size compare. You could also compare a "ls -lR local" vs "rclone ls target:dir" and compare them after a data massage. If you didn't clean out your original copy of the linked files, I bet they're still there "helping out."
And RESTARTING the copy will only copy new/modified files, it won't copy everything all over again. (like rsync -rt) It WILL take a few seconds (minutes/hours) to skip over the existing files doing what I think is a time/date/size check.)
What is it?? NOT SURE HERE -- isn't it "rclone sync" that will DELETE all the extra files found on the remote? Unsure how it would handle "-l" vs actual files, though. --dry-run is your friend.