More or less I am looking for someone to brainstorm with me. Long story short I have tons of data (roughly 10TB) I need to copy to my GDrive. It is not from another cloud storage, it will be a local to remote copy. What I would like to do is run some sort of rclone copy, nightly, for a certain amount of time. Then start it again the next night…until its all copied.
I know I could do something like this with a cronjob but the part that I can’t wrap my head around is this… How could I tell rclone what to upload and how long to upload it? Could I do it by an amount of data, number of folders/files? Should I separately create file lists using some criteria, then copy from that list? Any ideas/advice would be much appreciated…thanks in advance.
rclone will pick up from where it left off so you don’t need to do that.
In fact you might find that bwlimit is what you want - you can set a bandwidth limit for the night and for the day. The bandwidth limit for the day could be very small (don’t use 0 though as that means unlimited!) then you can just leave rclone running.
I will have a look at bwlimit. Thank you very much!
bwlimit does seem to be exactly what I want. is there a way to put this into a rclone config file of sorts in order to impose those bwlimits at all times, for whenever a rclone command is sent? or it needs to be added to each command?
You need to add it to each command, or you could set an environtment variable export RCLONE_BWLIMIT=...
One last question and I think I’ll have what I need. I saw in a post, somewhere here, that there is no way to copy a folder directly? That it will only copy the contents? Is that true or am I missing something? Is there an option I’m not seeing?
This is of course without specifying the folder on the destination. So “rclone copy /this/folder/ gdrive:” that won’t copy to gdrive /folder/contents right? It would be gdrive /source contents?
rclone copy works like rsync with a / on the end so it only copies the contents. This isn’t a big deal in practice - if you want the actual folder then just name it on the command line.
So lets say /tmp/a has /tmp/a/file.txt in
rclone copy /tmp/a gdrive:
will copy file.txt to gdrive:file.txt
If you want to copy the directory just do
rclone copy /tmp/a gdrive:a
which will copy file.txt to gdrive:a/file.txt
This all makes more sense if you realise that copy is really just rclone sync under the hood with no deletes, and sync synchronises two directories which need naming on the command line.