I was brainstorming a way to expedite my upload to Google Drive and had the following idea. I have access to an internet connection with upload that is over 25x faster than my home connection. I would like to use rclone to determine what local data needs to be uploaded to Google Drive and then copy it to a USB hard drive instead. Then I could take the USB disk to the site with faster internet and use rclone COPY to send it to Google Drive.
Ultimately this boils down to rclone running its initial scan against one remote (Google Drive) but performing the actual copy to another remote (USB storage). If this is possible now I can’t figure it out. I am marking it as a feature request so ncw will hopefully chime in.
You could do this with a bit of scripting I think…
First do a copy with --dry-run from local -> drive
Parse the log file to make a list of files which need to be copied
Use --files-from and this list of files to rclone copy from local -> usb
When on faster internet do rclone copy from usb -> drive (no need for the --files-from at this point)
I think the only hard part is the “parse the log file” part. You could do that with grep and sed, or you could edit in an editor or 100 different ways depending on what you find easiest.
I’ve been chewing over an approach like that. It’s possible but I’m just not confident enough in my own sed/grep voodoo. Could rclone have a parameter that would generate the file list for use with --files-from? That would be a lot cleaner and could be useful in a few use cases.
I have different files with the same names but in different sub directories. That would mean that if any of the files that share a name is added to files-from.txt then all of the files would be copied in the subsequent jobs, right? It seems like the files-from.txt would need to include path information in order to prevent this. I don’t see a way to get paths from the dry-run output. Any ideas?
I know --files-from takes paths. The part that concerns me is that our regex voodoo against the --dry-run output does not include any paths. So the resulting text file is just a list of filenames. Possible to modify the --dry-run output to include paths?
Hmm, for whatever reason I did not see any paths when reviewing the --dry-run log output. You are correct, of course. Sorry for the confusion. I was just trying to anticipate potential problems before they arose. This is working perfectly so far. Thanks for your help!