Simultaneous rclone copy commands on a single directory

Help and Support
1

I have a parent directory with 3 subdirectories inside (for example). Each subdirectory has a single file in it.

Let’s say I run rclone copy on that parent directory.

Then let’s say while that copy is ongoing that a new subdirectory is added to the parent directory. Now I know the original rclone copy does not pickup on the new directory. My question is, if I run the rclone copy again on the same parent directory, how can I make sure the files in the parent directory are not uploaded twice?

Thanks guys!

2

You don’t.

Because the first copy could still be copying the files and you’ve executed another instance of rclone so it too will attempt to upload the same files.

You could get around this with some bash scripting to execute the rclone copy in the subdir and then using bash to figure out where the file is located and place it in the same parent on the remote you’re copying to.

Another option would be to run the rclone command inside a script and have the script check to see if it’s running. If it’s already running it quits.

It may help if we know what you’re trying to do and why. There are likely to be other options.

3

It may help if we know what you’re trying to do and why. There are likely to be other options.

Well, I have a system that downloads files into my parent directory throughout the day. Every time a new file is added to the parent directory a rclone move is kicked off to move the file to the cloud drive. The problem is that this technique has potential to run multiple instances of rclone move simultaneously and I can’t tell the rclone command to target a specific file or folder because files added to this directory have an encrypted name that gets generated when they’re dropped in the parent directory.

So there is no way to have rclone add a hidden file next to files it’s already uploading so that it knows not to upload that file twice?

4

There isn’t at the moment. I’d been thinking about using OS facilities to see if a file was open which are difficult and not cross platform. However a temporary file alongside the file being uploaded is a good idea too.

There are other issues which could help like this one: https://github.com/ncw/rclone/issues/1119 - so rclone could loop after doing the first transfer and do some more…