Copying same file on multiple targets

Help and Support
1

Rclone v1.51.1
OS: Debian 10 x86 64bit
remote: Google Drive

I am trying to see if it's possible to upload a file to two directories without uploading the same file twice.

  • in the same remote. Copying it from the remote somewhere else in the remote counts as GDrive internal BW (at least from my tests). Is there some way of doing this automatically with rclone after the file has been copied on the first target? So copy it to remote: and afterwards copy it to remote:2nd
    -- If this is possible I could exclude B from appearing when the remote is mounted with --exclude "/2nd/**" correct?

  • different remote same google account, SA etc - I believe this is not possible.

  • using cp on a cached mount appears to re-upload the file

Thanks

2

You can use rclone copy to do a server side copy between accounts. You'll probably need the --drive-server-side-across-configs flag. You can't do this via a mount.

3

That is perfect. I take it there is no alternative (when automated) other than when inotify is triggered:

rclone copy file1 remote:
wait 5min
rclone copy --drive-server-side-across-config remote:file1 remote2:
4

--drive-server-side-across-config works perfectly if it's used on unencrypted remotes.
I am trying to use it on two encrypted mounts though. I created the second crypt remote by copying the first as to keep the encryption settings and passwords the same. The drive remotes are identical just use different SA (on which --drive-server-side-across-config works fine).

It always just downloads from the 1st and uploads it to the 2nd. What am I missing?

Solution to this: https://forum.rclone.org/t/copying-same-file-on-multiple-targets/14405/7

5

You aren't missing anything.

You can't use server side copies with encrypted remotes as rclone does the encryption so it has to download it and upload it.

6

if both remotes use the same encryption settings, then you can use --drive-server-side-across-config.
you can copy the encrypted files from the remote to remote.
let's say

  1. source remote is remote1:encbucket
  2. you have created an encrypted remote named secret1: on remote1:encbucket
  3. the dest remote is remote2:encbucket
  4. you have created an encrypted remote named secret2: on remote2:encbucket

if you rclone copy secret1: secret2:, then rclone will have to

  1. download the file from secret1:
  2. decrypt the file based on secret1: encryption settings
  3. encrypt the file based on secret2: encryption settings
  4. upload file to secret2:
    to be clear, rclone will not have to download the entire file, instead rclone will download a chunk at time, process that chunk and upload the chunk.

now if the encryption settings are the same for secret1: and secret2: then
rclone copy remote1:encbucket remote2:encbucket --drive-server-side-across-config
will not download the files to local computer.

does that make sense?

7

He could copy them at the unencrypted layer though. He could script a way to pull the lower level file name and copy. Assuming he wants the same nonce on both. I do all my syncs at the lower level to me backup drives.

8

I understand that, but if the two encrypted remotes have the same passwords, then it means the encryption is the same, which is valid since by copying the crypt remote to a new one and using it to ls I can see the contents.

@asdffdsa I am doing exactly that. I copied secret1 to secret2

9

That means you'd use the non encrypted remotes to copy as @calisro suggested. Having the keys the same doesn't change that you can't server side copy encrypted remotes.

10

To be more specific can't server side different encrypted remotes. If I do rclone copy --drive-server-side-across-config crypt:haa crypt:ho
it does a server side of haa to ho/haa

11

rclone copy secret1: to secret2: will force rclone to download and upload files.

but if you just copy the already encrypted files from the folders that secret1: and secret2: are based on
rclone copy remote1:encbucket remote2:encbucket --drive-server-side-across-config
then nothing needs to be downloaded and nothing needs to be decrypted and encrypted and uploaded

12

Thanks, I have understood the method of server side copying using the non encrypted remotes.

@Animosity022

I might be trying to make a lot of assumptions but please hear me out:

  • Server side copy encrypted files using the same crypt remote works fine
  • If I copy that remote and just change it's name, it's basically 100% the same remote

therefore (assumption) it should still work? Can it be that since normally almost all the encrypted remotes will have been setup with different passwords --> encryption used the default action is to fail without any check? Or doing said check would be difficult (outside comparing the password values in the config for the different mounts).
If that is true would it be possible to have a flag that attempts to copy them without trying to decrypt/encrypt with the users responsibility?

13

Why would you want to build additional code/logic for functionality that already exists?

You can do what you are asking by using the non encrypted remotes. Not sure the time/value add for adding that to the encrypted remote sections. You can always add a feature request on GitHub and it can be worked on or you can add it yourself and do a pull request if you find it valuable as well.

14

Honestly? Cause I don't really like the idea of having to call a script to identify the encrypted path from them decrypted ones.

But I respect the fact this is a niche request, there is a viable alternative and the above would require time from the devs, so I 'll pick calisro's answer as the solution and think whether this really merits a github request (even if it's a low urgency one)

15

Yep. Value is a funny thing as it might be valuable to quite a number of people, but not to me personally :slight_smile:

That's why I say to just put a request in and go from there.