Server side copy doesn't work with union drive

Suspected Bug
1

What is the problem you are having with rclone?

Rclone doesn't seem to be doing server side copy when copying or moving from google drive to a unioned google team drive. All the google drive and team drive are on the same account.

What is your rclone version (output from rclone version)

rclone v1.54.0
- os/arch: linux/amd64
- go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18.04.5 LTS, x86_64

Which cloud storage system are you using? (eg Google Drive)

Google Drive
Google Team Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move gd3:enc gd3-union:enc -P

The rclone config contents with secrets removed.

[gd3]
type = drive
scope = drive
server_side_across_configs = true
token = xxx

[gd3-1]
type = drive
scope = drive
server_side_across_configs = true
token = xxx
team_drive = xxx
root_folder_id =

[gd3-2]
type = drive
scope = drive
server_side_across_configs = true
token = xxx
team_drive = xxx
root_folder_id =

[gd3-union]
type = union
upstreams = gd3-1: gd3-2:
server_side_across_configs = true
create_policy = rand

A log from the command with the -vv flag

kelvin@ubuntu:~/# rclone move gd3:enc gd3-union:enc -P -vv
2021/02/10 16:06:07 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "move" "gd3:enc" "gd3-union:enc" "-P" "-vv"]
2021/02/10 16:06:07 DEBUG : Using config file from "/home/kelvin/.config/rclone/rclone.conf"
2021/02/10 16:06:07 DEBUG : Creating backend with remote "gd3:enc"
2021/02/10 16:06:07 DEBUG : Google drive root 'enc': root_folder_id = "xxx" - save this in the config to speed up startup
2021/02/10 16:06:08 DEBUG : Creating backend with remote "gd3-union:enc"
2021/02/10 16:06:08 DEBUG : Creating backend with remote "gd3-1:"
2021/02/10 16:06:08 DEBUG : Creating backend with remote "gd3-2:"
2021/02/10 16:06:08 DEBUG : Creating backend with remote "gd3-2:/enc"
2021/02/10 16:06:08 DEBUG : fs cache: renaming cache item "gd3-2:/enc" to be canonical "gd3-2:enc"
2021/02/10 16:06:08 DEBUG : fs cache: renaming cache item "gd3-1:/enc" to be canonical "gd3-1:enc"
2021/02/10 16:06:08 DEBUG : union root 'enc': actionPolicy = *policy.EpAll, createPolicy = *policy.Rand, searchPolicy = *policy.FF
2021-02-10 16:06:10 DEBUG : ibcbisbqveanqdui79sq55sm50/04vacjj390pageneof06l07ke4/3qsd4nfusrj6a3kspa6eud1fec/oqgglllnv4eoqgpimthj12sfbjm3c5kofvq6ascppscc0c419po0: Sending chunk 0 length 8388608
2021-02-10 16:06:11 DEBUG : ibcbisbqveanqdui79sq55sm50/20qf7h28e0jg5bgn0uu1bngq4o/4ku35896o487ue8be1jmu5ubis/4eq07k2gj6g9io8sk0h9rserfuopb0s984grl1slqipkp68tfrqg: MD5 = a583b72ce84dc981980e8483e1490829 OK
2021-02-10 16:06:11 INFO  : ibcbisbqveanqdui79sq55sm50/20qf7h28e0jg5bgn0uu1bngq4o/4ku35896o487ue8be1jmu5ubis/4eq07k2gj6g9io8sk0h9rserfuopb0s984grl1slqipkp68tfrqg: Copied (new)
2021-02-10 16:06:12 INFO  : ibcbisbqveanqdui79sq55sm50/20qf7h28e0jg5bgn0uu1bngq4o/4ku35896o487ue8be1jmu5ubis/4eq07k2gj6g9io8sk0h9rserfuopb0s984grl1slqipkp68tfrqg: Deleted
2021-02-10 16:06:13 DEBUG : ibcbisbqveanqdui79sq55sm50/04vacjj390pageneof06l07ke4/3qsd4nfusrj6a3kspa6eud1fec/oqgglllnv4eoqgpimthj12sfbjm3c5kofvq6ascppscc0c419po0: Sending chunk 16777216 length 8388608
2021-02-10 16:06:13 DEBUG : ibcbisbqveanqdui79sq55sm50/04vacjj390pageneof06l07ke4/3qsd4nfusrj6a3kspa6eud1fec/oqgglllnv4eoqgpimthj12sfbjm3c5kofvq6ascppscc0c419po0: Sending chunk 25165824 length 8388608
2021-02-10 16:06:14 DEBUG : ibcbisbqveanqdui79sq55sm50/04vacjj390pageneof06l07ke4/3qsd4nfusrj6a3kspa6eud1fec/oqgglllnv4eoqgpimthj12sfbjm3c5kofvq6ascppscc0c419po0: Sending chunk 33554432 length 8388608
2021-02-10 16:06:15 DEBUG : ibcbisbqveanqdui79sq55sm50/04vacjj390pageneof06l07ke4/3qsd4nfusrj6a3kspa6eud1fec/oqgglllnv4eoqgpimthj12sfbjm3c5kofvq6ascppscc0c419po0: Sending chunk 41943040 length 8388608
2021-02-10 16:06:16 DEBUG : ibcbisbqveanqdui79sq55sm50/04vacjj390pageneof06l07ke4/3qsd4nfusrj6a3kspa6eud1fec/oqgglllnv4eoqgpimthj12sfbjm3c5kofvq6ascppscc0c419po0: Sending chunk 50331648 length 8388608
2021-02-10 16:06:17 DEBUG : ibcbisbqveanqdui79sq55sm50/04vacjj390pageneof06l07ke4/3qsd4nfusrj6a3kspa6eud1fec/oqgglllnv4eoqgpimthj12sfbjm3c5kofvq6ascppscc0c419po0: Sending chunk 58720256 length 8388608
Transferred:       75.501M / 38.869 GBytes, 0%, 3.329 MBytes/s, ETA 3h18m53s
Checks:                 2 / 6, 33%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 290, 0%
Elapsed time:        51.6s
Checking:

Transferring:
 * ibcbisbqveanqdui79sq55…kofvq6ascppscc0c419po0: 28% /248.961M, 3.293M/s, 53s
 * ibcbisbqveanqdui79sq55…s9b16gahkski0ll7nmh7p0:100% /1.999M, 119.005k/s, 0s
 * ibcbisbqveanqdui79sq55…4mq3v5fnb9u7qjgcppnh5k:100% /360, 21/s, 0s
 * ibcbisbqveanqdui79sq55…v765vpickrrl88b90pldd0:  0% /3.385M, 0/s
2

I think to make this work, the union backend would need a server_side_across_configs option which it doesn't at the moment.

It is actually really complicated getting the union backend to do a server side copy - I can't think of an easy way to do it :frowning:

Can you achieve what you want a different way?

3

I was testing to see if it work and was hoping to spread out my backups from another google drive into multiple team drive. I guess I have to use my own bandwidth to do it then.

4

I see. Yes it would be nice if this works. Trying to work out how is hurting my head though :wink:

You could probably script moves to the underlying mounts.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.