From googledrive to an union of two servers

Help and Support
1

What is the problem you are having with rclone?

Im copying from google drive to a union of two servers, but not matter the policy I choose it always download to the server with the most files and least free space

rclone v1.60.1

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-121-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google drive and two servers

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy Tvshows2: union: --drive-stop-on-upload-limit --progress --transfers=4 --drive-chunk-size 64M

The rclone config contents with secrets removed.

[union]
type = union
upstreams = /home/rodrigo/disks /home/rodrigo/disks/1 /home/rodrigo/disks/3 /home/rodrigo/disks/4 /home/rodrigo/disks/5 /home/rodrigo/disks/6 /home/rodrigo/disks/7 /home/rodrigo/disks/8 /home/rodrigo/disks/9 /home/rodrigo/het2 /home/rodrigo/disks/10
action_policy = eplus
create_policy = eplus
search_policy = eplus

[Tvshows]
type = drive
scope = drive
token = 
team_drive = 0AC
root_folder_id = 

[Tvshows2]
type = drive
scope = drive
token = {
team_drive = 0AM6iBOkwk
root_folder_id = 

[Tvshows3]
type = drive
scope = drive
token = {
team_drive = 0A9PVA
root_folder_id = 

[series]
type = union
upstreams = Tvshows: Tvshows2: Tvshows3:

[prueba]
type = union
upstreams = /home/rodrigo/het2/ /home/rodrigo/prueba

[het]
type = union
upstreams = /home/rodrigo/disks /home/rodrigo/disks/1 /home/rodrigo/disks/3 /home/rodrigo/disks/4 /home/rodrigo/disks/5 /home/rodrigo/disks/6 /home/rodrigo/disks/7 /home/rodrigo/disks/8 /home/rodrigo/disks/9 /home/rodrigo/disks/10
action_policy = eplus
create_policy = eplus
search_policy = eplus

A log from the command with the -vv flag

https://pastebin.com/dNTugu2y
2

I'm pretty sure team drives don't support rclone about so that's the issue as it can't tell free space.

There's nothing in that log minus rampant pacer errors though so a bit tough to tell for sure.

3

yeah, team drives shows like infinite space, but the problem is I want to copy from google drive to two server that reports the free space.

4

You have unions on unions though with members not showing the proper free space and you have rclone mounts I'd imagine so I only have a partial idea of what is setup.

Are those all what? fixed disks? mounted rclone?

5

Those are my actual disk mounted on those directories but not using rclone

6

Can you provide:

rclone about /home/rodrigo/disks
rclone about /home/rodrigo/disks/1
rclone about /home/rodrigo/disks/3
rclone about /home/rodrigo/disks/4
rclone about /home/rodrigo/disks/5
rclone about /home/rodrigo/disks/6
rclone about /home/rodrigo/disks/7
rclone about /home/rodrigo/disks/8
rclone about /home/rodrigo/disks/9
rclone about /home/rodrigo/disks/10

Can you copy then one just small test file that's not on Google Drive so we don't get flooded with your pacer messages.

felix@gemini:~$ rclone copy /etc/hosts union: -vv
2022/12/24 14:46:17 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/12/24 14:46:17 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "/etc/hosts" "union:" "-vv"]
2022/12/24 14:46:17 DEBUG : Creating backend with remote "/etc/hosts"
2022/12/24 14:46:17 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/12/24 14:46:17 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2022/12/24 14:46:17 DEBUG : Creating backend with remote "union:"
2022/12/24 14:46:17 DEBUG : Creating backend with remote "/data"
2022/12/24 14:46:17 DEBUG : Creating backend with remote "/seed"
2022/12/24 14:46:17 DEBUG : union root '': actionPolicy = *policy.EpLus, createPolicy = *policy.EpLus, searchPolicy = *policy.EpLus
2022/12/24 14:46:17 DEBUG : hosts: Need to transfer - File not found at Destination
2022/12/24 14:46:17 DEBUG : hosts: md5 = f0b78c96a144844b9258ea1f50f0e940 OK
2022/12/24 14:46:17 INFO  : hosts: Copied (new)
2022/12/24 14:46:17 INFO  :
Transferred:   	        545 B / 545 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.1s

2022/12/24 14:46:17 DEBUG : 3 go routines active

and show where that one file was copied after.

I had no issues with 2 local remotes and eplus.

For clarification, eplus is not least free space as it is:

	Of all the upstreams on which the relative path exists choose the one with the least used space.

Which is different from free space.

If free space is your requirement, you want:

eplfs (existing path, least free space)	Of all the upstreams on which the relative path exists choose the one with the least free space.
7

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks
Total: 14.437 TiB
Used: 18.141 MiB
Free: 13.709 TiB
rodrigo@Ubuntu-2004-focal-64-minimal:~$

rclone copy /etc/hosts union: -vv
2022/12/24 22:12:43 DEBUG : rclone: Version "v1.60.1" starting with parameters [ "rclone" "copy" "/etc/hosts" "union:" "-vv"]
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/etc/hosts"
2022/12/24 22:12:43 DEBUG : Using config file from "/home/rodrigo/.config/rclone /rclone.conf"
2022/12/24 22:12:43 DEBUG : fs cache: adding new entry for parent of "/etc/hosts ", "/etc"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "union:"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/10 "
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/7"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/9"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/3"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/4"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/het2"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/1"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/8"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/5"
2022/12/24 22:12:43 DEBUG : Creating backend with remote "/home/rodrigo/disks/6"
2022/12/24 22:12:43 DEBUG : union root '': actionPolicy = *policy.EpLus, createP olicy = *policy.EpLus, searchPolicy = *policy.EpLus
2022/12/24 22:12:43 DEBUG : hosts: Need to transfer - File not found at Destinat ion
2022/12/24 22:12:44 DEBUG : hosts: md5 = 9293703f16586325c8788478c5b85980 OK
2022/12/24 22:12:44 INFO : hosts: Copied (new)
2022/12/24 22:12:44 INFO :
Transferred: 315 B / 315 B, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.4s

It was copied to disk6

8

If you share the abouts from all the disks, it should explain why.

9

2022/12/24 22:12:44 DEBUG : 4 go routines active
rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/1
Total: 14.437 TiB
Used: 196.268 GiB
Free: 13.518 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/3
Total: 14.437 TiB
Used: 1.182 TiB
Free: 12.527 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/4
Total: 14.437 TiB
Used: 253.914 GiB
Free: 13.461 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/5
Total: 14.437 TiB
Used: 1.643 TiB
Free: 12.067 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/6
Total: 14.437 TiB
Used: 18.098 MiB
Free: 13.709 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/7
Total: 14.437 TiB
Used: 18.094 MiB
Free: 13.709 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/8
Total: 14.437 TiB
Used: 18.094 MiB
Free: 13.709 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/9
Total: 14.437 TiB
Used: 18.094 MiB
Free: 13.709 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/disks/10
Total: 14.437 TiB
Used: 18.094 MiB
Free: 13.709 TiB

rodrigo@Ubuntu-2004-focal-64-minimal:~$ rclone about /home/rodrigo/het2
Total: 115.758 TiB
Used: 114.297 TiB
Free: 1.461 TiB

10

I tried all the policies and its always the same, picks the server with less free space, I tried it to run on both servers and its the same

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.