Detected overridden config | Better way to upload to crypt?

What is the problem you are having with rclone?

Detected overridden config

What is your rclone version (output from rclone version)

rclone v1.58.0-beta.5885.4c93378f0
- os/version: Microsoft Windows 11 Pro for Workstations 2009 (64 bit)
- os/kernel: 10.0.22000.318 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.3
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive;
TeamDrive > SelfDrive (server side)
SelfDrive > Crypt (manual upload due to crypt's nature)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move GSD:/ GMC:/ -L -P --ignore-case-sync --drive-server-side-across-configs --drive-impersonate email@domain.com -vv --log-file "C:\Users\PC\.Drive\rclone\Log Files\X-Drive\DriveMerge-Directories.log"

The rclone config contents with secrets removed.

[GMD] # Root
type = drive
client_id = 
client_secret = 
scope = drive
token = 
acknowledge_abuse = true
server_side_across_configs = true
skip_shortcuts = false

[GMC] # Used for a directory to be `server_side_across_configs` moved to GMDmedia later on.
type = drive
client_id = 
client_secret = 
scope = drive
root_folder_id = 
token = 
acknowledge_abuse = true
server_side_across_configs = true
skip_shortcuts = false

[GSD] # Used as a Cache Drive for cloning directories (team drive supported only.)
type = drive
client_id = 
client_secret = 
scope = drive
root_folder_id = 
token = 
acknowledge_abuse = true
server_side_across_configs = true
skip_shortcuts = false
team_drive = 0AKi5-xxxxxxxxxxx

[GMDmain] # Main Files
type = drive
client_id = 
client_secret = 
scope = drive
root_folder_id = 
token = 
acknowledge_abuse = true
server_side_across_configs = true
skip_shortcuts = false

[GMDbackups] # Backup Crypt
type = crypt
remote = GMD:\\Backups\
filename_encryption = standard
directory_name_encryption = false
password = 
password2 = 
acknowledge_abuse = true
server_side_across_configs = true
skip_shortcuts = false

[GMDmedia] # Media/Plex Crypt
type = crypt
remote = GMD:\Media
filename_encryption = standard
directory_name_encryption = false
password = 
password2 = 
acknowledge_abuse = true
server_side_across_configs = true
skip_shortcuts = false

A log from the command with the -vv flag

2021/11/13 15:53:58 DEBUG : rclone: Version "v1.58.0-beta.5885.4c93378f0" starting with parameters ["rclone" "move" "GSD:/" "GMC:/" "-L" "-P" "--ignore-case-sync" "--drive-server-side-across-configs" "--drive-impersonate" "email@domain.com" "-vv" "--log-file" "C:\\Users\\PC\\.Drive\\rclone\\Log Files\\X-Drive\\DriveMerge-Directories.log"]
2021/11/13 15:53:58 DEBUG : Creating backend with remote "GSD:/"
2021/11/13 15:53:58 DEBUG : Using config file from "C:\\Users\\PC\\.Drive\\rclone\\rclone.conf"
2021/11/13 15:53:58 DEBUG : GSD: detected overridden config - adding "{efpZV}" suffix to name
2021/11/13 15:53:58 DEBUG : fs cache: renaming cache item "GSD:/" to be canonical "GSD{efpZV}:"
2021/11/13 15:53:58 DEBUG : Creating backend with remote "GMC:/"
2021/11/13 15:53:58 DEBUG : GMC: detected overridden config - adding "{efpZV}" suffix to name
2021/11/13 15:53:58 DEBUG : fs cache: renaming cache item "GMC:/" to be canonical "GMC{efpZV}:"
2021/11/13 15:54:01 DEBUG : Google drive root '': Waiting for checks to finish
2021/11/13 15:54:01 DEBUG : Google drive root '': Waiting for transfers to finish
2021/11/13 15:54:01 INFO  : There was nothing to transfer
2021/11/13 15:54:01 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         3.6s

2021/11/13 15:54:01 DEBUG : 28 go routines active

hi,

that is a normal debug log message.
it happens when a remote from a config file is overridden by a command line parameter(s).

I see, could It be due to the duplication of server_side_across_configs or lack of acknowledge_abuse?

Glad that it is normal though. only other questions I had was..

  1. I was looking at the log and ive noticed in terminal also that the progress doesn't update when doing serverside moves, it does for copies though.
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 6 / 6, 100%
Renamed:                6
Elapsed time:        11.0s
  1. Is there a better way to go about this besides server side copying to 'My Drive' then moving them to the crypt through manual reuploading?
    I use my own 'shared-drive' with clonebot (gclone) to auto use Service Accounts into the 'shared-drive' as a sort of temp drive due to them not being able to use root folder id's for security reasons.
    As most of the media I get now of days is already from a Google Drive Share... (minus nzb / torrents that are auto moved to crypt through radarr/sonarr.)
  1. that is how rclone works.
  2. i know nothing about gclone and the other software.
    rclone itself has to crypt the files.
    you can use a cheap/free google virtual machine and run rclone on that.

other then that, perhaps @Animosity022 might have a suggestion.

gclone is only used for cloning share folders (serverside) to a teamdrive. as I don't have to configure much of anything Just put in the link and click my directory from the dropdown in the bot. I just self host the bot myself.

Media Cache is the team/sharedrive name Couldn't really think of anything else due to the files staying on there for max an hour before moving them over with the script in the OP.
Clone to SharedDrive

(Because screenshots aren't really that great due to blurriness I included mostly for 'tree' clarity)

  1. Media Cache - share drive (temp Location)
  2. Media Cache - My Drive (working folder)
  3. Media Library - My Drive (crypt container/folder for plex)

I feel like I'll need more ram as remoting into the desktop environment (even with a 'minimal' Linux desktop install.) sudo apt-get install --no-install-recommends ubuntu-desktop as its taking 3-5 mins to open firefox, and around another 5-10 mins to load a webpage in the vm.
And when I try to ssh back into it after loading up the DE and closing the connection takes a few minutes to connect after putting in the key. So its defiantly being bottlenecked from just idling.

Reason for using the Desktop Enviorment is to get the config files over to the ubuntu vm, sign into google drive etc. So I don't have to upload it to paste bin etc.

sorry, not sure the purpose of the linux vm?

note: you can copy a rclone config file from one machine to a second machine.
that second machine does not need:

  • a desktop
  • a web browser
  • to sign in to google drive.

i run rclone on pi zero with just 512MB and i know of a fellow rcloner running rclone in just 128MB.

yea i figured it out with nano... forgot it exists. reason for linux vm is its on the google cloud platform. I moved over the rclone config with nano however.
i put in a 2gb swap file just incase though.

sure, but you can just copy the config file as is, no need for nano over ssh/sftp

true but ssh is alot easier, only issue now is it can't find my crypt remotes even though the config is in correctly:

shadows@shadowdrivevm:~$ rclone config
Current remotes:

Name                 Type
====                 ====
GMC                  drive
GMD                  drive
GMDbackups           crypt
GMDmain              drive
GMDmedia             crypt
GSD                  drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
shadows@shadowdrivevm:~$ rclone lsd GMC:
          -1 2021-11-13 21:37:23        -1 A-Library
          -1 2021-11-13 21:37:26        -1 K-Library
          -1 2021-11-13 21:37:20        -1 S-Library
shadows@shadowdrivevm:~$ rclone lsd GMD:
          -1 2021-08-27 14:35:45        -1 Backups
          -1 2021-08-27 14:19:08        -1 Main
          -1 2021-08-27 14:19:15        -1 Media
shadows@shadowdrivevm:~$ rclone lsd GMDbackups:
2021/11/15 22:33:49 ERROR : : error listing: directory not found
2021/11/15 22:33:49 Failed to lsd with 2 errors: last error was: directory not found
shadows@shadowdrivevm:~$

rclone version is same as the one i use on windows, and yes I did a token refresh through headless mode.

shadows@shadowdrivevm:~$ rclone version
rclone v1.58.0-beta.5885.4c93378f0
- os/version: ubuntu 21.04 (64 bit)
- os/kernel: 5.11.0-1022-gcp (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.3
- go/linking: static
- go/tags: none

Only thing I can find diferently is:

- go/linking: dynamic
- go/tags: cmount

on windows vs on linux:

- go/linking: static
- go/tags: none

yes, ssh is easier, that is what i meant to write.

is here a reason you are using beta software?
best to use latest stable, v1.57.0

to get a list of remotes, use rclone listremotes

as for GMDbackups, are you sure that work on windows?

and this does not look correct, tho rclone is very forgiving with syntax issues.
remote = GMD:\\Backups\ should be remote = GMD:Backups

and if a command does not work, need to post the debug output

hmmm I just tried on windows again and i somehow broke my config apparently... Give me one moment.
Never mind I forgot the : config works fine on windows.

C:\Users\PC>rclone listremotes
GMC:
GMD:
GMDbackups:
GMDmain:
GMDmedia:
GSD:

C:\Users\PC>rclone lsd GMDmedia:
          -1 2021-08-27 08:54:11        -1 Archives
          -1 2021-08-27 08:55:34        -1 Media Library

C:\Users\PC>rclone lsd GMDbackups:
          -1 2021-09-12 14:32:34        -1 OS
          -1 2021-09-12 14:32:12        -1 Software

C:\Users\PC>

so the config file was stuck in temp format... somehow the drive root id wasnt copied over.
Deleting the config and repasting fixed it. https://i.imgur.com/y5ul5JQ.png

shadows@shadowdrivevm:~$ nano ~/.config/rclone/rclone.conf
shadows@shadowdrivevm:~$ shadows@shadowdrivevm:~$
shadows@shadowdrivevm:~$ rclone lsd GMDmedia:
          -1 2021-08-27 14:54:11        -1 Archives
          -1 2021-08-27 14:55:34        -1 Media Library
shadows@shadowdrivevm:~$

I managed to answer my own question for "better way to upload to crypt remotes" using the
vm I can set up 2 additional remotes with 2(4) service accounts to allocate 1.47tb+0.03tb for trickle
so I can use the main account for backups while the 2 service accounts use --drive-impersonate

well... I will probably be self hosting my site on the vm.

yea your def right on the nose: Just don;t know why the speed is so slow however. as it should be using googles bandwith regardless of across configs or not due to it being hosted on there network right?
rclone ram usage via htop Very little ram usage. CPU usage could be a bit lower but its fine :smiley:

what speed, no idea what you mean?
please describe the problem in detail?

looking at the log its significltly slower to my host machine at around 45mbs

shadows@shadowdrivevm:~$ cat '.config/rclone/drive/DriveMerge.log'
2021/11/16 04:41:04 INFO  :
Transferred:      140.221 GiB / 140.221 GiB, 100%, 16.019 MiB/s, ETA 0s
Checks:                 2 / 2, 100%
Transferred:            6 / 6, 100%
Elapsed time:   1h53m37.9s

16.019 MiB/s
Elapsed time: 1h53m37.9s

  • do a speedtest on both machines.
  • what was the rclone command?
  • about the google vm, which machine type is it, what are the specs?

Windows:

C:\Users\PC>speedtest.exe
   Speedtest by Ookla
     Server: Syringa Networks - Boise, ID (id = 1456)
        ISP: CenturyLink
    Latency:     2.76 ms   (0.08 ms jitter)
   Download:   881.50 Mbps (data used: 891.4 MB)
     Upload:   937.52 Mbps (data used: 446.0 MB)
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/9017a721-49f8-4516-9614-1d806c56c4a7

VM:

shadows@shadowdrivevm:~$ speedtest-cli --share
Retrieving speedtest.net configuration...
Testing from Google Cloud (35.223.156.35)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Cox - Omaha (Omaha, NE) [12.44 km]: 31.556 ms
Testing download speed................................................................................
Download: 1712.50 Mbit/s
Testing upload speed......................................................................................................
Upload: 621.32 Mbit/s
Share results: http://www.speedtest.net/result/12338022733.png
shadows@shadowdrivevm:~$
rclone copy GMC:/ 'GMDmedia:/Media Library/' -L -P --ignore-case-sync --drive-impersonate xxxxx@shiroxxxxxxx.com --log-level INFO --log-file '.config/rclone/drive/DriveMerge.log'

server machine type - e2-micro
CPU platform - Intel Broadwell
ram - 1gb + 4gb swap
storage - HDD

shadows@shadowdrivevm:~$ neofetch
            .-/+oossssoo+/-.               shadows@shadowdrivevm
        `:+ssssssssssssssssss+:`           ---------------------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 21.04 x86_64
    .ossssssssssssssssssdMMMNysssso.       Host: Google Compute Engine
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.11.0-1022-gcp
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 1 day, 1 hour, 47 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 1254 (dpkg)
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.1.4
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: /dev/pts/2
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Intel Xeon (2) @ 2.2GHz
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU Usage: 0%
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Memory: 193MiB / 968MiB (19%)
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Disk (/): 7.5G / 29G (26%)
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

shadows@shadowdrivevm:~$
  • when you run these commands, you need to use a debug log and post the output.
    not worth to second guess after the fact.
  • if you run the exact same command using the exact same version of rclone, on two machine and get different results, the odds there is a difference in the machine specs.

most likely due to ram. 32gbs on host & 1gb on vm.
I'll grab a 10gb base file and try running it again with debug log and post the results in a bit however.