Duplicate folders when copying to crypt inside Google Drive for school

What is the problem you are having with rclone?

Duplicate folders when copying to crypt inside Google Drive (for school)

What is your rclone version (output from rclone version)

rclone v1.50.2
- os/arch: linux/amd64
- go version: go1.13.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux version 5.4.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 9.2.1 20200
203 (Debian 9.2.1-28)) #1 SMP Debian 5.4.19-1 (2020-02-13)

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy ~/MyPrivateFolder SchoolDrive-crypt:/crypt

I also tried the following:

rclone copy ~/MyPrivateFolder SchoolDrive-crypt:/crypt
rclone copy ~/MyPrivateFolder SchoolDrive-crypt:/
rclone copy ~/MyPrivateFolder SchoolDrive-crypt:/mybackup
rclone copy ~/MyPrivateFolder SchoolDrive:/mybackup

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

~> rclone -vv copy /tmp/auto/ SchoolDrive-crypt:/crypt
2020/02/29 17:28:35 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "-vv" "copy" "/tmp/auto/" "SchoolDrive-crypt:/crypt"]
2020/02/29 17:28:35 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 17:28:35 DEBUG : SchoolDrive: Loaded invalid token from config file - ignoring
2020/02/29 17:28:35 DEBUG : SchoolDrive: Saved new token in config file
2020/02/29 17:28:36 INFO  : Encrypted drive 'SchoolDrive-crypt:/crypt': Waiting for checks to finish
2020/02/29 17:28:36 INFO  : Encrypted drive 'SchoolDrive-crypt:/crypt': Waiting for transfers to finish
2020/02/29 17:28:39 INFO  : somefile.el: Copied (new)
2020/02/29 17:28:39 INFO  :
Transferred:           152 / 152 Bytes, 100%, 58 Bytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:        2.6s

2020/02/29 17:28:39 DEBUG : 9 go routines active
2020/02/29 17:28:39 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "-vv" "copy" "/tmp/auto/" "SchoolDrive-crypt:/crypt"]

All of the following fail with essentially the same debug outputs. When I check SchoolDrive (a google drive at a school domain) via its web interface, I found there were multiple folders mybackup.

rclone -vv lsd SchoolDrive-crypt:/
rclone -vv lsd SchoolDrive-crypt:/crypt # or this command
rclone -vv lsd SchoolDrive-crypt: # or this command
rclone -vv lsd SchoolDrive:/mybackup # or this command
rclone -vv lsd SchoolDrive:mybackup # or this command

Output of the first one above

2020/02/29 17:37:08 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "-vv" "lsd" "SchoolDrive-crypt
:/crypt"]
2020/02/29 17:37:08 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 17:37:09 ERROR : : error listing: directory not found
2020/02/29 17:37:09 Failed to lsd with 2 errors: last error was: directory not found

rclone config dump

~> rclone config dump
{
    "SchoolDrive": {
        "acknowledge_abuse": "true",
        "client_id": "xxxxx.apps.googleusercontent.com",
        "client_secret": "xxxx",
        "root_folder_id": "xxxxx",
        "scope": "drive",
        "shared_with_me": "true",
        "token": "{\"access_token\":\"xxxx\",\"token_type\":\"Bearer\",\"refresh_token\":\"xxxx\",\"expiry\":\"2020-02-29T18:28:34.778074938-05:00\"}",
        "type": "drive"
    },
    "SchoolDrive-crypt": {
        "directory_name_encryption": "true",
        "filename_encryption": "standard",
        "password": "xxxx",
        "password2": "xxxx",
        "remote": "SchoolDrive:/mybackup",
        "type": "crypt"
    }
}
  1. please udpate to the latest version of rclone, v1.51.0
  2. test again

Unless there is a very strong reason (some bugfix relevant to my issue is included in a newer version), I'd rather stick to installing packages from the Debian repositories. I am in Debian testing, so I'd hope that its packages are relatively up-to-date.

Is there a single binary provided that I can run without installing it into the system?

https://rclone.org/install/

Wow, I read every text in https://rclone.org/downloads/ carefully; I wish the two pages were combined so users like me wouldn't asked questions like that and wasted your time... Sorry!

I thought, as a go program, it should come with a single binary...

it does come as a single binary.

Shall I redo the config or just copy and lsd?

no need redo the config.
so just copy and lsd.
perhaps try ls instead of lsd

the command you ran, it seems it only copied one file and no folders
rclone -vv copy /tmp/auto/ SchoolDrive-crypt:/crypt

I was using SchoolDrive-scrypt:/crypt instead of just SchoolDrive-scrypt:/crypt because a) the crypt config page [1] suggested specifying paths, b) I wanted to push different local folders into different encrypted remote folders.
[1] https://rclone.org/crypt/

$> ./rclone -vv copy /tmp/auto/ SchoolDrive-crypt:/crypt1
2020/02/29 18:00:32 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "copy" "/tmp/auto/" "SchoolDrive-crypt:/crypt1"]
2020/02/29 18:00:32 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 18:00:34 INFO  : Encrypted drive 'SchoolDrive-crypt:/crypt1': Waiting for checks to finish
2020/02/29 18:00:34 INFO  : Encrypted drive 'SchoolDrive-crypt:/crypt1': Waiting for transfers to finish
2020/02/29 18:00:36 INFO  : test.txt: Copied (new)
2020/02/29 18:00:36 INFO  :
Transferred:           152 / 152 Bytes, 100%, 58 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.5s

2020/02/29 18:00:36 DEBUG : 7 go routines active
2020/02/29 18:00:36 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "copy" "/tmp/auto/" "SchoolDrive-crypt:/crypt1"]

$> ./rclone -vv lsd SchoolDrive-crypt:/crypt1
2020/02/29 18:00:45 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "lsd" "SchoolDrive-crypt:/crypt1"]
2020/02/29 18:00:45 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 18:00:46 ERROR : : error listing: directory not found
2020/02/29 18:00:46 Failed to lsd with 2 errors: last error was: directory not found

$> ./rclone -vv ls SchoolDrive-crypt:/crypt1
2020/02/29 18:01:34 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "ls" "SchoolDrive-crypt:/crypt1"]
2020/02/29 18:01:34 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 18:01:35 Failed to ls: directory not found

try ls SchoolDrive: and lsd SchoolDrive:

Here, copying a whole directory over to SchoolDrive: then diong ls and lsd:

hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64 [3]> ./rclone -vv copy /tmp/rclone-v1.51.0-linux-amd64 SchoolDrive-crypt:
2020/02/29 18:07:45 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "copy" "/tmp/rclone-v1.51.0-linux-amd64" "SchoolDrive-crypt:"]
2020/02/29 18:07:45 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 18:07:46 INFO  : Encrypted drive 'SchoolDrive-crypt:': Waiting for checks to finish
2020/02/29 18:07:46 INFO  : Encrypted drive 'SchoolDrive-crypt:': Waiting for transfers to finish
2020/02/29 18:07:48 DEBUG : mtit8rmhu00ftvlbqq1ompqsr8: Sending chunk 0 length 8388608
2020/02/29 18:07:48 INFO  : git-log.txt: Copied (new)
2020/02/29 18:07:49 INFO  : README.html: Copied (new)
2020/02/29 18:07:49 INFO  : README.txt: Copied (new)
2020/02/29 18:07:51 INFO  : rclone.1: Copied (new)
2020/02/29 18:07:51 DEBUG : mtit8rmhu00ftvlbqq1ompqsr8: Sending chunk 8388608 length 8388608
2020/02/29 18:07:55 DEBUG : mtit8rmhu00ftvlbqq1ompqsr8: Sending chunk 16777216 length 8388608
2020/02/29 18:07:58 DEBUG : mtit8rmhu00ftvlbqq1ompqsr8: Sending chunk 25165824 length 8388608
2020/02/29 18:08:01 DEBUG : mtit8rmhu00ftvlbqq1ompqsr8: Sending chunk 33554432 length 782560
2020/02/29 18:08:02 INFO  : rclone: Copied (new)
2020/02/29 18:08:02 INFO  : 
Transferred:       35.441M / 35.441 MBytes, 100%, 2.321 MBytes/s, ETA 0s
Transferred:            5 / 5, 100%
Elapsed time:        15.2s

2020/02/29 18:08:02 DEBUG : 13 go routines active
2020/02/29 18:08:02 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "copy" "/tmp/rclone-v1.51.0-linux-amd64" "SchoolDrive-crypt:"]
hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64> ./rclone -vv ls /tmp/rclone-v1.51.0-linux-amd64 SchoolDrive-crypt:
2020/02/29 18:08:09 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "ls" "/tmp/rclone-v1.51.0-linux-amd64" "SchoolDrive-crypt:"]
Usage:
  rclone ls remote:path [flags]

Flags:
  -h, --help   help for ls

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command ls needs 1 arguments maximum: you provided 2 non flag arguments: ["/tmp/rclone-v1.51.0-linux-amd64" "SchoolDrive-crypt:"]


hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64 [1]> ./rclone -vv ls  SchoolDrive-crypt:
2020/02/29 18:08:16 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "ls" "SchoolDrive-crypt:"]
2020/02/29 18:08:16 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 18:08:17 Failed to ls: directory not found


hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64 [3]> ./rclone -vv lsd SchoolDrive-crypt:
2020/02/29 18:08:22 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "lsd" "SchoolDrive-crypt:"]
2020/02/29 18:08:22 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 18:08:23 ERROR : : error listing: directory not found
2020/02/29 18:08:23 Failed to lsd with 2 errors: last error was: directory not found

this is not a valid command
./rclone -vv ls /tmp/rclone-v1.51.0-linux-amd64 SchoolDrive-crypt:
ls should only one remote
try
./rclone -vv ls SchoolDrive-crypt:

as per the error in the log
Command ls needs 1 arguments maximum: you provided 2

Yeah I fixed that as shown below that faulty command... Should've removed embarrassing commands before posting along with correct ones...

This is what the web interface shows now:

Still the same after editing the config for the SchoolDrive-crypt to point it to a clean and empty new folder SchoolDrive:/mybackup1.

ok. i was helping with the basics, but i see that you need a more help than i can provide.
might be something with the config that is specific to gdrive, which i do not use.

luckily, i know someone you can help.
calling @thestigma

Thx! I've been living in terminal for 10y, and wouldn't want to waste dev time without going thru the docs myself and trying the basics. Still appreciate the help

perhaps this can help


"The --drive-shared-with-me hides the non shared with me stuff so rclone doesn’t see the existing folder and creates a duplicate"

But none of the folders used here are shared between anyone, though.

well, i could be wrong as i do not use gdrive.
but

  1. your folders are not shared.
  2. if i understand the flag --drive-shared-with-me seems to hides the non shared

so i would update the config file to remove --drive-shared-with-me and test again.

https://rclone.org/drive/#drive-shared-with-me
"Only show files that are shared with me"
"Instructs rclone to operate on your “Shared with me” folder (where Google Drive lets you access the files and folders others have shared with you)."

I switched shared_with_me in ~/.config/rclone/rclone.conf to false, re-ran the config for the drive and crypt drive ones, and pointed the crypt to a new folder. Now everything is working as expected. Thanks much! No idea how shared-with-me = true lurked its way into my config...

hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64 [130]> ./rclone copy /tmp/rclone-v1.51.0-linux-amd64 SchoolDrive-crypt:

hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64> ./rclone -vv ls SchoolDrive-crypt:
2020/02/29 19:16:24 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "ls" "SchoolDrive-crypt:"]
2020/02/29 19:16:24 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
 34328576 rclone
        0 git-log.txt
   834407 README.txt
  1046996 README.html
   943372 rclone.1
2020/02/29 19:16:26 DEBUG : 7 go routines active
2020/02/29 19:16:26 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "ls" "SchoolDrive-crypt:"]

hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64> ./rclone -vv lsd SchoolDrive-crypt:
2020/02/29 19:16:30 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "lsd" "SchoolDrive-crypt:"]
2020/02/29 19:16:30 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 19:16:32 DEBUG : 7 go routines active
2020/02/29 19:16:32 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "lsd" "SchoolDrive-crypt:"]

hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64> ./rclone -vv lsd SchoolDrive:/backup
2020/02/29 19:16:45 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "lsd" "SchoolDrive:/backup"]
2020/02/29 19:16:45 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
2020/02/29 19:16:46 DEBUG : 5 go routines active
2020/02/29 19:16:46 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "lsd" "SchoolDrive:/backup"]

hyiltiz@Debian /t/rclone-v1.51.0-linux-amd64> ./rclone -vv ls SchoolDrive:/backup
2020/02/29 19:16:50 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "ls" "SchoolDrive:/backup"]
2020/02/29 19:16:50 DEBUG : Using config file from "/home/hyiltiz/.config/rclone/rclone.conf"
 34336992 mtit8rmhu00ftvlbqq1ompqsr8
       32 1p94qhmrjt4j12obc1fdihpt8o
   834647 fokpg275rnbriue5ugt6rq2vcs
  1047284 g70prn2io3otimjtpnvg4d5pe8
   943644 qh4actq0mneruqa7t3icf2qfgk
2020/02/29 19:16:52 DEBUG : 5 go routines active
2020/02/29 19:16:52 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "ls" "SchoolDrive:/backup"]

glad to help.

live long and prosper!