Help needed to set up 2 G Drives on a new server and migrate data from 1st to 2nd

What is the problem you are having with rclone?

I'm not able to set up 2 remotes (Google Drive) and mount them both with encryption in order to transfer data from one to the other.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0

  • os/version: debian 11.7 (64 bit)
  • os/kernel: 5.10.0-20-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.5
  • go/linking: static
  • go/tags: none

Also running Quickbox Pro v3 script on the server

Which cloud storage system are you using? (eg Google Drive)

Old Google Drive that I will quit in 2 months and a new Google Shared Drive (non profit) in which I need to migrate data.

Current Situation :

  • Tried several options and reversed : create remote with new drive (let's call it gdrivenew) with encryption, then adding as a second remote the old gdrive (let's call it gdriveold) and also added as a 3rd step a "cryptold" in order to decrypt content from gdriveold (reusing encryption and salt passwords I had when first set up the gdriveold with encryption)
    Problem encountered : never able to browse directories of gdriveold in order to move data from it, always get an error directories cannot be found
  • Tried then to set up at rclone install the gdriveold using encryption, this time i can browse directories that are encrypted. Then adding a new remote for gdrivenew + "cryptnew" for encryption
    With this set up I am able to move data from gdriveold to gdrivenew using encryption. But cannot access these data from gdrivenew as remote is not mounted and if I try to mount it with rclone mount again I'm not seeing any data in the new directory where i mounted gdrivenew. So I tried to reinstall rclone and set up gdrivenew from isntall with encryption, gdrivenew is well mounted but once again I don't see any data I transferred from gdriveold previously.

Can anyone please guide me and point out what I surely did wrong ?

Thanks all for reading, and please let me know if more detail needed !

Cheers

SuissoNaBisso

post your rclone.config - remove all secrets before

and commands you use for mounting.

so here is the config file :
PS : gdrive and crypt are for the gdrivenew :slight_smile:

[gdrive]
type = drive
client_id = hidden
client_secret = hidden
scope = drive
team_drive = 0AL8J37IlDd0EUk9PVA
token = hidden
root_folder_id = 

[crypt]
type = crypt
directory_name_encryption = true
filename_encryption = standard
remote = gdrive:encrypt
password = hidden
password2 = hidden

[gdriveold]
type = drive
client_id = hidden
client_secret = hidden
scope = drive
token = hidden
team_drive = 

[cryptold]
type = crypt
remote = gdriveold:
password = hidden
password2 = hidden

and here is the kind of commande i try to move data from gdriveold to gdrive :

rclone move cryptold:/eBooks crypt:/Medias --bwlimit 20M
2023/07/16 18:18:32 ERROR : Encrypted drive 'cryptold:/eBooks': error reading source root directory: directory not found
2023/07/16 18:18:32 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2023/07/16 18:18:32 ERROR : Encrypted drive 'cryptold:/eBooks': error reading source root directory: directory not found
2023/07/16 18:18:32 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2023/07/16 18:18:32 ERROR : Encrypted drive 'cryptold:/eBooks': error reading source root directory: directory not found
2023/07/16 18:18:32 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2023/07/16 18:18:32 Failed to move: directory not found

But as said if I reverse the remotes set up, starting with gdriveold during rclone isntall and thenn adding the second, I can transfer the data from gdriveold to gdrivenew but the problem is I can't access these files after

Thanks !

What this command show?

rclone lsd cryptold:

nothing actually, no asnwers from rclone

what about:

rclone lsd gdriveold:
❯ rclone lsd gdriveold:
          -1 2019-12-30 12:31:46        -1 backup
          -1 2019-12-30 12:31:44        -1 encrypt

So also i just reconfigured the cryptold as follow now :

[cryptold]
type = crypt
remote = gdriveold:
password = hidden
password2 = hidden

No wwhen I run the move it seems to work ! but I don't see the moved files into gdrive, so wondering if I have to add the encrypt directory to crypt:encrypt maybe ?

Exactly. Good you spotted it yourself.

well just tried the following commands :

rclone move cryptold:/eBooks crypt:/encrypt --bwlimit 20M
rclone move cryptold:/eBooks crypt:encrypt --bwlimit 20M

No erros for both of them, but still I don't see the files appearing under gdrive...Feel like I'm close but still something that does not work as I want :smiley:

hahaha... you have to change config

and then

rclone copy cryptold:eBooks crypt:eBooks

Use copy not move - it is safer.

sorry actually i already did the change on config with :

[cryptold]
type = crypt
remote = gdriveold:encrypt
password = hidden
password2 = hidden

And just tried your command but still I don't see anything showing up in gdrive

because you probably moved some data e.g. Ebooks from cryptold: to somewhere random:) so it is gone from there

anyway do:

rclone lsd cryptold:

and

rclone lsd crypt:

also post what is your rclone.config now

so :

rclone lsd cryptold:
          -1 2019-12-30 12:31:57        -1 Anime
          -1 2020-01-03 09:07:15        -1 Docu
          -1 2019-12-30 12:31:48        -1 Movies
          -1 2019-12-30 12:31:54        -1 Music
          -1 2020-01-03 09:07:05        -1 Sport
          -1 2020-05-17 03:12:13        -1 Sport-UHD
          -1 2019-12-30 12:31:51        -1 TV
          -1 2020-02-12 03:12:13        -1 TV-UHD
          -1 2019-12-30 15:43:09        -1 UHD
          -1 2020-11-10 03:12:21        -1 Unpaid-users
          -1 2020-11-22 03:12:25        -1 eBooks

then

rclone lsd crypt:
          -1 2023-07-16 19:09:56        -1 .Trash-1000

and finally the config right now :

[gdrive]
type = drive
client_id = hidden
client_secret = hidden
scope = drive
team_drive = 0AL8J37IlDd0EUk9PVA
token = hidden
root_folder_id = 

[crypt]
type = crypt
directory_name_encryption = true
filename_encryption = standard
remote = gdrive:encrypt
password = hidden
password2 = hidden

[gdriveold]
type = drive
client_id = hidden
client_secret = hidden
scope = drive
token = hidden
team_drive = 

[cryptold]
type = crypt
remote = gdriveold:encrypt
password = hidden
password2 = hidden

reading cryptold is fine now ! but copying and accessing into crypt / gdrive seems to be missing something

OK

so now:

rclone copy cryptold:eBooks crypt:eBooks

and after:

rclone lsd crypt:

still gettin the same :

 rclone lsd crypt:
          -1 2023-07-16 19:09:56        -1 .Trash-1000

what does it show:

rclone lsf cryptold:eBooks

still what's inside eBooks as expected :

rclone lsf cryptold:eBooks
Dan Brown/
Guy-Roger Duvert/
J.K. Rowling/
Jojo Moyes/
Stephen King/

no - it is empty - you moved all files somewhere during your testing random commands. What is left are empty folders. Forget about ebooks for now. This is why you should not use move for testing if you care about your data:)

run:

rclone copy cryptold:Docu crypt:Docu

and then:

rclone lsd crypt:

Holy sh*t it worked :smiley:

I tried with another folder (unpaid-users) as it's less data and it just appeared in gdrive folder ! still uploading right now but I can see it through file explorer through x2go

thank you so much ! you really helped me a lot today !

your eBooks are probably in root folder of new gdrive - but encrypted - you can move this to :encrypt folder and all should be ok

in general use copy not move and only when happy with results delete old stuff

1 Like