Copying Data from Rclone cache drive to normal

What is the problem you are having with rclone?

I have setup a rclone cache drive and when i copy large files with rclone move to the mount, my memory maxes out
i am wondering if a non cached drive would make a difference?
So my question would be how do i copy my data from one mount to the other without it taking days to do?

What is your rclone version (output from rclone version)

rclone v1.52.2

  • os/arch: linux/amd64

  • go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu Linux 18.04

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/local/bin/rclone move ~/media/files/ ~/media_2/files/ --delete-empty-src-dirs
That is the command that maxes out the memory

Paste command here



#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

```[gdrive]
type = drive
scope = drive
token = {"access_token":"redacted"}
root_folder_id = redacted

[gcrypt]
type = crypt
remote = gcache:/crypt
filename_encryption = obfuscate
directory_name_encryption = true
password = redacted
password2 = redacted

[gcache]
type = cache
remote = gdrive:/gdrive
chunk_size = 10M
info_age = 1d
chunk_total_size = 1G ```

Paste config here

A log from the command with the -vv flag

Paste  log here

hello and welcome to the forum,

is there a specifc reason you are using cache backend?
https://rclone.org/cache/#status

you did not post your rclone mount command

I read somewhere it is better for performance,
i want to move away but how do i copy my files across?
my command is
/sbin/start-stop-daemon -S -u $USER -d $HOME --exec /usr/local/bin/rclone mount gcrypt: ~/media_2 -- --allow-other --cache-db-purge --buffer-size 0M --cache-tmp-upload-path ~/tmp/ --use-mmap --cache-tmp-wait-time 24h --dir-cache-time 72h --drive-chunk-size 32M --timeout 1h --vfs-cache-mode minimal &

you need to create new crypted remote that bypassed the cache and that points to your crypted gdrive folder
the new remote must use the same passwords and salts are the current crypted

i think the crypted files are at gdrive:/gdrive/crypt

do these commands, post the command and output
rclone lsd gdrive:
rclone lsd gdrive:/gdrive
rclone lsd gdrive:/gdrive/crypt

         -1 2020-07-07 15:36:50        -1 rclone


rclone lsd googled:/googled

2020/07/09 14:54:01 ERROR : : error listing: directory not found

2020/07/09 14:54:01 Failed to lsd with 2 errors: last error was: directory not found

rclone lsd gdrive:/gdrive/crypt
         -1 2020-07-07 12:54:56        -1 1.Fqypnqcfgf
         -1 2020-07-07 12:55:03        -1 115.cELyuI
         -1 2020-07-07 12:55:08        -1 129.ytwwjsyx
         -1 2020-07-07 12:16:42        -1 131.yjsvul
         -1 2020-07-07 12:55:07        -1 182.Bmux
         -1 2020-07-07 12:55:09        -1 222.qs pELTP
         -1 2020-07-07 12:55:02        -1 7.SqlA_bd_apwEA
         -1 2020-07-07 12:54:55        -1 83.MxlDvnwCjArnB
         -1 2020-07-07 12:55:01        -1 93.dBwL_fHOBxL```

What is the downside of using cache?

What is the upside of using cache?
https://rclone.org/cache/#status

i think i was correct, the root of the encypted folders is gdrive:/gdrive/crypt

so create a new remote, lets call it gcrypt2.
set the remote path to gdrive:/gdrive/crypt
make sure use the same extact settings / passwords and/or salt as you used for gcrypt:

then do a rclone lsd gcrypt2:

I did that and i get
rclone lsd googled:
-1 2020-04-06 19:42:27 -1 gdrive
-1 2020-07-07 15:36:50 -1 rclone

do
rclone listremotes
rclone config show googled: and redact any passwords and id

gcrypt:
gdrive:
googled:
googledc:```

[googled]
type = drive
client_id = 
client_secret =-
scope = drive
token = {"access_token":"

we discussed that you need to create a new crypted remote, using the same settings are gcrypt: but change the remote path to gdrive:/gdrive/crypt

i do not see it the new remote?

problem is i dont have the Salt i seem to have lost it :frowning:

about the salt, we can work around that.

is your config file password protected or plain text?

plain text so i can edit it etc

copy the remote gcrypt settings and paste it to the bottom of the config file.

then change the new remote you just pasted.
change the [gcrypt] to [gcrypt2]
change remote = gcache:/crypt to remote = gdrive:/gdrive/crypt
save the file and then do a
rclone lsd gcrypt2:

ok now we are seeing something

      -1 2020-07-07 12:54:55        -1 Documentaries
      -1 2020-07-07 12:54:56        -1 Downloaded
      -1 2020-07-07 12:55:01        -1 Kids_Movies
      -1 2020-07-07 12:55:02        -1 Kids_TV_Shows
      -1 2020-07-07 12:55:03        -1 Movies
      -1 2020-07-07 12:55:09        -1 TV Shows
      -1 2020-07-07 12:16:42        -1 rclone
      -1 2020-07-07 12:55:07        -1 temp
      -1 2020-07-07 12:55:08        -1 torrents

can you do a
rclone lsd gcrypt: and lets compare the two outputs

that gives lots of errors

2020/07/09 15:55:42 ERROR : /home/hd28/xxx/.cache/rclone/cache-backend/gcache.db: Error opening storage cache. Is there another rclone running on the same remote? failed to open a cache connection to "/home/hd28/xxx/.cache/rclone/cache-backend/gcache.db": timeout

2020/07/09 15:55:44 ERROR : /home/hd28/xxx/.cache/rclone/cache-backend/gcache.db: Error opening storage cache. Is there another rclone running on the same remote? failed to open a cache connection to "/home/hd28/xxx/.cache/rclone/cache-backend/gcache.db": timeout

2020/07/09 15:55:44 Failed to create file system for "gcrypt:": failed to make remote gcache:"/crypt" to wrap: failed to start cache db: failed to open a cache connection to "/home/hd28/xxx/.cache/rclone/cache-backend/gcache.db": timeout

when you post, you need to post both the command and the output, not just the output.

does this rclone lsd gcrypt2: give out the expected output

ok here are both
rclone lsd gcrypt2: does give my correct output

rclone lsd gcrypt2:
-1 2020-07-07 12:54:55 -1 Documentaries
-1 2020-07-07 12:54:56 -1 Downloaded
-1 2020-07-07 12:55:01 -1 Kids_Movies
-1 2020-07-07 12:55:02 -1 Kids_TV_Shows
-1 2020-07-07 12:55:03 -1 Movies
-1 2020-07-07 12:55:09 -1 TV Shows
-1 2020-07-07 12:16:42 -1 rclone
-1 2020-07-07 12:55:07 -1 temp
-1 2020-07-07 12:55:08 -1 torrents

rclone lsd gcrypt:
2020/07/09 15:58:58 ERROR : /home/hd28/xxx/.cache/rclone/cache-backend/gcache.db: Error opening storage cache. Is there another rclone running on the same remote? failed to open a cache connection to "/home/hd28/xxx/.cache/rclone/cache-backend/gcache.db": timeout
2020/07/09 15:59:00 ERROR : /home/hd28/xxx/.cache/rclone/cache-backend/gcache.db: Error opening storage cache. Is there another rclone running on the same remote? failed to open a cache connection to "/home/hd28/xxx/.cache/rclone/cache-backend/gcache.db": timeout
2020/07/09 15:59:00 Failed to create file system for "gcrypt:": failed to make remote gcache:"/crypt" to wrap: failed to start cache db: failed to open a cache connection to "/home/hd28/xxx/.cache/rclone/cache-backend/gcache.db": timeout