Revert standard filename encryption to simple obfuscated

#1

Hello,

as probably many users, i’m currently having a lot of problems with encrypted filename and cache.
Name of encrypted files are very long (too long) and i’m facing issues with hitting the chunks in cache because the full path of the chunk is too long (chunk not found)

I was wondering if there is a way to revert from standard filename encryption to “obfuscated” since this one does not “eat” so many chararcters.

My library is enterily crypted in standard and i don’t want to re-upload 8TB of files just to rename them.

Does anyone know a way to do this?

Thanks a lot

1 Like
#2

I just hit 63TB on my GD with standard and never have I seen a the issue you are talking about.

Can you share more details on the problem you are seeing with some logs?

#3

Some of my original folders are so deeper that every stage of the file path is encrypted with standard method, this method add some extra char to the encrypted path and you get a huge filename path.
This issue is reported here:

I finally rent a vps, set 2 config, one with standard encryption and the other with “obfuscate” settings and make a rlone sync between them.

However, i’m curious if someone have a script or something else to make the job without having to recopy whole library again…

Thanks anyway

#4

Do you have a specific reason for using cache?

#5

DOn’t want to redownload something that has already been downloaded 1 hour ago :slight_smile:
Speedup

#6

By the way, i’m still searching wich settings make the normal cache (not vfs) storing as longer as possible the chunck in the store?

#7

https://rclone.org/cache/#cache-info-age is the thing that determines how long something stays in the cache.

What’s the program causing the long file names that you are seeing?

#8

hello,

correct me if i’m wrong but the cache-info-age is only the structure, not the chunck isn’t it?
Is this not simply an “auto” recycling process determinded by the cache size?
i can not found anything on that…

#9

Sorry if I misunderstood the question.

chunks are kept based on total chunk size and it is a first in / first out method. You can’t do that based on time.

#10

although you can probably just remove the chunks based on time with a cronjob.

rclone delete /cache/dir --min-age 6h

or something along those lines.

#11

ok thank you for advice, my goal is to keep the chunk as long as possible.
So as i understodd, rclone will take care of recycling chunks with fifo

#12

Manually removing the chunks would lead to chunk not found message as the cache.db wouldn’t match the disk though. It’s preferable to configure the right total size.

#13

Should be okay. My understanding is the structure is kept in cache.db but not whether they exist on the file system. See here:

~/.cache/rclone/cache-backend/robgsc# rclone copy robgsc-cryptp:ct/kodi-headless.tar /tmp/

root@:~/.cache/rclone/cache-backend/robgsc# find . -ls
   662401      4 drwxr-xr-x   3 root     root         4096 May  9 13:18 .
   662716      4 drwxr-xr-x   3 root     root         4096 May  9 13:18 ./cloudp
   662772      4 drwxr-xr-x   3 root     root         4096 May  9 13:18 ./cloudp/o7aebe6dsvccihv3k25n0re7sk
   662773      4 drwxr-xr-x   2 root     root         4096 May  9 13:18 ./cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg
   662775  23796 -rwxr-xr-x   1 root     root     24366944 May  9 13:18 ./cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg/0

root@cloudc:~/.cache/rclone/cache-backend/robgsc# rm /tmp/kodi-headless.tar

root@cloudc:~/.cache/rclone/cache-backend/robgsc# rclone copy robgsc-cryptp:ct/kodi-headless.tar /tmp/ -vv

2019/05/09 13:19:32 DEBUG : rclone: Version "v1.47.0-015-gaba15f11-beta" starting with parameters ["rclone" "copy" "robgsc-cryptp:ct/kodi-headless.tar" "/tmp/" "-vv"]
2019/05/09 13:19:32 DEBUG : Using config file from "/root/.rclone.conf"
2019/05/09 13:19:33 DEBUG : robgsc: wrapped robgs:cloudp/o7aebe6dsvccihv3k25n0re7sk at root cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg
2019/05/09 13:19:33 INFO  : robgsc: Cache DB path: /root/.cache/rclone/cache-backend/robgsc.db
2019/05/09 13:19:33 INFO  : robgsc: Cache chunk path: /root/.cache/rclone/cache-backend/robgsc
2019/05/09 13:19:34 INFO  : robgsc: Chunk Memory: true
2019/05/09 13:19:34 INFO  : robgsc: Chunk Size: 32M
2019/05/09 13:19:34 INFO  : robgsc: Chunk Total Size: 20G
2019/05/09 13:19:34 INFO  : robgsc: Chunk Clean Interval: 1m0s
2019/05/09 13:19:34 INFO  : robgsc: Workers: 4
2019/05/09 13:19:34 INFO  : robgsc: File Age: 2.261904761904762w
2019/05/09 13:19:34 DEBUG : Adding path "cache/expire" to remote control registry
2019/05/09 13:19:34 DEBUG : Adding path "cache/stats" to remote control registry
2019/05/09 13:19:34 DEBUG : Adding path "cache/fetch" to remote control registry
2019/05/09 13:19:34 DEBUG : Cache remote robgsc:cloudp/o7aebe6dsvccihv3k25n0re7sk: new object '29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg'
2019/05/09 13:19:34 DEBUG : 29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg: find: warm object: 29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg, expiring on: 2019-05-25 09:18:44.882956241 -0400 EDT
2019/05/09 13:19:34 DEBUG : kodi-headless.tar: Couldn't find file - need to transfer
2019/05/09 13:19:34 DEBUG : cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg: moving offset set from 0 to 0
2019/05/09 13:19:34 DEBUG : cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg: cache reader closed 24366944
2019/05/09 13:19:34 INFO  : kodi-headless.tar: Copied (new)
2019/05/09 13:19:34 INFO  : 
Transferred:   	   23.232M / 23.232 MBytes, 100%, 8.983 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:        2.5s

2019/05/09 13:19:34 DEBUG : 12 go routines active
2019/05/09 13:19:34 DEBUG : rclone: Version "v1.47.0-015-gaba15f11-beta" finishing with parameters ["rclone" "copy" "robgsc-cryptp:ct/kodi-headless.tar" "/tmp/" "-vv"]
2019/05/09 13:19:34 DEBUG : Cache remote robgsc:cloudp/o7aebe6dsvccihv3k25n0re7sk: Services stopped
root@cloudc:~/.cache/rclone/cache-backend/robgsc# rm /tmp/kodi-headless.tar

root@cloudc:~/.cache/rclone/cache-backend/robgsc# rclone copy robgsc-cryptp:ct/kodi-headless.tar /tmp/ -vv^C

root@cloudc:~/.cache/rclone/cache-backend/robgsc# find . -ls
   662401      4 drwxr-xr-x   3 root     root         4096 May  9 13:18 .
   662716      4 drwxr-xr-x   3 root     root         4096 May  9 13:18 ./cloudp
   662772      4 drwxr-xr-x   3 root     root         4096 May  9 13:18 ./cloudp/o7aebe6dsvccihv3k25n0re7sk
   662773      4 drwxr-xr-x   2 root     root         4096 May  9 13:18 ./cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg
   662775  23796 -rwxr-xr-x   1 root     root     24366944 May  9 13:18 ./cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg/0
root@cloudc:~/.cache/rclone/cache-backend/robgsc# rm ./cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg/0
root@cloudc:~/.cache/rclone/cache-backend/robgsc# rclone copy robgsc-cryptp:ct/kodi-headless.tar /tmp/ -vv
2019/05/09 13:20:16 DEBUG : rclone: Version "v1.47.0-015-gaba15f11-beta" starting with parameters ["rclone" "copy" "robgsc-cryptp:ct/kodi-headless.tar" "/tmp/" "-vv"]
2019/05/09 13:20:16 DEBUG : Using config file from "/root/.rclone.conf"
2019/05/09 13:20:18 DEBUG : robgsc: wrapped robgs:cloudp/o7aebe6dsvccihv3k25n0re7sk at root cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg
2019/05/09 13:20:18 INFO  : robgsc: Cache DB path: /root/.cache/rclone/cache-backend/robgsc.db
2019/05/09 13:20:18 INFO  : robgsc: Cache chunk path: /root/.cache/rclone/cache-backend/robgsc
2019/05/09 13:20:18 INFO  : robgsc: Chunk Memory: true
2019/05/09 13:20:18 INFO  : robgsc: Chunk Size: 32M
2019/05/09 13:20:18 INFO  : robgsc: Chunk Total Size: 20G
2019/05/09 13:20:18 INFO  : robgsc: Chunk Clean Interval: 1m0s
2019/05/09 13:20:18 INFO  : robgsc: Workers: 4
2019/05/09 13:20:18 INFO  : robgsc: File Age: 2.261904761904762w
2019/05/09 13:20:18 DEBUG : Adding path "cache/expire" to remote control registry
2019/05/09 13:20:18 DEBUG : Adding path "cache/stats" to remote control registry
2019/05/09 13:20:18 DEBUG : Adding path "cache/fetch" to remote control registry
2019/05/09 13:20:18 DEBUG : Cache remote robgsc:cloudp/o7aebe6dsvccihv3k25n0re7sk: new object '29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg'
2019/05/09 13:20:18 DEBUG : 29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg: find: warm object: 29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg, expiring on: 2019-05-25 09:19:34.297478127 -0400 EDT
2019/05/09 13:20:18 DEBUG : kodi-headless.tar: Couldn't find file - need to transfer
2019/05/09 13:20:19 DEBUG : cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg: moving offset set from 0 to 0
2019/05/09 13:20:19 DEBUG : cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg: 0: chunk retry storage: 0
2019/05/09 13:20:19 DEBUG : cloudp/o7aebe6dsvccihv3k25n0re7sk/29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg: 0: chunk retry storage: 1
2019/05/09 13:20:19 DEBUG : worker-1 <29vh66gtpm4s9671vajn5pk71kp74qbomnlvp5radnljth15c1sg>: partial downloaded chunk 0
2019/05/09 13:20:20 DEBUG :

Although I can’t argue with that. :slight_smile: Its always better to use the supplied functions.

#14

may I ask how long the filename is unencrypted?

i am still playing with the thought of using the crypt of rclone and going away from encfs. but the long file names bother me and scare me off.

#15

Hello, i have no more issues with obfuscate filename encryption and cache…

1 Like