Rclone Crypt Dropbox

Hello. I have a very specific issue.

I need to encrypt files that I already have uploaded on dropbox. They are TiB and TiB of files so its impossible to download all files and reupload them through a crypt remote (standard encryption). What I was trying is an rclone moveto, moving from source (my dropbox remote, not encrypted) to destination (my dropbox remote, encrypted). This works and I can encrypt and move the files to the encrypted remote without downloading and then reuploading. This is ok but the problem is that I have so many files on dropbox to encrypt, I need a fast way to make this a batch fast task. It takes a lot of time, a lot of bandwith and I usually hit rate limits so this is 0% viable for me.

I was looking for something like --drive-server-side-across-configs=true, so it takes no bandwith and very little time to move the files to the encrypted remote (keep in mind that source is not encrypted and destinations is encrypted, although they are the same path/folder).

I know there is an option for doing this operation in a fast way between encrypted remotes (using --crypt-server-side-across-configs flag), but I haven´t found any option for doing this operation the same way between an unencrypted remote and an encrypted one.

I would be very grateful if someone could help me with this specific issue. Thanks!

cloud provider: Dropbox

Rclone version: 1.58.1

OS: Windows 10 Pro

That's because you can't.

Rclone is doing the encryption so you have to download and re-upload it.

You can rent a cheap unlimited box and let it go as that's really the only solution or use your own bandwidth.

Ok.

I don´t know what do you mean with "rent a cheap unlimited box". Can you explain me in detail whats that and how can I use that to make this process?

Thank you!

Find a seedbox/VPS provider that has unmetered/unlimited bandwidth.

An example would be:

BlackHOST - The Web is what we make of it!

I used that when I moved my data from Google to Dropbox as an example.

Ok thanks. I will study the option of those vps for 12 usd / month. Seems good service.

Also, do you know any service like this ones with 10 gbps bandwith?

I personally did not find anything cheap along those lines. You'd probably want to test it out and see what works best for you.

Thank you very much. So then there is no possibility of encrypting my dropbox files without using bandwidth right? No way of doing that as an internal server-side operation I mean.

Absolutely zero ways.

Rclone does the encryption so it has to pass through rclone.

One important fact is that I only need to encrypt the filename, not the file content. Thats why I´m looking for something that is equivalent to a rename operation. I want to hide my filenames to dropbox, not their content itself. So I need something that changes the filename and the extension (I have tried renaming the file adding .bin but the file gets corrupt and doesn´t work).

Hmm, you'd be able to script something I believe as that would work if you aren't talking about the content.

You'd probably have to.

  1. Make a full directory / file listings
  2. Create something local with 0 byte files or something
  3. On a local mount with the encrypted remote, move that 0 byte file to the mounted encrypted remote
  4. Validate things work and probably bundle that up into a script to move all the files on dropbox as that'll work without issue

I'd run with one file to test / validate the process.

Ok, I more or less understand. The thing is:

  • What kind of rclone encryption I should use (standard, off...)?
  • With 0 bytes local files it should work fast and easy, but how can I do it with my big files (100gb) that are already on dropbox?
  • What you mean is renaming my big files with the names given to the 0 bytes files?

The thing is encrypting only the filename so it seems as a normal dropbox rename operation.

Sorry for being annoying, I´m not rclone expert... :frowning:

hi,

might test/create a rclone crypt remote and move files into that crypt.
using --crypt-no-data-encryption

and this way I could move my files as an internal move operation without taking hours and bandwidth? (as with --drive-server-side-across-configs=true).

So here's what I did for a simple test.

Sharing the passwords doesn't matter as this is a local thing for me and going to be removed anyway.

[test]
type = alias
remote = /data/test

[testcrypt]
type = crypt
remote = test:crypt
password = 9xQ4lOGX8DPmK_1cWXJkQghPr_DkVZM-HRpEbBbl4tLPZ9-cTPP_lUWNzD7rgMtYZgyLwRyTD5_nr5q2A6bsrwObFsMLSs3n5eTsgd1FyQWT8wI8FG1sEvhC6XEEhG0AoVqO0Oab_qTfcaXw3jY4N-bdoPzr1TYJBGzSeVvkRqjih9YcBNBqR4wAwHCASRjE-kat5eLpc9nZdKU5S9Q8g-JDzwX0qcTi1jvhk7v4O6DnmjChEezb3Pkhow
password2 = L6YkE6OgnLoKsqIpcFqAP4daHjkJEQd6FryMyCn7Jp5jP4FHV5ZU5rSgIvh_O9goN_pXr9kgx1HMhUz8V3GHw38eN4GpGHO5CE7qOAJ1XGqkmyNnpkfyXvJNl0J5aAlMyju3VYHqnuUOpbca0cilh5AOi8ti8mO2CYTUEHGhnK-P40P2jS2axEwPVSB1evhFPgn1FHfUZEn-uiRmu3drIxcgXM98otkqU4f1OG1TUtt3l_pa1dnxMFhzjQ
directory_name_encryption = true
filename_encryption = standard
no_data_encryption = true

That's my rclone.conf

Here's an example of moving one file via rclone.

felix@gemini:/data/test$ vi file
felix@gemini:/data/test$ cat file
This is a test file
felix@gemini:/data/test$ rclone copyto test:file testcrypt:file
felix@gemini:/data/test$ cd crypt/
felix@gemini:/data/test/crypt$ cat k072gvbe6gitl3j2u47snvg7c8
This is a test file
felix@gemini:/data/test/crypt$ ls -al
total 4
drwxrwxr-x 1 felix felix 52 Aug  4 10:25 .
drwxrwxr-x 1 felix felix 18 Aug  4 10:25 ..
-rw-rw-r-- 1 felix felix 20 Aug  4 10:25 k072gvbe6gitl3j2u47snvg7c8

So you get a name there.

Test doing that to see if that works and does it server side. I'm not sure if it does or does not.

If not, you have to manually move a file to the crypt name and generate all the names / move scripts.

If you use the latest beta with the new --server-side-across-configs which works for all backends then I think it should do it server side.

Thank you very much to all of you!

I will study the different options and try to do this task without using my bandwith due to all the information you guys have given to me.

<3

rclone move --server-side-across-configs=true --transfers=1 "noncrypt:folder/file" "crypt:folder" --crypt-no-data-encryption -v

  • I´m using latest beta 1.60.
  • both remotes are the same dropbox account.

It still uses my bandwidth as if it was 2 different servers. Encrypted remote (crypt) has no_data_encryption = true.

Is there something wrong in the command?

Can you share your rclone.conf and the debug -vv of that command?

C:***\rclone-v1.60.0-beta.6389.918bd6d3c-windows-amd64>rclone move --drive-server-side-across-configs=true --transfers=1 "gonzalo_1:FARMEO_1861/plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot" "gonzalo_1_crypt1:" -vv --crypt-no-data-encryption
2022/08/04 17:00:18 DEBUG : rclone: Version "v1.60.0-beta.6389.918bd6d3c" starting with parameters ["rclone" "move" "--drive-server-side-across-configs=true" "--transfers=1" "gonzalo_1:FARMEO_1861/plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot" "gonzalo_1_crypt1:" "-vv" "--crypt-no-data-encryption"]
2022/08/04 17:00:18 DEBUG : Creating backend with remote "gonzalo_1:FARMEO_1861/plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot"
2022/08/04 17:00:18 DEBUG : Using config file from "C:\Users\jorge\Downloads\rclone-v1.60.0-beta.6389.918bd6d3c-windows-amd64\rclone.conf"
2022/08/04 17:00:19 DEBUG : fs cache: adding new entry for parent of "gonzalo_1:FARMEO_1861/plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot", "gonzalo_1:FARMEO_1861"
2022/08/04 17:00:19 DEBUG : Creating backend with remote "gonzalo_1_crypt1:"
2022/08/04 17:00:19 DEBUG : gonzalo_1_crypt1: detected overridden config - adding "{94O0Q}" suffix to name
2022/08/04 17:00:19 DEBUG : Creating backend with remote "gonzalo_1:FARMEO_1745"
2022/08/04 17:00:19 DEBUG : fs cache: renaming cache item "gonzalo_1_crypt1:" to be canonical "gonzalo_1_crypt1{94O0Q}:"
2022/08/04 17:00:20 DEBUG : plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot: Need to transfer - File not found at Destination
2022/08/04 17:00:22 DEBUG : plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot.bin: Uploading chunk 1/2163
2022/08/04 17:00:27 DEBUG : plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot.bin: Uploading chunk 2/2163
2022/08/04 17:00:30 DEBUG : plot-k32-2022-02-28-18-46-25df8eb5d9e7ec1a524d5e2327be136072f476f45501ecf846cdecb07fe6122a.plot.bin: Uploading chunk 3/2163

[gonzalo_1]
type = dropbox
client_id = ****************
client_secret = **************************
token = {"access_token":"**********************"}

[gonzalo_1_crypt1]
type = crypt
remote = gonzalo_1:FARMEO_1861
password = *********************************
directory_name_encryption = false
filename_encryption = standard
no_data_encryption = true

Its uploading chunks, not server side operation.

Sorry as I worded that so bad. It does work for a non crypt, but does not work for a crypt in the middle.

felix@gemini:~/rclone-v1.60.0-beta.6389.918bd6d3c-linux-amd64$ ./rclone copyto DB:hosts DBMovies::hosts --server-side-across-configs -vvv
2022/08/04 11:06:46 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/08/04 11:06:46 DEBUG : rclone: Version "v1.60.0-beta.6389.918bd6d3c" starting with parameters ["./rclone" "copyto" "DB:hosts" "DBMovies::hosts" "--server-side-across-configs" "-vvv"]
2022/08/04 11:06:46 DEBUG : Creating backend with remote "DB:hosts"
2022/08/04 11:06:46 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/08/04 11:06:46 DEBUG : fs cache: adding new entry for parent of "DB:hosts", "DB:"
2022/08/04 11:06:46 DEBUG : Creating backend with remote "DBMovies:"
2022/08/04 11:06:46 DEBUG : hosts: Need to transfer - File not found at Destination
2022/08/04 11:06:48 DEBUG : hosts: dropbox = 1bbf7015266112d432e56a311be9ab2d2be0dfbd9c2103e66f0c7c5237e8ae83 OK
2022/08/04 11:06:48 INFO  : hosts: Copied (server-side copy) to: :hosts
2022/08/04 11:06:48 INFO  :
Transferred:   	        327 B / 327 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         2.2s

2022/08/04 11:06:48 DEBUG : 8 go routines active
2022/08/04 11:06:48 INFO  : Dropbox root '': Commiting uploads - please wait...
2022/08/04 11:06:48 INFO  : Dropbox root '': Commiting uploads - please wait...

For a crypt there.

felix@gemini:~/rclone-v1.60.0-beta.6389.918bd6d3c-linux-amd64$ ./rclone copyto DB:hosts TESTDBC:hosts --server-side-across-configs --crypt-server-side-across-configs -vvv
2022/08/04 11:04:46 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/08/04 11:04:46 DEBUG : rclone: Version "v1.60.0-beta.6389.918bd6d3c" starting with parameters ["./rclone" "copyto" "DB:hosts" "TESTDBC:hosts" "--server-side-across-configs" "--crypt-server-side-across-configs" "-vvv"]
2022/08/04 11:04:46 DEBUG : Creating backend with remote "DB:hosts"
2022/08/04 11:04:46 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/08/04 11:04:46 DEBUG : fs cache: adding new entry for parent of "DB:hosts", "DB:"
2022/08/04 11:04:46 DEBUG : Creating backend with remote "TESTDBC:"
2022/08/04 11:04:46 DEBUG : TESTDBC: detected overridden config - adding "{Db_Y9}" suffix to name
2022/08/04 11:04:46 DEBUG : Creating backend with remote "DB:testing"
2022/08/04 11:04:47 DEBUG : fs cache: renaming cache item "TESTDBC:" to be canonical "TESTDBC{Db_Y9}:"
2022/08/04 11:04:47 DEBUG : hosts: Need to transfer - File not found at Destination
2022/08/04 11:04:48 DEBUG : 0vret8lkk0hhuhm67lgn1pakak: Uploading chunk 1/1
2022/08/04 11:04:48 DEBUG : 0vret8lkk0hhuhm67lgn1pakak: Uploading chunk 2/1
2022/08/04 11:04:49 DEBUG : Dropbox root 'testing': Adding "/testing/0vret8lkk0hhuhm67lgn1pakak" to batch
2022/08/04 11:04:50 DEBUG : Dropbox root 'testing': Batch idle for 500ms so committing
2022/08/04 11:04:50 DEBUG : Dropbox root 'testing': Committing sync batch length 1 starting with: /testing/0vret8lkk0hhuhm67lgn1pakak
2022/08/04 11:04:50 DEBUG : Dropbox root 'testing': Committed sync batch length 1 starting with: /testing/0vret8lkk0hhuhm67lgn1pakak
2022/08/04 11:04:50 INFO  : hosts: Copied (new)
2022/08/04 11:04:50 INFO  :
Transferred:   	        327 B / 327 B, 100%, 108 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         4.1s

2022/08/04 11:04:50 DEBUG : 13 go routines active
2022/08/04 11:04:50 INFO  : Dropbox root 'testing': Commiting uploads - please wait...
2022/08/04 11:04:50 INFO  : Dropbox root '': Commiting uploads - please wait...