Filters in rclone cryptdecode

Feature
1

It would be cool if rclone cryptdecode had support for filters. One use case is moving files across a server that you don't trust. For example:

rclone cryptdecode --reverse encrytped-remote: pictures/ --include '*.png'

This could be then fed to ssh command:

ssh admin@server for f in $encrypted_files; do rclone moveto remote1:$f remote2:$f; done

Currently, the only method that works is to run rclone cryptdecode in a for loop:

rclone_remote=encrypted-remote:; rclone_path=pictures; for i in $(rclone lsf $rclone_remote$rclone_path | grep -i png); do rclone cryptdecode --reverse $rclone_remote $rclone_path$i; done
2

Feel free to submit a PR to add that, sponsor it or ask someone to do it for you :slight_smile:

1 Like
3

I wouldn't mind chipping in a few hundreds for this particular feature. I'm not a go developer unfortunately, so besides a lousy shell script can't do much. If someone is though please let me know.

4

Rclone cryptdecode doesn't list the backend so doesn't support any wildcards.

However I think an easier fix for your problem would be to set up an actual crypt remote pointing at the remote server wouldn't it?

5

In this scenario there are two untrusted remotes (google and aws) and I hoped that you can move/copy between them using encrypted paths (e.g. rclone move-encrypted remote1: remote2: --include "**.pattern"). This could be done either on a local machine or on the server (using ssh like in the example above), but doing it on the server is just a bit faster. There may be a better way to do it, but I couldn't think of anything other than feeding encrypted paths to rclone commands.

6

I do not get why you want to complicate it like this.

You must trust your server - as even for your rclone move-encrypted you require rclone.config. So why not just move file(s) between two crypt remotes? What is logic here?

7

The idea was that rclone.conf on the servers don't have decryption passwords - only your own machine can see which encrypted paths correspond to which files. The benefit here is that all decryption always happens on machines you physically own. The downside as I found out recently is that your machine needs to coordinate all the operations, in this example requiring decrypting file names on the fly. It may not be a common use case, but if you are a journalist for example these extra precautions may be valuable sometimes.

8

This seems to be working - it decrypts paths names on a local computer and sends them to a server over ssh to move selected files between two remotes:

IFS=$'\n'; for i in $(rclone ls '<DECRYPTED-REMOTE-1>:' --include '<YOUR-FILTER>' | awk '!($1="")' | cut -c 2- ); do rclone cryptdecode --reverse <DECRYPTED-REMOTE-1>: $i | cut -f2- | xargs -I {} ssh <REMOTE-SERVER> rclone moveto <ENCRYPTED-REMOTE-1>:<PATH-TO-CRYPT>/{} <ENCRYPTED-REMOTE-2>:<PATH-TO-CRYPT>/{} --progress; done

Conveniently right before my google drive quota was about to expire :partying_face:

1 Like
9

The above command transferred file names one at a time, which was very slow. Command below transfers the entire list of files as a file and then reads it on the server with --files-from option - much much faster:

IFS=$'\n'; for i in $(rclone ls '<DECRYPTED-REMOTE-1:>' --include '<YOUR-FILTER>' | awk '!($1="")' | cut -c 2- ); do rclone cryptdecode --reverse <DECRYPTED-REMOTE-1:> $i | cut -f2-; done | ssh <REMOTE-SERVER> "cat > temp-rclone-files" && ssh <REMOTE-SERVER> rclone moveto <ENCRYPTED-REMOTE-1>:<PATH-TO-CRYPT>/ <ENCRYPTED-REMOTE-2>:<PATH-TO-CRYPT> --files-from=./temp-rclone-files --progress
10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.