Filters in rclone cryptdecode

It would be cool if rclone cryptdecode had support for filters. One use case is moving files across a server that you don't trust. For example:

rclone cryptdecode --reverse encrytped-remote: pictures/ --include '*.png'

This could be then fed to ssh command:

ssh admin@server for f in $encrypted_files; do rclone moveto remote1:$f remote2:$f; done

Currently, the only method that works is to run rclone cryptdecode in a for loop:

rclone_remote=encrypted-remote:; rclone_path=pictures; for i in $(rclone lsf $rclone_remote$rclone_path | grep -i png); do rclone cryptdecode --reverse $rclone_remote $rclone_path$i; done

Feel free to submit a PR to add that, sponsor it or ask someone to do it for you :slight_smile:

1 Like

I wouldn't mind chipping in a few hundreds for this particular feature. I'm not a go developer unfortunately, so besides a lousy shell script can't do much. If someone is though please let me know.

Rclone cryptdecode doesn't list the backend so doesn't support any wildcards.

However I think an easier fix for your problem would be to set up an actual crypt remote pointing at the remote server wouldn't it?

In this scenario there are two untrusted remotes (google and aws) and I hoped that you can move/copy between them using encrypted paths (e.g. rclone move-encrypted remote1: remote2: --include "**.pattern"). This could be done either on a local machine or on the server (using ssh like in the example above), but doing it on the server is just a bit faster. There may be a better way to do it, but I couldn't think of anything other than feeding encrypted paths to rclone commands.

I do not get why you want to complicate it like this.

You must trust your server - as even for your rclone move-encrypted you require rclone.config. So why not just move file(s) between two crypt remotes? What is logic here?

The idea was that rclone.conf on the servers don't have decryption passwords - only your own machine can see which encrypted paths correspond to which files. The benefit here is that all decryption always happens on machines you physically own. The downside as I found out recently is that your machine needs to coordinate all the operations, in this example requiring decrypting file names on the fly. It may not be a common use case, but if you are a journalist for example these extra precautions may be valuable sometimes.

This seems to be working - it decrypts paths names on a local computer and sends them to a server over ssh to move selected files between two remotes:

IFS=$'\n'; for i in $(rclone ls '<DECRYPTED-REMOTE-1>:' --include '<YOUR-FILTER>' | awk '!($1="")' | cut -c 2- ); do rclone cryptdecode --reverse <DECRYPTED-REMOTE-1>: $i | cut -f2- | xargs -I {} ssh <REMOTE-SERVER> rclone moveto <ENCRYPTED-REMOTE-1>:<PATH-TO-CRYPT>/{} <ENCRYPTED-REMOTE-2>:<PATH-TO-CRYPT>/{} --progress; done

Conveniently right before my google drive quota was about to expire :partying_face:

1 Like

The above command transferred file names one at a time, which was very slow. Command below transfers the entire list of files as a file and then reads it on the server with --files-from option - much much faster:

IFS=$'\n'; for i in $(rclone ls '<DECRYPTED-REMOTE-1:>' --include '<YOUR-FILTER>' | awk '!($1="")' | cut -c 2- ); do rclone cryptdecode --reverse <DECRYPTED-REMOTE-1:> $i | cut -f2-; done | ssh <REMOTE-SERVER> "cat > temp-rclone-files" && ssh <REMOTE-SERVER> rclone moveto <ENCRYPTED-REMOTE-1>:<PATH-TO-CRYPT>/ <ENCRYPTED-REMOTE-2>:<PATH-TO-CRYPT> --files-from=./temp-rclone-files --progress

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.