Best way to find crypt files that aren't seen within the crypt remote

Help and Support
1

I believe I have quite a few files in a crypt structure that aren’t viewable within that crypt remote. What i’ve been doing is keeping a crypt directory mirrored to a remote google drive like this:

rclone --checksum sync GD1:crypt GD2:crypt

This mirrors that crypt structure physically. The issue seems to be that the size of that crypt is HUGELY different when I do a size logically like

rclone size GD1-crypt: and rclone size GD1:crypt where GD1-crypt: is the logical crypt remote of GD1:crypt (I realize there will be some overhead from encryption but I’m seeing massive differences)

This leads me to believe there are files in GD1:crypt that cannot be read within the GD-crypt: remote. I suppose the reasons could be too long of a name or even a different key used long ago by accident. Im not sure.

The question here though is how can I identify those files that are not supposed to be there? I know there is a rclone cryptdecode that I could use to reverse the names and do a difference but that command is only useful file by file and not across a 8TB remote.

Any hints/suggestions? The only way I can thing of doing this is to sync the files to a different directory from within the crypt remote and delete the old directory but that is a LOT of data to move around and would need to be done across 3 different google drives.

2

hmm.

rclone lsjson zonegd-cryptp: --encrypted --recursive 2>&1 | awk -F, ‘{print $3}’ | awk -F: ‘{print $2}’ | sed ‘s/"/**//’ | sed ‘s/"//’

that may work to generate a exclude list to pass into rclone ls.

3

Well, I answered my own question (I think). Maybe someone will find this helpful. It isn’t perfect as I am only looking at the actual file name and directory so if they exist in multiple places it excludes both but its pretty decent.

rclone lsjson robgs-cryptp: --encrypted --recursive 2>&1 > alljson.txt

cat alljson.txt | grep ‘":"’ | awk -F’","’ ‘{print $3}’ | awk -F’":"’ ‘{print $2}’ | sed ‘s/^/**//’ > allfilter

rclone lsjson robgs:cloudp --exclude-from ./allfilter 2>&1 > odd

4

If you do rclone size -vv then you’ll see an error for every undecryptable file name

$ rclone -vv size secret:
2018/12/12 21:49:18 DEBUG : rclone: Version "v1.45-DEV" starting with parameters ["rclone" "-vv" "size" "secret:"]
2018/12/12 21:49:18 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2018/12/12 21:49:18 DEBUG : file.txt: Skipping undecryptable file name: illegal base32 data at input byte 4
Total objects: 0
Total size: 0 Bytes (0 Bytes)
2018/12/12 21:49:18 DEBUG : 2 go routines active
2018/12/12 21:49:18 DEBUG : rclone: Version "v1.45-DEV" finishing with parameters ["rclone" "-vv" "size" "secret:"]
$

You can at least identify the files like that.

1 Like