Some files not visible

I’ve been having a problem lately and I don’t really know where it’s coming from. Some uploaded files are not visible in the Mount/File Manager after uploading, but in the browser at Google online in the WebUI. Not even after I found it or scanned it on Plex…Before, they were visible in the folders.

Use the settings from here:

The only thing I’ve changed is I’ve always started with allow-non-empty. But I had read that it could be problematic, so I took it out.

use the following commands:

rclone mount --allow-other --dir-cache-time 96h --drive-chunk-size 32M --timeout 1h --log-level INFO --log-file="/home/hd1/ttopxelp/rclonelogs/rclone.log" secretG: /home/hd1/ttopxelp/cloud-media/ &

rclone move -P --fast-list --max-transfer 750G --bwlimit 50M --transfers 3 --checkers 3 --drive-chunk-size 32M --tpslimit 3 -v /home/hd1/ttopxelp/media/ secretG:/

Errors from the log:
2019/04/24 11:38:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “TV Shows/Stromberg”: not an encrypted file - no “.bin” suffix
2019/04/24 11:50:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “TV Shows/American Dad/Season 16”: not an encrypted file - no “.bin” suffix
2019/04/24 11:50:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Aftershock (2010)”: not an encrypted file - no “.bin” suffix
2019/04/24 11:55:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “TV Shows/Bosch/Season 05”: not an encrypted file - no “.bin” suffix
2019/04/24 12:21:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Blutige Erdbeeren (1970)”: not an encrypted file - no “.bin” suffix
2019/04/24 12:24:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Bumblebee (2018)”: not an encrypted file - no “.bin” suffix
2019/04/24 12:29:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Coal Miner’s Daughter (1980)”: not an encrypted file - no “.bin” suffix
2019/04/24 12:32:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Cul-de-sac (1966)”: not an encrypted file - no “.bin” suffix
2019/04/24 12:49:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Fantomas (1964)”: not an encrypted file - no “.bin” suffix
2019/04/24 12:50:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Fantomas Unleashed (1965)”: not an encrypted file - no “.bin” suffix
2019/04/24 12:51:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Fantomas vs. Scotland Yard (1967)”: not an encrypted file - no “.bin” suffix
2019/04/24 13:08:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Gattaca (1997)”: not an encrypted file - no “.bin” suffix
2019/04/24 13:08:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “TV Shows/The Good Place/Season 02”: not an encrypted file - no “.bin” suffix
2019/04/24 13:18:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Get Carter (1971)”: not an encrypted file - no “.bin” suffix
2019/04/24 13:20:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Gorky Park (1983)”: not an encrypted file - no “.bin” suffix
2019/04/24 18:23:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “”: not an encrypted file - no “.bin” suffix
2019/04/24 18:30:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies”: not an encrypted file - no “.bin” suffix
2019/04/24 18:30:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Fantomas vs. Scotland Yard (1967)”: not an encrypted file - no “.bin” suffix
2019/04/24 21:37:39 ERROR : Filme/25 km h (2018).mkv: ReadFileHandle.Read error: low level retry 1/10: read tcp 95.211.216.138:34756->172.217.17.42:443: read: connection reset by peer
2019/04/25 09:22:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “”: not an encrypted file - no “.bin” suffix
2019/04/25 09:22:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Aftershock (2010)”: not an encrypted file - no “.bin” suffix
2019/04/25 09:22:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies”: not an encrypted file - no “.bin” suffix
2019/04/25 09:40:15 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 1/10: EOF
2019/04/25 09:40:16 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 2/10: EOF
2019/04/25 09:40:16 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 3/10: EOF
2019/04/25 09:40:17 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 4/10: EOF
2019/04/25 09:40:17 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 5/10: EOF
2019/04/25 09:40:17 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 6/10: EOF
2019/04/25 09:40:18 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 7/10: EOF
2019/04/25 09:40:18 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 8/10: EOF
2019/04/25 09:40:18 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 9/10: EOF
2019/04/25 09:40:19 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: low level retry 10/10: EOF
2019/04/25 09:40:19 ERROR : Filme/A Most Wanted Man (2014).idx: ReadFileHandle.Read error: EOF
2019/04/25 09:40:21 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 1/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:28 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 2/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:28 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 3/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:29 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 4/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:30 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 5/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:30 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 6/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:31 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 7/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:31 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 8/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:31 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 9/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:32 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 10/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:32 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:32 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 1/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:33 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 2/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:33 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 3/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:33 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 4/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:34 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 5/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:34 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 6/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:35 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 7/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:35 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 8/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:35 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 9/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:36 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: low level retry 10/10: failed to authenticate decrypted block - bad password?
2019/04/25 09:40:36 ERROR : Filme/A Most Wanted Man (2014).sub: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password?
2019/04/25 12:26:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “”: not an encrypted file - no “.bin” suffix
2019/04/25 12:26:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “TV Shows/Stromberg”: not an encrypted file - no “.bin” suffix
2019/04/26 11:29:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Music”: not an encrypted file - no “.bin” suffix
2019/04/26 11:29:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “”: not an encrypted file - no “.bin” suffix
2019/04/26 11:29:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies”: not an encrypted file - no “.bin” suffix
2019/04/26 12:45:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “Movies/Gattaca (1997)”: not an encrypted file - no “.bin” suffix
2019/04/26 15:08:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “”: not an encrypted file - no “.bin” suffix
2019/04/26 15:08:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “TV Shows”: not an encrypted file - no “.bin” suffix
2019/04/26 15:09:37 NOTICE: Encrypted drive ‘secretG:’: ChangeNotify was unable to decrypt “TV Shows/Stromberg”: not an encrypted file - no “.bin” suffix

Folders/files from the error message all have a .bin extension

config:

[secretG]
type = crypt
remote = Google:Plex
filename_encryption = off
password = XXX
password2 = XXX

[Google]
type = drive
client_id = XXX
client_secret = XXX
token = {"access_token":"XXX,"token_type":"Bearer","refresh_token":"1/Q9BKN5_uWypMnu9BergsklZPKqKUN4n6CcZ1lfwTl5E","expiry":"2019-04-26T15:52:36.953095436+02:00"}

does it have something to do with the expiration date of my token?

own ID/API? yes
rclone Version 1.47
Ubuntu 16.04.

log:

How can that be, always use the same upload command and never changed anything in the config (since more than a year)…

Thank you in advance for your help.

Those errors mean you have unecrypted files in your mount. Since you have file name encryption off, makes it a bit tougher, but each file you get an error for means it’s unencrypted.

how can that be when I upload with the above command to my encrypted crypt (secretG:)? Shouldn’t they be encrypted when uploading with the command? And does the extension .bin not mean that they are encrypted?

That file is not encrypted.

That file seems to have a bad password to it.

I’d probably try to figure out what you changed along the way as something changed as you have files that aren’t encrypted and a bad password.

Ok I understood I read also so. But as already written I have not changed anything except the allow non empty taken out and uploaded with the 1.47…

the file has a suffix like this

additionally if I move it to another folder, from Movies to Filme (use one to upload and the other to finally save), then the file is visible and also readable…but both folders were always readable before and I haven’t changed anything about them either

I can’t tell you what changed from your side unfortunately as we can only see what the logs are saying.

allow-non-empty is related to over mounting, which creates bad artifacts usually but looks like this:

  • you have a mount on /media/somemount
    if /media/somemount is not mounted and you write a file to /media/somemount/tv.mkv, it gets written to the local file system
  • if you mount say rclone or anything to /media/somemount, the file tv.mkv disappears as you mount on top of it.

If you don’t have that allow-non-empty, you generate an error and can look at the mount point and move / see what is there.

That flag has nothing to do with encryption or anything like that.

The goal would be to figure out what changed to get an unencrypted file into your encrypted area.

doesn’t it mean that all files are encrypted if I upload with this command, with which I also uploaded the file Blutige Erdbeeren? Or did I misunderstand something?

Why tougher?

I guess tougher is a bad word, but seeing non junk file names is easier to eyeball and see what’s wrong.

You can find a find on your mount to look for non .bin files

find /mount -type f -not -name *.bin

Sorry, maybe I’m getting something wrong or we’re talking at cross purposes. :sweat_smile:

Shouldn’t the files in my mount and all of them be without .bin, therefore decrypted? Otherwise Plex wouldn’t be able to read them after all.

If I executed your command now, would it just list all files or not?

You’d mount your base Google Drive and go into the directory that is encrypted and look for ‘not .bin’ files. If you mount the encrypted drive, we’re already aware it cannot see the files so that would be pointless :slight_smile:

OK funny…unfortunately I didn’t wait for your answer and executed the command as on the encrypted drive. It just went all through. And suddenly the files are there. lol

What’s going on? :face_with_raised_eyebrow:

My ESP powers are limited since it’s Friday so I need log files and can help explain :slight_smile:

1 Like

From what I read here, if anyone had such abilities then thought you :wink:

In my opinion there is nothing new in the logs

If you look at those logs, you’ve had things broken for quite some time:

2019/03/28 16:02:42 NOTICE: Encrypted drive 'secretG:': ChangeNotify was unable to decrypt "Movies/Donnie Darko (2001)": not an encrypted file - no ".bin" suffix
2019/03/28 16:04:42 NOTICE: Encrypted drive 'secretG:': ChangeNotify was unable to decrypt "Movies/Dune (1984)": not an encrypted file - no ".bin" suffix
2019/03/28 16:07:42 NOTICE: Encrypted drive 'secretG:': ChangeNotify was unable to decrypt "Movies/The Fly (1986)": not an encrypted file - no ".bin" suffix

You got a lot of download quota errors too:

2019/04/15 02:05:05 ERROR : Cartoons/The Ren and Stimpy Show/Season 04/The Ren and Stimpy Show - S04E04 - Blazing Entrails.mkv: ReadFileHandle.Read error: low level retry 8/10: open file failed: googleapi: Erro
r 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2019/04/15 02:05:05 ERROR : Cartoons/The Ren and Stimpy Show/Season 04/The Ren and Stimpy Show - S04E04 - Blazing Entrails.mkv: ReadFileHandle.Read error: low level retry 9/10: open file failed: googleapi: Erro
r 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2019/04/15 02:05:06 ERROR : Cartoons/The Ren and Stimpy Show/Season 04/The Ren and Stimpy Show - S04E04 - Blazing Entrails.mkv: ReadFileHandle.Read error: low level retry 10/10: open file failed: googleapi: Err
or 403: The download quota for this file has been exceeded., downloadQuotaExceeded

Any reason why you didn’t setup directory and file name encryption? Did something go on with your GD as I wonder if that’s the case. Those errors have been going on for a few weeks from the logs.

It definitely seem like your GD has something going on.

The download error message came because I was using the intensive analysis for a while at Plex…

The files with the bin message have all always been moved with the same command. As part of multiple files where others from this all do not have this error message. And also these are all provided with a .bin when I look online at google

Why should I encrypt my file names? What is the advantage? Except that Google doesn’t see what I have…For me it was easier not to do it so I have a better overview. Can also see online in the browser at google exactly where the files are. Since I am not so CLI versed it was easier for me

I ran the command now against my base google drive and nothing was listed

The advantage is the obvious one that people cannot see your filenames nor what’s being stored in them.

I can’t tell you what changed other than the log file is saying based on what you are doing now.

Pick one file that seems to be giving you a problem and try to run some commands against the one line.

rclone ls SecretG:Filme/A Most Wanted Man (2014).idx -vv as an example. If that shows up with a .bin and can’t be decrypted. It gives you a starting point.

I easily recreated your message by dropping a few files in there unecrypted:

rclone mount testenc: /Test
2019/04/26 12:59:21 NOTICE: Encrypted drive 'testenc:': ChangeNotify was unable to decrypt "": not an encrypted file - no ".bin" suffix
2019/04/26 12:59:21 NOTICE: Encrypted drive 'testenc:': ChangeNotify was unable to decrypt "Blah": not an encrypted file - no ".bin" suffix
2019/04/26 12:59:21 NOTICE: Encrypted drive 'testenc:': ChangeNotify was unable to decrypt "Blah/Test": not an encrypted file - no ".bin" suffix

it shows no .bin :confused:

If you run a ls on your encrypted remote, it wouldn’t show a .bin extension since it is encrypted and decrypts it.

What were you expecting to happen?