i have moved my PMS from Ubuntu to Windows and have changed other parts of my setup as well. At first glance, everything seems to work fine, but i now get into some weird trouble.
When i am performing a library scan for my Plex Media Server, i receive lots of unexpected EOF error messages. I first thought it was because of the rclone union instead of mounting the GD remote but these errors also occur when plex is scanning the files on the gdrive_crypt remote.
What is your rclone version (output from rclone version)
rclone v1.53.3
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 10 64bit
Which cloud storage system are you using? (eg Google Drive)
GD
The command you were trying to run (eg `rclone copy /tmp
mount movies-u: c:\mnt\union\movies-u --stats=0 --allow-other --dir-cache-time 96h --log-level INFO --log-file c:\logs\rc-movies-u.log --drive-use-trash --config "c:\config\rclone\rclone.conf" --vfs-cache-poll-interval 1m --drive-v2-download-min-size 100m
The rclone config contents with secrets removed.
[gdrive]
type = drive
client_id = xxx
client_secret = xxx
token = {"access_token":"xxx"}
team_drive =
root_folder_id = xxx
[gdrive_crypt]
type = crypt
remote = gdrive:Video/
filename_encryption = standard
password = xxx
password2 = xxx
[movies-u]
type = union
upstreams = gdrive_crypt:/Filme:nc c:\mnt\local\movies-local
[EDIT]
After some additional reading in the forum i came across this thread which covers a similar issue. As my previous setup matches the one of the OP, i guess that i am having the exact same issue without noticing it earlier.
However, i am curious why plex never seemed to have an issue with the file until i changed my setup.
When i try to copy a file from the logs manually, the size reported by GD actually doesn't seem to match with the actual size:
The file cannot be written to my local drive as the process restarts when it gets past the 41G.
I have made several attempts to scan my media library within the last 10 days and it looks like the list of files being reported with EOF is dynamic and increasing which i would not understand.
Does rclone have a function that would be able to find these corrupted files?
--allow-other can be removed as it does nothing on windows.
tho that is the solution to the problem.
that rclone mount uses --vfs-cache-poll-interval 1m,
yet --vfs-cache-mode is not set so it defaults to off.
thus, that flag does nothing
is that intentional?
have you tried --vfs-cache-mode=full?
what is the purpose of -drive-v2-download-min-size 100m
As always guys, thanks for your immediate responses and your help!
After reading through several posts here, i figured out that my files might have been corrupted because of my workflow which assumed that only files could be moved by my rclone cron job that have been copied to their destination by 100 %. Maybe this was not the case and therefore resulted in files that were still transferring between to directories on my box but have been moved to gdrive by a script.
So i ran a complete media scan on a mount spilling out debug log info to find which files have to be removed. It turns out that due to their size some of them must be corrupt so i don't doubt that the others are, too.
I have not but will give it a try. I was wondering why the the 1m poll wouldn't refresh the cache for my mounts, thanks for the explanation.
This was an outcome of the research i did where someone has described an issue that sounded familiar to mine, but it wasn't. i have already removed this flag.
Looks like right now, but hopefully not in the long run. I am a Windows guy and will be, especially because i couldn't get a screen reader to work properly in terminal.
Hmm, if you are moving a file and it aborts part oft he way or even at 99%, it should fail the upload and nothing should be there as there is no partial file uploads on Google Drive.
Maybe we have to take into consideration the extra layer added by mergerfs which i used to provide a unified view of the local media and remote media for my PMS.
It would definitely be easier to trace back if i had noticed it earlier and could nail it down to a certain time frame. Actually i cannot tell if this really happened with my previous setup where Radarr was moving the files received from nzbget to the local upstream of a mergerfs union from where the files have been uploaded to google or if it has just happened with the new setup where i started to use the rclone union instead but have only moved about five to ten files from local to remote so far and none of them was affected by the EOF issue.
Same here. What i can say is that only files that have been uploaded within the last 30 days have been affected. But as far as i can remember, nothing has changed with my setup.
In some cases, data loss is quite significant as the file that has been uploaded is about 50 or 200 MB from 10 or 15 GB.
