EDIT: Just found this log when reading a file. Is all this just because of the daily quota? I thought it was only upload and not download? I tried to generate new ID clients OAuth 2.0, but it's not working:
2022/02/16 10:30:01 ERROR : media/movies/The Dry (2021)/The.Dry.2020.FRENCH.1080p.BluRay.mHD.x264-ETHERUM.mkv: vfs cache: failed to download: vfs reader: failed to write to cache file: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Hello there,
So just to give a little bit of context before technical details: I've been using rClone, gDrive, and Plex/Jellyfin for more than a year now (using this script), everything was fine and working until yesterday.
A weird bug happened : everything on my Plex/Library vanished (I did not changed anything on my server for month, but I think this bug came from the fact that my os hard drive was full and everything was frozen until I freed some space), heart attack, checked on my gDrive and all the files are still there phew.
I just launched a new Plex/Jellyfin scan, and tried to read something: Working!
Next morning, when the agent finished scanning my 4000+ media library, impossible to read anything.
Tried to add a new media: Working! But after a few hours, even the new media does not work anymore. I tried to delete Plex database and rebuilt it from scratch : exact same issue.
Here's some info about my server :
rclone version:
rclone v1.57.0
- os/version: debian 10.11 (64 bit)
- os/kernel: 4.19.0-18-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none
mergerfs version:
2.33.3~debian-buster
Linux version:
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
cloud storage system:
Google Drive
rclone config:
[gdrive]
type = drive
client_id = xxxxxx.apps.googleusercontent.com
client_secret = WxxxxxY0
scope = drive
token = {"access_token":"yaxx.A0xxxxxyW-TzxxxxxxLCg","token_type":"Bearer","refresh_token":"1//09xxxxx-SCxxxxwF-Lxxx2_Daxxxxxx0zqxphmiI","expiry":"2022-02-16T10:38:24.782183057+01:00"}
root_folder_id = 0AxxxxxxxVA
rClone command / script:
######################
#### Mount Script ####
######################
## Version 0.96.9.3 ##
######################
####### EDIT ONLY THESE SETTINGS #######
# INSTRUCTIONS
# 1. Change the name of the rclone remote and shares to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: include custom command and bind mount settings
# 4. Optional: include extra folders in mergerfs mount
# REQUIRED SETTINGS
RcloneRemoteName="gdrive" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
RcloneMountShare="/data/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone
RcloneMountDirCacheTime="720h" # rclone dir cache time
LocalFilesShare="/data/local" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable
RcloneCacheShare="/data/cache" # location of rclone cache files without trailing slash e.g. /mnt/user0/mount_rclone
RcloneCacheMaxSize="0G" # Maximum size of rclone cache
RcloneCacheMaxAge="0h" # Maximum age of cache files
MergerfsMountShare="/data/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable
DockerStart="" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page
MountFolders=\{"torrents/downloads,media/movies,media/movies4k,media/tv,media/tv4k"\} # comma separated list of folders to create within the mount
# Note: Again - remember to NOT use ':' in your remote name above
# OPTIONAL SETTINGS
# Add extra paths to mergerfs mount in addition to LocalFilesShare
LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable
LocalFilesShare3="ignore"
LocalFilesShare4="ignore"
# Add extra commands or filters
Command1="--rc"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""
CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter
RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended
VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them
####### END SETTINGS #######
###############################################################################
##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING #######
###############################################################################
####### Preparing mount location variables #######
RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount
LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount
MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location
####### create directories for rclone mount and mergerfs mounts #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName # for script files
mkdir -p $RcloneCacheShare/cache/$RcloneRemoteName # for cache files
if [[ $LocalFilesShare == 'ignore' ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested."
LocalFilesLocation="/tmp/$RcloneRemoteName"
eval mkdir -p $LocalFilesLocation
else
echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders."
eval mkdir -p $LocalFilesLocation/"$MountFolders"
fi
mkdir -p $RcloneMountLocation
if [[ $MergerfsMountShare == 'ignore' ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Not creating MergerFS folders as requested."
else
echo "$(date "+%d.%m.%Y %T") INFO: Creating MergerFS folders."
mkdir -p $MergerFSMountLocation
fi
####### Check if script is already running #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}"
echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running."
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running."
exit
else
echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
fi
####### Checking have connectivity #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online"
ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform
if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online"
else
echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run"
rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
exit
fi
####### Create Rclone Mount #######
# Check If Rclone Mount Already Created
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."
# Creating mountcheck file in case it doesn't already exist
echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote."
touch mountcheck
rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse
# Check bind option
if [[ $CreateBindMount == 'Y' ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
else
echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}"
ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
fi
echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}"
else
RCloneMountIP=""
echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}"
fi
# create rclone mount
rclone mount \
$Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \
--allow-other \
--uid 911 \
--gid 911 \
--umask 000 \
--dir-cache-time $RcloneMountDirCacheTime \
--attr-timeout $RcloneMountDirCacheTime \
--log-level INFO \
--poll-interval 10s \
--cache-dir=$RcloneCacheShare/cache/$RcloneRemoteName \
--drive-pacer-min-sleep 10ms \
--drive-pacer-burst 1000 \
--vfs-cache-mode full \
--vfs-cache-max-size $RcloneCacheMaxSize \
--vfs-cache-max-age $RcloneCacheMaxAge \
--vfs-read-ahead 1G \
--bind=$RCloneMountIP \
$RcloneRemoteName: $RcloneMountLocation &
# Check if Mount Successful
echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds"
# slight pause to give mount time to finalise
sleep 5
echo "$(date "+%d.%m.%Y %T") INFO: continuing..."
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount."
else
echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers"
docker stop $DockerStart
rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
exit
fi
fi
####### Start MergerFS Mount #######
if [[ $MergerfsMountShare == 'ignore' ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested."
else
if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place."
else
# check if mergerfs already installed
if [[ -f "/bin/mergerfs" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount"
else
# Build mergerfs binary
echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now."
mkdir -p /mnt/user/appdata/other/rclone/mergerfs
docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build
mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin
# check if mergerfs install successful
echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds"
sleep 5
if [[ -f "/bin/mergerfs" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount."
else
echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting."
rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
exit
fi
fi
# Create mergerfs mount
echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount."
# Extra Mergerfs folders
if [[ $LocalFilesShare2 != 'ignore' ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount."
LocalFilesShare2=":$LocalFilesShare2"
else
LocalFilesShare2=""
fi
if [[ $LocalFilesShare3 != 'ignore' ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount."
LocalFilesShare3=":$LocalFilesShare3"
else
LocalFilesShare3=""
fi
if [[ $LocalFilesShare4 != 'ignore' ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount."
LocalFilesShare4=":$LocalFilesShare4"
else
LocalFilesShare4=""
fi
# make sure mergerfs mount point is empty
mv $MergerFSMountLocation $LocalFilesLocation
mkdir -p $MergerFSMountLocation
# mergerfs mount command
mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
# check if mergerfs mount successful
echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created."
if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created."
else
echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers."
docker stop $DockerStart
rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
exit
fi
fi
fi
####### Starting Dockers That Need Mergerfs Mount To Work Properly #######
# only start dockers once
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: dockers already started."
else
# Check CA Appdata plugin not backing up or restoring
if [ -f "/tmp/ca.backup2/tempFiles/backupInProgress" ] || [ -f "/tmp/ca.backup2/tempFiles/restoreInProgress" ] ; then
echo "$(date "+%d.%m.%Y %T") INFO: Appdata Backup plugin running - not starting dockers."
else
touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started
echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."
docker start $DockerStart
fi
fi
rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
echo "$(date "+%d.%m.%Y %T") INFO: Script complete"
exit
FFmpeg logs:
/videos/792fbfb0-773b-9877-cf37-e314635b9a25/hls1/main/0.ts
{"Protocol":0,"Id":"792fbfb0773b9877cf37e314635b9a25","Path":"/data/mount_mergerfs/gdrive/media/tv/South Park/Season 23/South.Park.S23E07.TRUEFRENCH.1080p.BluRay.x264-FTMVHD.mkv","EncoderPath":null,"EncoderProtocol":null,"Type":0,"Container":"mkv,webm","Size":758642368,"Name":"South.Park.S23E07.TRUEFRENCH.1080p.BluRay.x264-FTMVHD","IsRemote":false,"ETag":"e1755aabe81c4e4e976ad2dce147dc6c","RunTimeTicks":12477440000,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"OpenToken":null,"RequiresClosing":false,"LiveStreamId":null,"BufferMs":null,"RequiresLooping":false,"SupportsProbing":true,"VideoType":0,"IsoType":null,"Video3DFormat":null,"MediaStreams":[{"Codec":"h264","CodecTag":null,"Language":"eng","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"1080p BluRay US (LATENCY)","VideoRange":"SDR","localizedUndefined":null,"localizedDefault":null,"localizedForced":null,"DisplayTitle":"1080p BluRay US (LATENCY) - H264 - SDR","NalLengthSize":"0","IsInterlaced":false,"IsAVC":false,"ChannelLayout":null,"BitRate":4864089,"BitDepth":8,"RefFrames":1,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":true,"IsForced":false,"Height":1080,"Width":1920,"AverageFrameRate":25,"RealFrameRate":25,"Profile":"High","Type":1,"AspectRatio":"16:9","Index":0,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":"yuv420p","Level":41,"IsAnamorphic":null},{"Codec":"ac3","CodecTag":null,"Language":"fre","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"AC3 2.0 Game One FR (FTMVHD)","VideoRange":null,"localizedUndefined":null,"localizedDefault":null,"localizedForced":null,"DisplayTitle":"AC3 2.0 Game One FR (FTMVHD) - Fre - Dolby Digital - Stereo - Default","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":"stereo","BitRate":192000,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":2,"SampleRate":48000,"IsDefault":true,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":0,"AspectRatio":null,"Index":1,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null},{"Codec":"ass","CodecTag":null,"Language":"fre","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"FTMV Te@m Since 2oo8 (FTMVHD)","VideoRange":null,"localizedUndefined":"Non d\u00E9fini","localizedDefault":"Par d\u00E9faut","localizedForced":"Forc\u00E9","DisplayTitle":"FTMV Te@m Since 2oo8 (FTMVHD) - Fre - Par d\u00E9faut - Forc\u00E9","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":null,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":true,"IsForced":true,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":2,"AspectRatio":null,"Index":2,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null}],"MediaAttachments":[],"Formats":[],"Bitrate":5056089,"Timestamp":null,"RequiredHttpHeaders":{},"TranscodingUrl":null,"TranscodingSubProtocol":null,"TranscodingContainer":null,"AnalyzeDurationMs":null,"DefaultAudioStreamIndex":null,"DefaultSubtitleStreamIndex":null}
/usr/lib/jellyfin-ffmpeg/ffmpeg -f matroska,webm -i file:"/data/mount_mergerfs/gdrive/media/tv/South Park/Season 23/South.Park.S23E07.TRUEFRENCH.1080p.BluRay.x264-FTMVHD.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -pix_fmt yuv420p -preset veryfast -crf 23 -maxrate 4864089 -bufsize 9728178 -profile:v:0 high -level 41 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -vf "scale=trunc(min(max(iw\,ih*dar)\,min(1920\,1080*dar))/2)*2:trunc(min(max(iw/dar\,ih)\,min(1920/dar\,1080))/2)*2" -start_at_zero -vsync -1 -codec:a:0 aac -ac 2 -ab 384000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/config/data/transcodes/fbd19e15648736fd258d918eb48b5e2a%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/config/data/transcodes/fbd19e15648736fd258d918eb48b5e2a.m3u8"
ffmpeg version 4.4.1-Jellyfin Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-static --enable-gmp --enable-gnutls --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --arch=amd64 --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-vdpau --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
[matroska,webm @ 0x557231c91280] Read error at pos. 0 (0x0)
[matroska,webm @ 0x557231c91280] EBML header parsing failed
file:/data/mount_mergerfs/gdrive/media/tv/South Park/Season 23/South.Park.S23E07.TRUEFRENCH.1080p.BluRay.x264-FTMVHD.mkv: Invalid data found when processing input
Thanks for anyone that could help me solve this issue!