I'm running in full
mode now, and might have hit another corner case, but it will take some work for me to replicate again. During a post-processing operation, a "No such file or directory" error was reported right after moving over the file, before the upload started.
I think the key piece here may be that the cache wouldn't accommodate the moved file without some purging happening. Limit is set to 40GB, here's the log including the post-processing entries where the move is started, rclone purging some files, the post processor unable to read the file it just moved, and the eventual completion of the file copy.
2020/06/19 12:29:11 INFO : Cleaned the cache: objects 107 (was 107) in use 1, to upload 0, uploading 0, total size 39.037G (was 39.020G)
2020/06/19 12:30:11 INFO : Cleaned the cache: objects 107 (was 107) in use 1, to upload 0, uploading 0, total size 39.055G (was 39.037G)
Jun 19 12:30:52 HOST CouchPotato.py[359]: 06-19 12:30:52 INFO [tato.core.plugins.renamer] Moving "/data/download/process/REDACTED.mkv" to "/data/media/REDACTED.mkv"
2020/06/19 12:31:11 INFO : REDACTED2.mp4: vfs cache: removed cache file as over quota
2020/06/19 12:31:11 INFO : REDACTED2.mp4: vfs cache: removed metadata from cache as over quota
2020/06/19 12:31:11 INFO : Cleaned the cache: objects 107 (was 108) in use 2, to upload 0, uploading 0, total size 39.496G (was 39.055G)
2020/06/19 12:32:11 INFO : REDACTED3.mp4: vfs cache: removed cache file as over quota
2020/06/19 12:32:11 INFO : REDACTED3.mp4: vfs cache: removed metadata from cache as over quota
2020/06/19 12:32:11 INFO : REDACTED4.mp4: vfs cache: removed cache file as over quota
2020/06/19 12:32:11 INFO : REDACTED4.mp4: vfs cache: removed metadata from cache as over quota
2020/06/19 12:32:11 INFO : REDACTED5.mp4: vfs cache: removed cache file as over quota
2020/06/19 12:32:11 INFO : REDACTED5.mp4: vfs cache: removed metadata from cache as over quota
2020/06/19 12:32:11 INFO : REDACTED6.mp4: vfs cache: removed cache file as over quota
2020/06/19 12:32:11 INFO : REDACTED6.mp4: vfs cache: removed metadata from cache as over quota
2020/06/19 12:32:11 INFO : Cleaned the cache: objects 103 (was 107) in use 2, to upload 0, uploading 0, total size 39.714G (was 39.496G)
Jun 19 12:32:38 HOST CouchPotato.py[359]: 06-19 12:32:38 ERROR [tato.core.plugins.renamer] Couldn't move file "/data/download/process/REDACTED.mkv" to "/data/media/REDACTED.mkv": Traceback (most recent call last):
Jun 19 12:32:38 HOST CouchPotato.py[359]: OSError: [Errno 2] No such file or directory: '/data/media/REDACTED.mkv'
2020/06/19 12:33:11 INFO : REDACTED7.mp4: vfs cache: removed cache file as over quota
2020/06/19 12:33:11 INFO : REDACTED7.mp4: vfs cache: removed metadata from cache as over quota
2020/06/19 12:33:11 INFO : REDACTED8.mp4: vfs cache: removed cache file as over quota
2020/06/19 12:33:11 INFO : REDACTED8.mp4: vfs cache: removed metadata from cache as over quota
2020/06/19 12:33:11 INFO : Cleaned the cache: objects 101 (was 103) in use 1, to upload 0, uploading 1, total size 39.803G (was 39.714G)
2020/06/19 12:34:11 INFO : Cleaned the cache: objects 101 (was 101) in use 1, to upload 0, uploading 1, total size 39.803G (was 39.803G)
2020/06/19 12:35:11 INFO : Cleaned the cache: objects 101 (was 101) in use 1, to upload 0, uploading 1, total size 39.803G (was 39.803G)
2020/06/19 12:36:11 INFO : Cleaned the cache: objects 101 (was 101) in use 1, to upload 0, uploading 1, total size 39.803G (was 39.803G)
2020/06/19 12:37:11 INFO : Cleaned the cache: objects 101 (was 101) in use 1, to upload 0, uploading 1, total size 39.803G (was 39.803G)
2020/06/19 12:38:11 INFO : Cleaned the cache: objects 101 (was 101) in use 1, to upload 0, uploading 1, total size 39.803G (was 39.803G)
2020/06/19 12:38:16 INFO : REDACTED.mkv: Copied (new)
2020/06/19 12:38:16 INFO : REDACTED.mkv: vfs cache: upload succeeded try #1
This happened 3 times actually, with all files in question being > 10GB, so extra time for file move/copy operations may have factored in to this. Is that enough to go on @ncw? I wasn't running with debug logging on at the time, and there were no errors at the INFO level. This was with build v1.52.1-120-gfcc32a46-vfs-beta and the following mount options:
--attr-timeout 0 \
--dir-cache-time 24h \
--poll-interval 15s \
--vfs-cache-mode full \
--vfs-cache-max-age 6h \
--vfs-cache-max-size 40G \
--fast-list \
--drive-skip-gdocs \
--drive-use-trash=false \
--drive-disable-http2=false \
--allow-other \
--umask 002 \
--gid 1001 \
--uid 1000 \
--rc \
--use-mmap \
--log-level INFO