Corrupted data streaming after vfs/Meta files removed

calisro · September 9, 2020, 1:27pm

What is the problem you are having with rclone?

If files are removed from the cache directory manually, rclone will then from there forward stream bad data. It doesn't correct itself from the file removal. This happens even if rclone is restarted. The only way to fully fix this issue is to fully shut rclone, clear the cached data, and then restart it. If you try to clear it while rclone is running it will continue to fail through a restart if that data was attempted to be accessed again. I'm sure this is because of memory cache.

Although it is not typical to remove files from the cached vfs/vfsMeta directories, I would expect that if that was done, rclone would be able to recover from that situation rather than stream bad data even through rclone restarts.

I haven't been able to reproduce this using rclone ONLY (rather than removing files from the backend) but I had my player last night not be able to play anything after a very large metadata export of data to the webdav mount and I did not remove anything directly from the cache manually when that happened. I'm not sure if that had something to do with the new multithreaded behavior of the cache or not but this test case seems to reproduce the symptoms using ffprobe (which is what the player uses before it attempts to play).

What is your rclone version (output from `rclone version`)

rclone v1.54.0-beta.4765.233bed6a7

os/arch: linux/amd64
go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

linux 64

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)


root@s163042:/data/bin# rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full &
[1] 26202
root@s163042:/data/bin# 
root@s163042:/data/bin# 
root@s163042:/data/bin# 
root@s163042:/data/bin# 2020/09/09 09:08:44 NOTICE: Encrypted drive 'robgs-cryptp:Media/Videos/Series/Westworld/Season.3/': WebDav Server started on http://127.0.0.1:1111/

root@s163042:/data/bin# find /tmp/rclonetest
/tmp/rclonetest
/tmp/rclonetest/vfsMeta
/tmp/rclonetest/vfsMeta/robgs-cryptp
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
/tmp/rclonetest/vfs
/tmp/rclonetest/vfs/robgs-cryptp
/tmp/rclonetest/vfs/robgs-cryptp/Media
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
root@s163042:/data/bin# ffprobe http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv
ffprobe version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, matroska,webm, from 'http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv':
  Metadata:
    encoder         : libebml v1.3.10 + libmatroska v1.5.2
    creation_time   : 2020-03-16T01:12:25.000000Z
  Duration: 01:11:40.32, start: 0.000000, bitrate: 5017 kb/s
    Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)
    Metadata:
      BPS-eng         : 4375484
      DURATION-eng    : 01:11:40.296000000
      NUMBER_OF_FRAMES-eng: 103104
      NUMBER_OF_BYTES-eng: 2351985067
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:1(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 640 kb/s (default)
    Metadata:
      title           : English
      BPS-eng         : 640000
      DURATION-eng    : 01:11:40.288000000
      NUMBER_OF_FRAMES-eng: 134384
      NUMBER_OF_BYTES-eng: 344023040
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:2(eng): Subtitle: subrip
    Metadata:
      title           : English
      BPS-eng         : 45
      DURATION-eng    : 01:11:37.833000000
      NUMBER_OF_FRAMES-eng: 801
      NUMBER_OF_BYTES-eng: 24284
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:3(eng): Subtitle: subrip
    Metadata:
      title           : English [SDH]
      BPS-eng         : 51
      DURATION-eng    : 01:11:37.833000000
      NUMBER_OF_FRAMES-eng: 871
      NUMBER_OF_BYTES-eng: 27815
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
root@s163042:/data/bin# find /tmp/rclonetest
/tmp/rclonetest
/tmp/rclonetest/vfsMeta
/tmp/rclonetest/vfsMeta/robgs-cryptp
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
/tmp/rclonetest/vfs
/tmp/rclonetest/vfs/robgs-cryptp
/tmp/rclonetest/vfs/robgs-cryptp/Media
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
root@s163042:/data/bin# rm /tmp/rclonetest/*/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
root@s163042:/data/bin# ffprobe http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv
ffprobe version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[matroska,webm @ 0x55fe0d91e080] Read error at pos. 1 (0x1)
[matroska,webm @ 0x55fe0d91e080] EBML header parsing failed
http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv: Invalid data found when processing input
root@s163042:/data/bin# fg
rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full
^C
root@s163042:/data/bin# rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full &
[1] 27153
root@s163042:/data/bin# 
root@s163042:/data/bin# 2020/09/09 09:10:35 NOTICE: Encrypted drive 'robgs-cryptp:Media/Videos/Series/Westworld/Season.3/': WebDav Server started on http://127.0.0.1:1111/

root@s163042:/data/bin# ffprobe http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv
ffprobe version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[matroska,webm @ 0x55f326258080] Read error at pos. 1 (0x1)
[matroska,webm @ 0x55f326258080] EBML header parsing failed
http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv: Invalid data found when processing input
root@s163042:/data/bin# find /tmp/rclonetest
/tmp/rclonetest
/tmp/rclonetest/vfsMeta
/tmp/rclonetest/vfsMeta/robgs-cryptp
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
/tmp/rclonetest/vfs
/tmp/rclonetest/vfs/robgs-cryptp
/tmp/rclonetest/vfs/robgs-cryptp/Media
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
root@s163042:/data/bin# cat /tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
{
	"ModTime": "2020-09-09T09:09:39.625444-04:00",
	"ATime": "2020-09-09T09:10:39.67216989-04:00",
	"Size": 2697148405,
	"Rs": [
		{
			"Pos": 0,
			"Size": 19918848
		},
		{
			"Pos": 2697146887,
			"Size": 1518
		}
	],
	"Fingerprint": "2697148405,2020-03-16 02:41:49.909 +0000 UTC",
	"Dirty": false
}
root@s163042:/data/bin# rm /tmp/rclonetest/*/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
root@s163042:/data/bin# ffprobe http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv
ffprobe version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[matroska,webm @ 0x55c42c9f0080] Read error at pos. 1 (0x1)
[matroska,webm @ 0x55c42c9f0080] EBML header parsing failed
http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv: Invalid data found when processing input
root@s163042:/data/bin# fg
rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full
^C
root@s163042:/data/bin# find /tmp/rclonetest
/tmp/rclonetest
/tmp/rclonetest/vfsMeta
/tmp/rclonetest/vfsMeta/robgs-cryptp
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
/tmp/rclonetest/vfsMeta/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
/tmp/rclonetest/vfs
/tmp/rclonetest/vfs/robgs-cryptp
/tmp/rclonetest/vfs/robgs-cryptp/Media
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3
/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
root@s163042:/data/bin# rm /tmp/rclonetest/*/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
root@s163042:/data/bin# rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full &
[1] 27760
root@s163042:/data/bin# find /tmp/rclonetest2020/09/09 09:11:49 NOTICE: Encrypted drive 'robgs-cryptp:Media/Videos/Series/Westworld/Season.3/': WebDav Server started on http://127.0.0.1:1111/
fprobe http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv
ffprobe version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, matroska,webm, from 'http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv':
  Metadata:
    encoder         : libebml v1.3.10 + libmatroska v1.5.2
    creation_time   : 2020-03-16T01:12:25.000000Z
  Duration: 01:11:40.32, start: 0.000000, bitrate: 5017 kb/s
    Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)
    Metadata:
      BPS-eng         : 4375484
      DURATION-eng    : 01:11:40.296000000
      NUMBER_OF_FRAMES-eng: 103104
      NUMBER_OF_BYTES-eng: 2351985067
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:1(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 640 kb/s (default)
    Metadata:
      title           : English
      BPS-eng         : 640000
      DURATION-eng    : 01:11:40.288000000
      NUMBER_OF_FRAMES-eng: 134384
      NUMBER_OF_BYTES-eng: 344023040
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:2(eng): Subtitle: subrip
    Metadata:
      title           : English
      BPS-eng         : 45
      DURATION-eng    : 01:11:37.833000000
      NUMBER_OF_FRAMES-eng: 801
      NUMBER_OF_BYTES-eng: 24284
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Stream #0:3(eng): Subtitle: subrip
    Metadata:
      title           : English [SDH]
      BPS-eng         : 51
      DURATION-eng    : 01:11:37.833000000
      NUMBER_OF_FRAMES-eng: 871
      NUMBER_OF_BYTES-eng: 27815
      _STATISTICS_WRITING_APP-eng: mkvmerge v42.0.0 ('Overtime') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-03-16 01:12:25
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES





root@s163042:/data/bin# rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full -vv --log-file=out.log &
[1] 8645
root@s163042:/data/bin# 
root@s163042:/data/bin# 
root@s163042:/data/bin# ffprobe http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv
ffprobe version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[matroska,webm @ 0x5593447bf080] Read error at pos. 1 (0x1)
[matroska,webm @ 0x5593447bf080] EBML header parsing failed
http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv: Invalid data found when processing input
root@s163042:/data/bin# curl http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv > bad
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2572M  100 2572M    0     0  61.9M      0  0:00:41  0:00:41 --:--:-- 45.4M
root@s163042:/data/bin# fg                                                                
rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full -vv --log-file=out.log
^C
root@s163042:/data/bin# rm /tmp/rclonetest/*/robgs-cryptp/Media/Videos/Series/Westworld/Season.3/Westworld.S03E01.720p.webdl.h264.mkv
root@s163042:/data/bin# rclone serve webdav robgs-cryptp:Media/Videos/Series/Westworld/Season.3/ --addr localhost:1111 --cache-dir /tmp/rclonetest --vfs-cache-max-age 30000h --vfs-cache-mode full -vv --log-file=out.log &
[1] 9503
root@s163042:/data/bin# curl http://localhost:1111/Westworld.S03E01.720p.webdl.h264.mkv > good
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2572M  100 2572M    0     0  50.5M      0  0:00:50  0:00:50 --:--:-- 31.4M
root@s163042:/data/bin# diff bad good
Binary files bad and good differ

cmp bad good
bad good differ: byte 1, line 1
root@s163042:/data/bin# head -c 10 good 
Eߣ�B��Broot@s163042:/data/bin# head -c 10 bad 
root@s163042:/data/bin#

The rclone config contents with secrets removed.

Not relevant here.  It is a google drive remote but the remote isn't part of the issue.  It would happen with any remote including a local one.

A log from the command with the `-vv` flag

2020/09/09 09:13:40 DEBUG : rclone: Version "v1.54.0-beta.4765.233bed6a7" starting with parameters ["rclone" "serve" "webdav" "robgs-cryptp:Media/Videos/Series/Westworld/Season.3/" "--addr" "localhost:1111" "--cache-dir" "/tmp/rclonetest" "--vfs-cache-max-age" "30000h" "--vfs-cache-mode" "full" "-vv" "--log-file=out.log"]
2020/09/09 09:13:40 DEBUG : Using config file from "/root/.rclone.conf"
2020/09/09 09:13:40 DEBUG : Creating backend with remote "robgs-cryptp:Media/Videos/Series/Westworld/Season.3/"
2020/09/09 09:13:40 DEBUG : Creating backend with remote "robgs:cloudp/lp08934q11u6a5gslo2ddds5b4/hkitmqcr303hk1a33m19ttdq24/nhcij8dbshqodics7ddemvfbok/djnongmgohg5h2quccdhajavlg/bk5uildi65trven0k7k9keplik"
2020/09/09 09:13:41 DEBUG : vfs cache: root is "/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3"
2020/09/09 09:13:41 DEBUG : vfs cache: metadata root is "/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3"
2020/09/09 09:13:41 DEBUG : Creating backend with remote "/tmp/rclonetest/vfs/robgs-cryptp/Media/Videos/Series/Westworld/Season.3"
2020/09/09 09:13:41 DEBUG : vfs cache RemoveNotInUse (maxAge=108000000000000000, emptyOnly=false): item Westworld.S03E01.720p.webdl.h264.mkv not removed, freed 0 bytes
2020/09/09 09:13:41 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 15.025M (was 0)
2020/09/09 09:13:41 NOTICE: Encrypted drive 'robgs-cryptp:Media/Videos/Series/Westworld/Season.3/': WebDav Server started on http://127.0.0.1:1111/
2020/09/09 09:14:00 DEBUG : /Westworld.S03E01.720p.webdl.h264.mkv: OpenFile: flags=O_RDONLY, perm=----------
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: Open: flags=O_RDONLY
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: newRWFileHandle: 
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: >newRWFileHandle: err=<nil>
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: >Open: fd=Westworld.S03E01.720p.webdl.h264.mkv (rw), err=<nil>
2020/09/09 09:14:01 DEBUG : /Westworld.S03E01.720p.webdl.h264.mkv: >OpenFile: fd=Westworld.S03E01.720p.webdl.h264.mkv (rw), err=<nil>
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): openPending: 
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: vfs cache: checking remote fingerprint "2697148405,2020-03-16 02:41:49.909 +0000 UTC" against cached fingerprint "2697148405,2020-03-16 02:41:49.909 +0000 UTC"
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: vfs cache: truncate to size=2697148405
2020/09/09 09:14:01 DEBUG : : Added virtual directory entry vAdd: "Westworld.S03E01.720p.webdl.h264.mkv"
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): >openPending: err=<nil>
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): _readAt: size=32768, off=0
2020/09/09 09:14:01 DEBUG : vfs cache: looking for range={Pos:0 Size:32768} in [{Pos:0 Size:15753216} {Pos:2697146887 Size:1518}] - present true
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: ChunkedReader.RangeSeek from -1 to 15753216 length -1
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): >_readAt: n=32768, err=<nil>
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: ChunkedReader.Read at -1 length 4096 chunkOffset 15753216 chunkSize 134217728
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: ChunkedReader.openRange at 15753216 length 134217728
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): _readAt: size=32768, off=32768
2020/09/09 09:14:01 DEBUG : vfs cache: looking for range={Pos:32768 Size:32768} in [{Pos:0 Size:15753216} {Pos:2697146887 Size:1518}] - present true
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): >_readAt: n=32768, err=<nil>
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): _readAt: size=32768, off=65536
2020/09/09 09:14:01 DEBUG : vfs cache: looking for range={Pos:65536 Size:32768} in [{Pos:0 Size:15753216} {Pos:2697146887 Size:1518}] - present true
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): >_readAt: n=32768, err=<nil>
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): _readAt: size=32768, off=98304
2020/09/09 09:14:01 DEBUG : vfs cache: looking for range={Pos:98304 Size:32768} in [{Pos:0 Size:15753216} {Pos:2697146887 Size:1518}] - present true
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): >_readAt: n=32768, err=<nil>
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): close: 
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv: vfs cache: setting modification time to 2020-03-16 02:41:49.909 +0000 UTC
2020/09/09 09:14:01 DEBUG : Westworld.S03E01.720p.webdl.h264.mkv(0xc001b81940): >close: err=<nil>
2020/09/09 09:14:01 INFO  : /Westworld.S03E01.720p.webdl.h264.mkv: GET from 127.0.0.1:53108

ncw · September 9, 2020, 1:51pm

I managed to replicate this with a mount

rclone mount drive:test --vfs-cache-mode full -vv /mnt/tmp/

$ cat /mnt/tmp/2001-05-28-115008.jpg > /tmp/0
$ rm ~/.cache/rclone/vfs*/drive/test/2001-05-28-115008.jpg 
$ cat /mnt/tmp/2001-05-28-115008.jpg > /tmp/1
$ md5sum /tmp/0 /tmp/1
65d5cf63ee5b41f7fbc879b37d631fbe  /tmp/0
ca7e7f427c4d5f92409cf8c7c77db911  /tmp/1
$ hd /tmp/1
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00084575
$

So what happens is that rclone doesn't notice the file is missing and re-creates it from scratch

2020/09/09 14:39:09 DEBUG : /: Lookup: name="2001-05-28-115008.jpg"
2020/09/09 14:39:09 DEBUG : /: >Lookup: node=2001-05-28-115008.jpg, err=<nil>
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: Attr: 
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: >Attr: a=valid=1s ino=0 size=542069 mode=-rw-rw-r--, err=<nil>
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: Open: flags=OpenReadOnly
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: Open: flags=O_RDONLY
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: newRWFileHandle: 
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: >newRWFileHandle: err=<nil>
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: >Open: fd=2001-05-28-115008.jpg (rw), err=<nil>
2020/09/09 14:39:09 DEBUG : 2001-05-28-115008.jpg: >Open: fh=&{2001-05-28-115008.jpg (rw)}, err=<nil>
2020/09/09 14:39:09 DEBUG : &{2001-05-28-115008.jpg (rw)}: Read: len=131072, offset=0
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: Read: len=131072, offset=131072
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): _readAt: size=131072, off=0
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): openPending: 
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg: vfs cache: checking remote fingerprint "542069,2001-05-28 10:50:08 +0000 UTC,65d5cf63ee5b41f7fbc879b37d631fbe" against cached fingerprint "542069,2001-05-28 10:50:08 +0000 UTC,65d5cf63ee5b41f7fbc879b37d631fbe"

This is the line which creates the file full of 0s

2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg: vfs cache: truncate to size=542069

2020/09/09 14:39:10 DEBUG : : Added virtual directory entry vAdd: "2001-05-28-115008.jpg"
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): >openPending: err=<nil>
2020/09/09 14:39:10 DEBUG : vfs cache: looking for range={Pos:0 Size:131072} in [{Pos:0 Size:542069}] - present true
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): _readAt: size=131072, off=131072
2020/09/09 14:39:10 DEBUG : vfs cache: looking for range={Pos:131072 Size:131072} in [{Pos:0 Size:542069}] - present true
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): >_readAt: n=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: >Read: read=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): >_readAt: n=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: >Read: read=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: Read: len=131072, offset=393216
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: Read: len=131072, offset=262144
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): _readAt: size=131072, off=393216
2020/09/09 14:39:10 DEBUG : vfs cache: looking for range={Pos:393216 Size:131072} in [{Pos:0 Size:542069}] - present true
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): _readAt: size=131072, off=262144
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: Read: len=20480, offset=524288
2020/09/09 14:39:10 DEBUG : vfs cache: looking for range={Pos:262144 Size:131072} in [{Pos:0 Size:542069}] - present true
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): _readAt: size=20480, off=524288
2020/09/09 14:39:10 DEBUG : vfs cache: looking for range={Pos:524288 Size:17781} in [{Pos:0 Size:542069}] - present true
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): >_readAt: n=17781, err=EOF
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: >Read: read=17781, err=<nil>
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): >_readAt: n=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: >Read: read=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): >_readAt: n=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: >Read: read=131072, err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: Flush: 
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): RWFileHandle.Flush
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: >Flush: err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: Release: 
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): RWFileHandle.Release
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): close: 
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg: vfs cache: setting modification time to 2001-05-28 10:50:08 +0000 UTC
2020/09/09 14:39:10 DEBUG : 2001-05-28-115008.jpg(0xc000272100): >close: err=<nil>
2020/09/09 14:39:10 DEBUG : &{2001-05-28-115008.jpg (rw)}: >Release: err=<nil>

What do you think rclone should do here?

I could maybe check to see if the file exists or not and throw an error if it doesn't exist when truncating it. I'm not 100% sure that won't cause a problem elsewhere but I can give it a go.

If you'd have removed the files with rclone stopped it would have reacted properly.

calisro · September 9, 2020, 1:59pm

Thing is, if it recreates if from scratch, why would it preserve the zeros as if its 'good' data. If it had to recreate it, it should know its zeros and that its not good 'real' data.

If the files are removed, I would have thought rclone would have flagged it to redownload the file. What I see happening is the file is filling with zeros, rclone is then assuming it is 'good' data in there and streaming it. If the vfs file is missing, it really should wipe that file/meta and regrab from the source?

Yes I agree there. If it was used 'properly' my test case wouldn't have happened. I have two things Im trying to do here.

I had a VERY strange thing happen last night when my player couldn't play anything. I did a full export of all meta files in the library to the webdav. What the does behind the covers is do HEADs on files to check if they exist and replace some files (nfo/jpgs) with fresh data. When this was done, the vfs cache was corrupt. I couldn't play anything. I manually went and figured out that if I shut rclone, wipe cache, and start it again everything worked. But to that end, it broke something which is bad. In that scenario, nothing was manually removed from the cache. I can't recreate this but I will try a new export and see if it happens again....

**I will say that I am not 100% sure if something else happened though. I may be chasing a unicorn. I'll see what happens when I redo the export.

I would like the ability to manually prune things from the cache by deleting the file and meta for certain file types. If rclone checks if those files are gone and re-populates them when accessed, then I can do that. The way it is now, if that is done, it streams a partially empty file. I have specific logic that I'd like to implement to keep certain files around and that would be very niche to add to rclone itself while on the filesystem it is rather easy to implement by deleting files.

ncw · September 11, 2020, 3:39pm

I agree that this is undesirable behaviour and it shouldn't be too difficult to fix...

I've made an attempt here.

You'll get an error like this

2020/09/11 16:25:07 ERROR : 2001-05-28-115008.jpg: vfs cache: detected external removal of cache file

Can you see if it fixes your situation? You should be able to delete files from the cache while rclone is running. If rclone has the file open then it will likely keep using it even though it is deleted. If not then this code should kick in.

Note that you don't need to remove the Meta files as they are small and if the main file is missing when rclone restarts then rclone will delete it.

v1.54.0-beta.4769.d9e9b0621.fix-vfs-cache-rm on branch fix-vfs-cache-rm (uploaded in 15-30 mins)

calisro · September 11, 2020, 5:02pm

Yes. that appears to be working better.Thanks for handling that!

system · September 14, 2020, 5:02pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.