Corrupted(md5) on transfer from webdav server to rclone mount using: vfs-cache-mode full

Suspected Bug
1

What is the problem you are having with rclone?

  • When using the webdav backend with --vfs-cache-mode full, downloaded files are corrupted, usually at the end of the file, this does not happen when uploading files. (image attached at the end of the post)

  • I have already tested with direct access to webdav, and the file is without problems.

  • I have already tested mounting the webdav directly in AirExplorer via AirLiveDrive, and the file is without problems.

  • I have already tested without any rclone flag, and it is without problems.

  • I have already tested changing to --vfs-cache-mode off and the file is also without problems.

  • The problem is only downloading files when using --vfs-cache-mode full.

  • I have not tested in other backends, only in webdav, probably everything should be normal in other backends.

  • I tested with an older version in the logs, but I also used the current one, and the problem did not change, I put the 2 versions used.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0-DEV

  • os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
  • os/kernel: 10.0.22631.3958 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.22.0
  • go/linking: dynamic
  • go/tags: cmount

or

rclone v1.67.0

  • os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
  • os/kernel: 10.0.22631.3958 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.22.4
  • go/linking: static
  • go/tags: cmount

I tested with other versions and they all have the same problem, but I'm on the most current version.

Which cloud storage system are you using? (eg Google Drive)

Webdav

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe mount ^
--config="rclone_mod2.conf" ^
-vv ^
 ^
--cache-dir="K:\RClone_Cache" ^
--cache-tmp-upload-path="K:\RClone_Cache\temp\BACKUP\upload" ^
--cache-tmp-wait-time 1m ^
--cache-chunk-path="K:\RClone_Cache\temp\BACKUP\chunks" ^
--cache-db-path="K:\RClone_Cache\temp\BACKUP\db" ^
--temp-dir="K:\RClone_Cache\temp_dir" ^
 ^
 ^
--backup-dir="K:\RClone_Cache\backup_dir" ^
--fs-cache-expire-duration=99999h ^
--fs-cache-expire-interval=0 ^
--ignore-checksum ^
 ^
 ^
--vfs-cache-mode full ^
--vfs-cache-max-age 10m ^
--vfs-cache-max-size 490G ^
 ^
 ^
--vfs-read-ahead=1M ^
--buffer-size 0 ^
 ^
 ^
--use-server-modtime ^
--no-modtime ^
--no-checksum ^
--no-gzip-encoding ^
--vfs-fast-fingerprint ^
 ^
 ^
--webdav-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,LeftSpace,LeftCrLfHtVt" ^
 ^
 ^
--exclude="**/.inProgress/**" ^
--exclude="**/.stfolder/**" ^
--exclude=".tmp" ^
--exclude=".partial" ^
--exclude=".stignore" ^
 ^
--network-mode ^
 ^
--volname="Usenet_RClone" ^
--stats 5s ^
--stats-one-line ^
--progress ^
 ^
 ^
--rc ^
--rc-addr 127.0.0.1:6572 ^
 ^
--attr-timeout=99999h ^
--dir-cache-time=99999h ^
--cache-info-age=99999h ^
usenet:/batman P:\
pause

The rclone config contents with secrets removed.

[usenet]
type = webdav
url  = http://user:pass@localhost:8000
vendor = other

ps: no problem with file from webdav direct access.

A log from the command with the -vv flag

Log using --vfs-cache-mode full (with bug)

Log using --vfs-cache-mode off (without hex bug)

Hex Compare shows its using --vfs-cache-mode full

image
image1920×1080 336 KB

Could you give some urgency to this case since it affects all webdav backend users who use --vfs-cache-mode full as well?

@ncw @Animosity022
Thank you for your attention.

2

This is ancient (and deprecated ages ago) caching mechanism replaced by VFS nowadays. Do not mixed both as results can be unpredictable. Remove it and try again.

3

I removed it, nothing changed. :confused:

5

hi,
please, let's try not post logs for some custom compiled rclone, only use latest stable, ok?
cannot spend time looking into those logs.
rclone: Version "v1.66.0-DEV" starting with parameters ["rclone_mod2.exe

if i understand your "bug", so far, not able to reproduce

rclone config redacted webdav:
[webdav]
type = webdav
vendor = nextcloud
user = XXX
pass = XXX
url = https://XXX.your-storageshare.de/remote.php/dav/files/admin

# mount command and snippet
rclone: Version "v1.67.0" starting with parameters ["C:\\data\\rclone\\rclone" "mount" "webdav:" "b:\\rclone\\mount\\webdav" "--config=C:\\data\\rclone\\rclone.conf" "--inplace" "--vfs-cache-mode=full" "--dir-cache-time=1s" "--vfs-cache-max-age=1m" "--vfs-cache-poll-interval=1m" "--cache-dir=d:\\rclone\\cache\\webdav" "--log-level=DEBUG" "--log-file=.\\\\log.mount.webdav.txt"]
DEBUG : test/file.ext: sha1 = 9c83efedc998b3334d48b8754a86475de9328bf8 OK

# copy/upload file.ext from local to mount
rclone copy d:\files\tmp\src   b:\rclone\mount\webdav\test -vv --inplace

# copy/download file.ext from mount to local
rclone copy b:\rclone\mount\webdav\test d:\files\tmp\dst   -vv --inplace

# use rclone to download and calculate the checksum of the file in webdav server itself
rclone hashsum sha1 webdav:test --download 
9c83efedc998b3334d48b8754a86475de9328bf8  file.ext

# use rclone to calculate checksum of local source file and local dest file
rclone hashsum sha1 d:\files\tmp
9c83efedc998b3334d48b8754a86475de9328bf8  src/file.ext
9c83efedc998b3334d48b8754a86475de9328bf8  dst/file.ext

# use double commander to calculate checksum of local source file and local dest file
9c83efedc998b3334d48b8754a86475de9328bf8 *tmp\src\file.ext
9c83efedc998b3334d48b8754a86475de9328bf8 *tmp\dst\file.ext
1 Like
6

I did something I hadn't tried before, I copied the file from mount to a folder in Windows, and checked the hash/hex, and it was perfect, I looked at the rclone log and it didn't return err=EOF.

When I copy the file it comes out perfect, when I open it with a program like Beyond Compare, it comes out corrupted.

7

fwiw, as mentioned, your mount command is way too complex, has deprecated flags, flags that do nothing on a mount, flags that that are rarely used.
makes it hard to debug.

so, start over, with the absolute simplest command possible, as few flags as possible, as many defaults as possible.
then post the top thirty lines of a debug log.

what is K:, local, network drive, usbdrive or what?
what is the file system?

might try --local-no-preallocate --local-no-sparse

what about file sizes?

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.