Slow/stalling reads from mounted Unraid share

ThreePinkApples · August 7, 2021, 12:21pm

What is the problem you are having with rclone?

I'm using RClone to mount a share on my Unraid server. This is needed to make a certain backup application I'm using to actually backup the server.
The problem is that the read frequently stalls for long periods of time. When it is actually reading the transfer speed is fair, but it stops several times a day for extended periods (I haven't been able to measure this as I can't just sit and stare at task manager forever).
I'm assuming I'm doing something wrong with the vfs or cache settings, but I have no idea what that could be. Perhaps the cache being full and not being cleaned? The backup application also requires write access to work.
I've let it run for well over a week now, and I've been able to upload about 7TB out of 21
I've also noticed that just browsing the mount in Windows file explorer shows that it is slow to update if files are added/removed on the server. I would want this to behave as close as possible to a direct network mount.

What is your rclone version (output from `rclone version`)

v1.56.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

WIndows 10 21H1 64Bit

Which cloud storage system are you using? (eg Google Drive)

Local Unraid samba share

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

mount --allow-other --file-perms 0666 --dir-perms 0777 --dir-cache-time 72h --drive-chunk-size 64M --log-level INFO --vfs-read-chunk-size-limit off "\\TOWER\media" X: --config "rclone.conf" --vfs-cache-mode writes --volname Media

The rclone config contents with secrets removed.

[unraid]
type = alias
remote = local:\\TOWER\media

A log from the command with the `-vv` flag

2021/08/07 14:14:00 DEBUG : rclone: Version "v1.56.0" starting with parameters ["C:\\RClone\\rclone.exe" "mount" "--allow-other" "--file-perms" "0666" "--dir-perms" "0777" "--dir-cache-time" "72h" "--drive-chunk-size" "64M" "--vfs-read-chunk-size-limit" "off" "\\\\TOWER\\media" "X:" "--config" "C:\\Users\\inste\\.config\\rclone\\rclone.conf" "--vfs-cache-mode" "writes" "--volname" "Media" "-vv"]
2021/08/07 14:14:00 DEBUG : Creating backend with remote "\\\\TOWER\\media"
2021/08/07 14:14:00 NOTICE: Config file "C:\\Users\\inste\\.config\\rclone\\rclone.conf" not found - using defaults
2021/08/07 14:14:00 DEBUG : fs cache: renaming cache item "\\\\TOWER\\media" to be canonical "//?/UNC/TOWER/media"
2021/08/07 14:14:00 NOTICE: --allow-other flag does nothing on Windows
2021/08/07 14:14:00 INFO  : Local file system at //?/UNC/TOWER/media: poll-interval is not supported by this remote
2021/08/07 14:14:00 DEBUG : vfs cache: root is "\\\\?\\C:\\Users\\inste\\AppData\\Local\\rclone\\vfs\\local\\UNC\\TOWER\\media"
2021/08/07 14:14:00 DEBUG : vfs cache: metadata root is "\\\\?\\C:\\Users\\inste\\AppData\\Local\\rclone\\vfs\\local\\UNC\\TOWER\\media"
2021/08/07 14:14:00 DEBUG : Creating backend with remote "\\\\?\\C:\\Users\\inste\\AppData\\Local\\rclone\\vfs\\local\\UNC\\TOWER\\media"
2021/08/07 14:14:00 DEBUG : fs cache: renaming cache item "\\\\?\\C:\\Users\\inste\\AppData\\Local\\rclone\\vfs\\local\\UNC\\TOWER\\media" to be canonical "//?/C:/Users/inste/AppData/Local/rclone/vfs/local/UNC/TOWER/media"
2021/08/07 14:14:00 DEBUG : fs cache: switching user supplied name "\\\\?\\C:\\Users\\inste\\AppData\\Local\\rclone\\vfs\\local\\UNC\\TOWER\\media" for canonical name "//?/C:/Users/inste/AppData/Local/rclone/vfs/local/UNC/TOWER/media"
2021/08/07 14:14:00 DEBUG : Network mode mounting is disabled
2021/08/07 14:14:00 DEBUG : Mounting on "X:" ("Media")
2021/08/07 14:14:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/08/07 14:14:00 DEBUG : Local file system at //?/UNC/TOWER/media: Mounting with options: ["-o" "attr_timeout=1" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=Media"]
2021/08/07 14:14:00 DEBUG : Local file system at //?/UNC/TOWER/media: Init:
2021/08/07 14:14:00 DEBUG : Local file system at //?/UNC/TOWER/media: >Init:
2021/08/07 14:14:00 DEBUG : /: Statfs:
2021/08/07 14:14:00 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:9275983458 Bfree:3737087023 Bavail:3737087023 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/08/07 14:14:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:00 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:00 DEBUG : /: Readlink:
2021/08/07 14:14:00 DEBUG : /: >Readlink: linkPath="", errc=-40
The service rclone has been started.
2021/08/07 14:14:00 DEBUG : /: Statfs:
2021/08/07 14:14:00 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:9275983458 Bfree:3737087023 Bavail:3737087023 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/08/07 14:14:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:00 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:00 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:00 DEBUG : /: Opendir:
2021/08/07 14:14:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:00 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:00 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:00 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:14:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:00 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:00 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:00 DEBUG : /: Opendir:
2021/08/07 14:14:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:00 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:00 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:00 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:14:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:00 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:00 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:00 DEBUG : /: Opendir:
2021/08/07 14:14:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:00 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:00 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:00 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:14:03 DEBUG : /autorun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:03 DEBUG : /autorun.inf: >Getattr: errc=-2
2021/08/07 14:14:03 DEBUG : /autorun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:03 DEBUG : /autorun.inf: >Getattr: errc=-2
2021/08/07 14:14:03 DEBUG : /autorun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:03 DEBUG : /autorun.inf: >Getattr: errc=-2
2021/08/07 14:14:03 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:03 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:03 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:03 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:03 DEBUG : /: Opendir:
2021/08/07 14:14:03 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:03 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:03 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:03 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:03 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:14:03 DEBUG : /AutoRun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:03 DEBUG : /AutoRun.inf: >Getattr: errc=-2
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Opendir:
2021/08/07 14:14:09 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:09 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:09 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:09 DEBUG : /: Statfs:
2021/08/07 14:14:09 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:9275983458 Bfree:3737087023 Bavail:3737087023 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/08/07 14:14:09 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:09 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Opendir:
2021/08/07 14:14:09 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:09 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:09 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:09 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:09 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Opendir:
2021/08/07 14:14:09 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:09 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:09 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:09 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:09 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/08/07 14:14:09 DEBUG : /: >Getattr: errc=0
2021/08/07 14:14:09 DEBUG : /: Opendir:
2021/08/07 14:14:09 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/08/07 14:14:09 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/08/07 14:14:09 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/08/07 14:14:09 DEBUG : /: Releasedir: fh=0x0
2021/08/07 14:14:09 DEBUG : /: >Releasedir: errc=0
2021/08/07 14:15:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/08/07 14:16:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/08/07 14:17:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/08/07 14:18:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/08/07 14:19:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/08/07 14:20:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/08/07 14:21:00 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

asdffdsa · August 7, 2021, 12:31pm

hello and welcome to the forum,

NOTICE: --allow-other flag does nothing on Windows
that is from the log file, so that flag can be removed.
the top lines of the log file are missing?
what is the backup software?
sharing a rclone mount over smb can be slow, it has been discussed in the forum.
on the unraid server, instead of smb, setup a sftp server.
note that rclone can as a sftp server using rclone serve sftp /path/to/local/folder

ThreePinkApples · August 7, 2021, 12:36pm

Hi!
Update log with the missing line.
Software is Backblaze. I recently moved the hard drives out of my desktop computer and into the Unraid server, was not aware of the limitations of Backblaze personal license. Been using it for several years so I've tried several methods of tricking into backing up the server.

I'm going to test out SFTP, didn't think of that. Had been unsuccessful in googling for answers around SMB issues, I just end up with answers focusing on mounting Google Drive shares as a SMB

asdffdsa · August 7, 2021, 12:38pm

--drive-chunk-size, that is for gdrive,

for what it is worth, rclone has support for backblaze b2 and you can run rclone on the unraid server

oh, you are using the backblaze windows software.
the odds are that it does not support reading from sftp.

ThreePinkApples · August 7, 2021, 1:00pm

Made an attempt with FTP, it connects just fine, but Backblaze seem unable to actually start uploading. It just gives up while scanning files. Might be that it detects it being a network hosted drive for all I know.

New logs here: 2021/08/07 14:57:17 DEBUG : rclone: Version "v1.56.0" starting with parameters [ - Pastebin.com

asdffdsa · August 7, 2021, 3:16pm

the problem with ftp is that it does not support checksums, imho, cannot be trusted for backups.

not familiar with that windows software.

ThreePinkApples · August 7, 2021, 3:35pm

I believe Backblaze does its own checksumming. It also uploads files in small chunks so it can handle being abrupted and pick up mostly where it left off. I'll have to experiment more, but reverted back to my working (but slow) setup for now

asdffdsa · August 7, 2021, 3:43pm

the ftp data transfer between unraid and win cannot be validated, so if a packet was corrupted, then backblaze would checksum that corrupted data.
if sftp is not an option, then ftps is better than ftp.

system · October 7, 2021, 11:44am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.