Problem Opening Documents - Rclone Mount

matgas23 · May 11, 2020, 10:48am

What is the problem you are having with rclone?

Hi everyone! I notice a problem with "rclone mount".
Opening an office file (.docx, .xlsx....) or a pdf takes a long buffering time. Some minutes or more for a 16kb file... Instead streaming video work perfectly even with large files.
I noticed that i can open office file created with GoogleDocuments but it doesn't work with files created with Microsoft Office and aren't present on cache.
Thanks for any clarifications.

What is your rclone version (output from `rclone version`)

rclone v1.51.0

os/arch: windows/amd64
go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Recently I used this configuration.

mount --allow-other --dir-cache-time 72h --drive-chunk-size 64M --log-level INFO --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit off GoogleDrive: X: --vfs-cache-mode writes

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

2020/05/11 12:31:08 DEBUG : rclone: Version "v1.51.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.51.0-windows-amd64\\rclone.exe" "mount" "--allow-other" "--dir-cache-time" "72h" "--drive-chunk-size" "64M" "--vfs-read-chunk-size" "32M" "--vfs-read-chunk-size-limit" "off" "GoogleDrive:" "X:" "--vfs-cache-mode" "writes" "-vv"]
2020/05/11 12:31:08 DEBUG : Using config file from "C:\\Users\\matte\\.config\\rclone\\rclone.conf"
2020/05/11 12:31:08 DEBUG : Google drive root '': Mounting on "X:"
2020/05/11 12:31:08 DEBUG : vfs cache root is "C:\\Users\\matte\\AppData\\Local\\rclone\\vfs\\GoogleDrive"
2020/05/11 12:31:08 DEBUG : Adding path "vfs/forget" to remote control registry
2020/05/11 12:31:08 DEBUG : Adding path "vfs/refresh" to remote control registry
2020/05/11 12:31:08 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2020/05/11 12:31:08 DEBUG : Google drive root '': Mounting with options: ["-o" "fsname=GoogleDrive:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "attr_timeout=1" "-o" "atomic_o_trunc" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=GoogleDrive" "-o" "allow_other"]
2020/05/11 12:31:08 DEBUG : Google drive root '': Init:
2020/05/11 12:31:08 DEBUG : Google drive root '': >Init:
2020/05/11 12:31:08 DEBUG : /: Statfs:
2020/05/11 12:31:09 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:3932160 Bfree:3107103 Bavail:3025253 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2020/05/11 12:31:09 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2020/05/11 12:31:09 DEBUG : /: >Getattr: errc=0
2020/05/11 12:31:09 DEBUG : /: Readlink:
2020/05/11 12:31:09 DEBUG : /: >Readlink: linkPath="", errc=-40
The service rclone has been started.
2020/05/11 12:31:09 DEBUG : /: Statfs:
2020/05/11 12:31:09 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:3932160 Bfree:3107103 Bavail:3025253 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0

...

2020/05/11 12:31:16 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.Read at -1 length 4096 chunkOffset 0 chunkSize 33554432
2020/05/11 12:31:16 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.openRange at 0 length 33554432
2020/05/11 12:31:17 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: >Read: n=4096
2020/05/11 12:31:17 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: Getattr: fh=0x0
2020/05/11 12:31:17 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: >Getattr: errc=0
2020/05/11 12:31:17 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: Read: ofst=109078, fh=0x0
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ReadFileHandle.seek from 4096 to 109078 (fs.RangeSeeker)
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.RangeSeek from 4096 to 109078 length -1
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.Read at -1 length 4096 chunkOffset 109078 chunkSize 33554432
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.openRange at 109078 length 33554432
2020/05/11 12:31:17 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2020/05/11 12:31:17 DEBUG : /: >Getattr: errc=0
2020/05/11 12:31:17 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2020/05/11 12:31:17 DEBUG : /: >Getattr: errc=0
2020/05/11 12:31:17 DEBUG : /: Opendir:
2020/05/11 12:31:17 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2020/05/11 12:31:17 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2020/05/11 12:31:17 DEBUG : /: >Opendir: errc=0, fh=0x1
2020/05/11 12:31:17 DEBUG : /: Statfs:
2020/05/11 12:31:17 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:3932160 Bfree:3107103 Bavail:3025253 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2020/05/11 12:31:17 DEBUG : /: Releasedir: fh=0x1
2020/05/11 12:31:17 DEBUG : /: >Releasedir: errc=0
2020/05/11 12:31:17 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: >Read: n=22
2020/05/11 12:31:17 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: Read: ofst=107386, fh=0x0
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ReadFileHandle.seek from 109100 to 107386 (fs.RangeSeeker)
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.RangeSeek from 109100 to 107386 length -1
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.Read at -1 length 4096 chunkOffset 107386 chunkSize 33554432
2020/05/11 12:31:17 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.openRange at 107386 length 33554432
2020/05/11 12:31:19 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: >Read: n=1714
2020/05/11 12:31:19 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: Read: ofst=107390, fh=0x0
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ReadFileHandle.seek from 109100 to 107390 (fs.RangeSeeker)
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.RangeSeek from 109100 to 107390 length -1
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.Read at -1 length 4096 chunkOffset 107390 chunkSize 33554432
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.openRange at 107390 length 33554432
2020/05/11 12:31:19 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: >Read: n=1710
2020/05/11 12:31:19 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: Read: ofst=107432, fh=0x0
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ReadFileHandle.seek from 109100 to 107432 (fs.RangeSeeker)
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.RangeSeek from 109100 to 107432 length -1
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.Read at -1 length 4096 chunkOffset 107432 chunkSize 33554432
2020/05/11 12:31:19 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.openRange at 107432 length 33554432
2020/05/11 12:31:20 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: >Read: n=1668
2020/05/11 12:31:20 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: Read: ofst=107451, fh=0x0
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ReadFileHandle.seek from 109100 to 107451 (fs.RangeSeeker)
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.RangeSeek from 109100 to 107451 length -1
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.Read at -1 length 4096 chunkOffset 107451 chunkSize 33554432
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.openRange at 107451 length 33554432
2020/05/11 12:31:20 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: >Read: n=1649
2020/05/11 12:31:20 DEBUG : /Scuola/Quarantena2020/Inglese/Orwell1504.docx: Read: ofst=107455, fh=0x0
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ReadFileHandle.seek from 109100 to 107455 (fs.RangeSeeker)
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.RangeSeek from 109100 to 107455 length -1
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.Read at -1 length 4096 chunkOffset 107455 chunkSize 33554432
2020/05/11 12:31:20 DEBUG : Scuola/Quarantena2020/Inglese/Orwell1504.docx: ChunkedReader.openRange at 107455 length 33554432

asdffdsa · May 11, 2020, 1:30pm

hello and welcome to forum,

--allow-other does nothing on windows, so you can remove it.

i would remove most of the flags and test again
is there a reason for this --vfs-read-chunk-size 32M

mount GoogleDrive: X: --vfs-cache-mode writes --log-level INFO

matgas23 · May 11, 2020, 2:45pm

Thanks for your reply.
I tried with mount GoogleDrive: X: --vfs-cache-mode writes --log-level INFO but it doesn't reduce buffering. However there is no particular reason for --vfs-read-chunk-size 32M (it was just a test)

calisro · May 11, 2020, 2:51pm

It won't do very well in this use case without a cache. It has to open and read pieces of that file over and over to get what the application needs. A cache would help since then the file would be local. They are working to make VFS cache work better and that would probably fix this but until then the unmaintained cache backend might serve you better for this use case. Its essentially trying to do scattered reads on your file.

asdffdsa · May 11, 2020, 2:54pm

i have a mount using --vfs-cache-mode=writes --dir-cache-time=10s
and with a ms word and excel documents, i can open/edit/save files instantly.

you could try `--vfs-cache-mode=full' which would download the file to the local cache.

matgas23 · May 11, 2020, 2:58pm

I tried with --vfs-cache-mode=full and it does the job but it doesn't fit perfectly to my needs becouse I need also to stream media....

matgas23 · May 11, 2020, 2:59pm

could this solve my problem? https://rclone.org/cache/

asdffdsa · May 11, 2020, 2:59pm

i run multiple mounts to the same remote

for media streaming
rclone mount GoogleDrive: X: --read-only

for editing files
rclone mount GoogleDrive: Y: --vfs-cache-mode=full

asdffdsa · May 11, 2020, 3:12pm

hi,
do you think this is gdrive issue, as per limits on transactions per second?

i never run into this problems on s3 wasabi.
open/edit/save operations are almost as fast my local drive.

asdffdsa · May 11, 2020, 3:19pm

what version of winfsp are you using?
latest stable is v1.6

matgas23 · May 11, 2020, 3:23pm

I have the latest version. winfsp v1.6.20027

matgas23 · May 11, 2020, 3:30pm

Probably this could be the solution. I created an "rclone cache" following this guide and it seems fit to my needs.
Thanks for your help
Matteo

system · May 14, 2020, 3:30pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.