I want to use rclone mount for OneDrive without cache

Le_Chi_Kien · July 14, 2023, 3:40pm

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I want to use rclone mount for OneDrive without cache (--vfs-cache-mode off) because my server just has 25gb (including 7gb empty). But it still use with cache, when writing it need to much storage lead to out of disk, don't like Google Drive.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: Microsoft Windows Server 2022 Datacenter 21H2 (64 bit)
- os/kernel: 10.0.20348.1850 Build 20348.1850.1850 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone mount --vfs-cache-mode off "onedrive:/" X:

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
token = {"access_token":"**}
drive_id = **
drive_type = business

A log from the command with the `-vv` flag

2023/07/15 00:39:32 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "mount" "-vv" "--vfs-cache-mode" "off" "onedrive:/" "y:"]
2023/07/15 00:39:32 DEBUG : Creating backend with remote "onedrive:/"
2023/07/15 00:39:32 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/07/15 00:39:33 DEBUG : fs cache: renaming cache item "onedrive:/" to be canonical "onedrive:"
2023/07/15 00:39:33 DEBUG : OneDrive root '': Next delta token is: **
2023/07/15 00:39:33 NOTICE: OneDrive root '': --vfs-cache-mode writes or full is recommended for this remote as it can't stream
2023/07/15 00:39:33 DEBUG : Network mode mounting is disabled
2023/07/15 00:39:33 DEBUG : Mounting on "y:" ("onedrive")
2023/07/15 00:39:33 DEBUG : OneDrive root '': Mounting with options: ["-o" "attr_timeout=1" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=onedrive"]
2023/07/15 00:39:33 DEBUG : OneDrive root '': Init:
2023/07/15 00:39:33 DEBUG : OneDrive root '': >Init:
2023/07/15 00:39:33 DEBUG : /: Statfs:
2023/07/15 00:39:34 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:1342177280 Bfree:1243831512 Bavail:1243831512 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/15 00:39:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/15 00:39:34 DEBUG : /: >Getattr: errc=0
2023/07/15 00:39:34 DEBUG : /: Readlink:
2023/07/15 00:39:34 DEBUG : /: >Readlink: linkPath="", errc=-40
2023/07/15 00:39:34 DEBUG : /: Getxattr: name="non-existant-a11ec902d22f4ec49003af15282d3b00"
2023/07/15 00:39:34 DEBUG : /: >Getxattr: errc=-40, value=""
2023/07/15 00:39:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/15 00:39:34 DEBUG : /: >Getattr: errc=0
2023/07/15 00:39:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/15 00:39:34 DEBUG : /: >Getattr: errc=0
2023/07/15 00:39:34 DEBUG : /: Opendir:
The service rclone has been started.
2023/07/15 00:39:34 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/07/15 00:39:34 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2023/07/15 00:39:34 DEBUG : /: >Opendir: errc=0, fh=0x0
2023/07/15 00:39:34 DEBUG : /: Getpath: Getpath fh=0
2023/07/15 00:39:34 DEBUG : /: >Getpath: errc=0, normalisedPath=""
2023/07/15 00:39:34 DEBUG : /: Releasedir: fh=0x0
2023/07/15 00:39:34 DEBUG : /: >Releasedir: errc=0
2023/07/15 00:39:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/15 00:39:34 DEBUG : /: >Getattr: errc=0
2023/07/15 00:39:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/15 00:39:34 DEBUG : /: >Getattr: errc=0
2023/07/15 00:39:34 DEBUG : /: Opendir:
2023/07/15 00:39:34 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/07/15 00:39:34 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2023/07/15 00:39:34 DEBUG : /: >Opendir: errc=0, fh=0x0
2023/07/15 00:39:34 DEBUG : /: Getpath: Getpath fh=0
2023/07/15 00:39:34 DEBUG : /: >Getpath: errc=0, normalisedPath=""
2023/07/15 00:39:34 DEBUG : /: Releasedir: fh=0x0
2023/07/15 00:39:34 DEBUG : /: >Releasedir: errc=0
2023/07/15 00:39:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/15 00:39:34 DEBUG : /: >Getattr: errc=0
2023/07/15 00:39:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/15 00:39:34 DEBUG : /: >Getattr: errc=0
2023/07/15 00:39:34 DEBUG : /: Opendir:
2023/07/15 00:39:34 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/07/15 00:39:34 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2023/07/15 00:39:34 DEBUG : /: >Opendir: errc=0, fh=0x0
2023/07/15 00:39:34 DEBUG : /: Getpath: Getpath fh=0
2023/07/15 00:39:34 DEBUG : /: >Getpath: errc=0, normalisedPath=""
2023/07/15 00:39:34 DEBUG : /: Releasedir: fh=0x0
2023/07/15 00:39:34 DEBUG : /: >Releasedir: errc=0

asdffdsa · July 14, 2023, 3:44pm

welcome to the forum,

if the cache is disabled, then rclone is not using cache.
and based on the short log snippet, rclone is not using cache.

tho if you did use the cache and rclone stored files inside the cache,
you might need to manually remove old, stale files from the cache.

if you used the default location of the cache, can you find using
https://rclone.org/commands/rclone_config_paths/

Le_Chi_Kien · July 14, 2023, 3:54pm

Config file: C:\Users\Administrator\AppData\Roaming\rclone\rclone.conf
Cache dir: C:\Users\Administrator\AppData\Local\rclone
Temp dir: C:\Users\ADMINI~1\AppData\Local\Temp\2

Temp dir is the location of cache I want to ask.

maxwellb · July 14, 2023, 4:00pm

Your log doesn't show which file access is occuring. vfs-cache-mode off documentation describes some limitations on allowed operations to files used in this mode. Plus, according to your log, a cache mode is recommended as it can't stream.

OneDrive root '': --vfs-cache-mode writes or full is recommended for this remote as it can't stream

Purely conjecture here, but Windows temporary files may be the culprit. If Windows is detecting that the file access is not supported for the operation it is trying to perform (which is not detailed here) then it may be using its own temporary files before uploading to the mounted drive. For example, video players may perform seek operations, and office applications need read/write.

From vfs-cache-mode full documentation:

This mode should support all normal file system operations.

Try turning on different cache modes and controlling the amount of memory used to smaller amounts (such as buffer-size=20M and vfs-read-ahead=100M, but you really would have to test). The more that rclone can support the operations you are performing, the more that it can control the limits.

A similar post has recommended mounting a remote read-only to limit the OS interaction. Keep in mind what applications and files you have open so that memory/disk can be reclaimed as actively as possible, whether that is the OS or rclone.

Good luck!

asdffdsa · July 14, 2023, 4:08pm

if the cache is off, then rclone should not be using local storage.
what proof do you have that rclone mount is using local storage?

perhaps use rclone copy which does not use cache at all

Le_Chi_Kien · July 14, 2023, 4:10pm

This image with the dir I've answer on previous reply

asdffdsa · July 14, 2023, 4:18pm

your rclone mount command is not using cache. your logs show that.
the cache is the not problem.
the problem is that rclone does not support streamed uploads for onedrive, so rclone needs to create a temp file.

note: rclone copy does not that have problem.

gdrive supports streamed uploads, onedrive does NOT support streamed uploads.

basically, when you copy a file into the mount without using cache, rclone does not know the file size,
so this is the sequence of events.

you copy source file into rclone mount.
rclone copies source file to temp dir.
rclone moves/uploads the temp file from temp dir to onedrive

jwink3101 · July 14, 2023, 9:24pm

I do not know how the underlying file-level APIs work, but you'd think that the file system (or whatever tool is doing the copy) would communicate the size. I guess that is just not how they work, but I am surprised.

I say this because you can use rcat and other stream uploads with OneDrive if you know the size.

Le_Chi_Kien · July 17, 2023, 12:35pm

So I need to use any flags to rclone don't use temp? Because once files has big size (4-5GB/file) and I need to use without temp or cache

kapitainsky · July 17, 2023, 12:51pm

Do not use mount to upload files to your remote.

You can always copy them directly:

rclone copy file.txt onedrive:path/to/file1

Le_Chi_Kien · July 20, 2023, 4:09pm

Like previous post, I can't use copy because my files is very big, and my disk is lack so I had to use mount without temp like Google Drive. Anyone have solution? I can't buy new disk.

kapitainsky · July 20, 2023, 4:28pm

You are mixing things I think. If you have this file already than obviously you have space to store it.

No need to use mount.

Simply copy it to remote using ‘rclone copy’. It will not create any temp files.

If your system has limited resources do not use mount at all. Especially not for writing. Mount purpose is to create impression that it is yet another local disk. And it comes with some cost.

system · August 19, 2023, 4:28pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.