Help with caching for rclone on router

Help and Support
1

Please can someone advise as to the appropriate cache / memory configuration options for my use case? I want to make my 'OneDrive:/Scanned Documents' folder available using rclone on my router. It has 512MB ram.

So at the moment I have:

rclone mount "OneDrive:/Scanned Documents/" /tmp/OneDrive/ --cache-dir /tmp --vfs-cache-mode full --vfs-cache-max-size 10M --buffer-size 10M --uma
sk 000 --allow-other

And I have /tmp/OneDrive set up as a samba share.

My full script is:

#!/bin/sh /etc/rc.common
# Copyright (C) 2007 OpenWrt.org

export PATH=/usr/sbin:/usr/bin:/sbin:/bin

exec &> /tmp/mount-onedrive.log

START=97
STOP=5

start() {
        rclone mkdir /tmp/OneDrive
        rclone mount "OneDrive:/Scanned Documents/" /tmp/OneDrive/ --cache-dir /tmp --vfs-cache-mode full --vfs-cache-max-size 10M  --buffer-size 10M --umask 000 --allow-other --daemon
}

stop() {
        fusermount -zu /tmp/OneDrive
        rclone rmdir /tmp/OneDrive
}

So the idea is that my scanner can then write to my samba share and files will get sent directly to my OneDrive.

I want to avoid eating up all memory on my router and I also want to avoid writing to the non-volatile memory.

2

I'd basically keep most of the defaults and just add:

      --use-mmap                             Use mmap allocator (see docs).

That reduces the memory allocation.

And

      --buffer-size SizeSuffix               In memory buffer size when reading files for each --transfer. (default 16Mi)

Depending on how many concurrent files you plan to have open, you might want to reduce this to 1M perhaps.

Running rclone on a tiny 512M device feels like a bad idea imo. See how it works out though for your use case.

3

Thanks. So download seems OK, but if I upload a large file it just goes at full rate and eats up all my router RAM. What controls the memory usage here? I want the upload cache to be 1-10MBs.

4

One thing that makes life easier is we have a great template that we ask folks to use. Sadly, you deleted it, an angel lost her wings and here we are.

I think you are using onedrive so:

https://rclone.org/onedrive/#onedrive-chunk-size

5

I have the feeling samba must be caching? If I upload a 250MB file it goes at full speed to my router from LAN.

6

No idea as running rclone and using samba are not something I'd recommend. I use rclone my client devices.

7

When writing to a rclone mount, the behaviour appears to be that file is saved in storage, and then once the file is saved in storage rclone begins to upload it. Is it possible instead to have rclone upload as the file is written to the rclone mount directory?

BTW I am using:

root@OpenWrt:/etc/init.d# rclone version
rclone v1.52.2_2020-06-24
8

That's an old version you'd want to update.

You'd need a debug log to see what it's doing.

9

So what about those who want to be able to upload files larger than they have storage space for? Is there a way to avoid being limited to uploading files < storage space on device on which rclone is run on?

In my case I am keen to set it so that it does not cache on upload. Or better limit the cache to 10MB for upload. Is that possible? But setting 'vfs-cache-mode off' results in a copy in Windows failing. Otherwise setting cache to 10MB as I thought I did above results in the entire 200MB file transferring very quickly (but I only have 2-3Mbyte/s upload rate), then hang. Then file transfer completes.

root@OpenWrt:/tmp# rclone mount "OneDrive:/Scanned Documents/" /tmp/OneDrive/ --cache-dir /tmp --vfs-cache-mode off --buffer-size 10M --umask 000 --allow-other
2021/09/11 16:35:44 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2021/09/11 16:35:45 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:46 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:47 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:48 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:49 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:50 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:51 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:52 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2021/09/11 16:35:53 ERROR : SCAN_20210614_100214_001547.zip: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
10

If you application requires write modes, you have to use that and that will store is locally first and upload it. There's no way around that.

11

Thank you so much for your continued input here. Actually I am only needing to upload PDFs of around 20MB from my scanner. For that purpose is simply only:

rclone mount "OneDrive:/Scanned Documents/" /tmp/OneDrive/ --cache-dir /tmp --vfs-cache-mode writes --umask 000 --allow-other --daemon

okay?

But for completeness is there no way to have this mount upload mechanism facilitate uploads of files with sizes greater than the available storage on the device from which rclone is run? In other words chunk is written to mount, rclone uploads chunk, and repeat? Rather than entire file uploaded, then rclone uploads entire file.

12

There is no other way as you need the space.

13

Thank you for confirming.

Does my init.d script below seem reasonable to you? The reason for mount in /tmp is to avoid writing to the router non-volatile memory. The reason for the 'mkdir' is because /tmp is reset on router reboot. Is the rclone mount as daemon and stop using 'fusermount -zu' OK? I have the script arranged to start before samba start and to stop before samba stop.

#!/bin/sh /etc/rc.common
# Copyright (C) 2007 OpenWrt.org

export PATH=/usr/sbin:/usr/bin:/sbin:/bin

exec &> /tmp/mount-onedrive.log

START=97
STOP=4

start() {
        rclone mkdir /tmp/OneDrive
        rclone mount "OneDrive:/Scanned Documents/" /tmp/OneDrive/ --cache-dir /tmp --vfs-cache-mode writes --umask 000 --allow-other --daemon
}

stop() {
        fusermount -zu /tmp/OneDrive
        rclone rmdir /tmp/OneDrive
}
14

I'm using rclone to backup my cloud accounts for one year. It was crashed on 128MB router with OpenWRT, but on 512 MB it's ok.
Some settings from my scripts:

export GOGC=20                          # for rclone
ATTR="$ATTR --use-mmap --transfers 2 --buffer-size=4M "
15

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.