Google filestream + rclone (slow writes) + 2

What is the problem you are having with rclone?

Hi all,

This is just a quick question I hope. What are the best settings to use with google drive as a backend, I'm trying to emulate something like google filestream which works on mac but not my linux machine. I write nearly all of my code inside my google drive so my standard operations are frequent but small file changes and reads.

Currently the flags I have are:

/usr/bin/rclone mount google: /home/craggles/gdrive -vv --vfs-cache-mode full --vfs-cache-max-size 64G --vfs-cache-max-age 1200h --dir-cache-time 1200h --vfs-read-chunk-size-limit 0 --poll-interval 120s

But simple file read and write operations still take suspiciously long and it doesn't feel like the disk is being used much or at all. A particularly example is when using git in VScode is often fails with lock files and such because the file read and write lag.

Any advice welcome,

Craggles

What is your rclone version (output from rclone version)

rclone version
rclone v1.57.0
- os/version: arch 21.2.0 (64 bit)
- os/kernel: 5.10.79-1-MANJARO (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone mount google: /home/craggles/gdrive -vv --vfs-cache-mode full --vfs-cache-max-size 64G --vfs-cache-max-age 1200h --dir-cache-time 1200h --vfs-read-chunk-size-limit 0 --poll-interval 120s

The rclone config contents with secrets removed.

rclone config show google
--------------------
[google]
type = drive
client_id =
client_secret =
scope =
root_folder_id =
service_account_file =

A log from the command with the -vv flag

rclone.service - rclone Amazon Cloud Drive FUSE mount
     Loaded: loaded (/home/craggles/.config/systemd/user/rclone.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-12-10 11:36:40 GMT; 2h 32min ago
       Docs: http://rclone.org/docs/
   Main PID: 1486175 (rclone)
      Tasks: 26 (limit: 38402)
     Memory: 291.1M
        CPU: 4min 27.197s
     CGroup: /user.slice/user-1002.slice/user@1002.service/app.slice/rclone.service
             └─1486175 /usr/bin/rclone mount google: /home/craggles/gdrive -vv --vfs-cache-mode full --vfs-cache-max-size 64G --vfs-cache-max-age 1200h --d>

Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: Lookup: name="cspell.yaml"
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: >Lookup: node=<nil>, err=no such file or directory
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: Lookup: name="cspell.yml"
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: >Lookup: node=<nil>, err=no such file or directory
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: Lookup: name="package.json"
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: >Lookup: node=<nil>, err=no such file or directory
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: Lookup: name="cspell.config.cjs"
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : +projects/2021_rl_early_stopping/: >Lookup: node=<nil>, err=no such file or directory
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : /: Attr:
Dec 10 14:09:29 narchie rclone[1486175]: DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>

Good post, now it is much easier to understand your setup!

I can give you a few general tips and then I will leave the rest to the experts in Google Drive mounts.

It is my understanding that you cannot use both client id and service account in the same remote, so you should delete one of the options. I would delete the service account unless you have a very specific reason to use it.

I would also start by having all tuning parameters at defaults unless you have tested and verified that they have a positive effect in your situation. Defaults are typically the best choice unless you have a situation that differs significantly from main-stream usage. I therefore suggest you try removing --dir-cache-time, --vfs-read-chunk-size-limit and --poll-interval to get a simple setup and then start testing/optimizing from there (if needed).

I don’t use mount much, but I do daily syncs and can see that Google Drive has some sort of rate limiting on the number of files that can be created/updated/deleted per second. This may be part of the reason why your mount cannot catch up if you do very frequent but small file changes and reads (to a high number of files).

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.