Are there optimal Rclone settings to speed up the scanning of a drive with multiple small files?

Help and Support
1

Afternoon all,

So I’m experiencing some painfully slow scanning of some google drive folders using Lidarr and Airsonic and I was wondering how much the configuration of my rclone mount might affect this, if at all?

I was hoping to move my music collection to my dedicated server, however that may not be realistic if scanning google drive mounts is incredibly slow.

2

These are my existing mount settings. Is there anything in there that could be tweakable to better deal with multiple smaller files?

[Unit]
Description=RClone Service

[Service]
Type=notify
Environment=RCLONE_CONFIG=//home/craftyclown/.config/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount "RichFlixCrypt:/Encrypted/" /home/craftyclown/Skull \
   --allow-other \
   --buffer-size 256M \
   --dir-cache-time 72h \
   --drive-chunk-size 32M \
   --log-level INFO \
   --log-file /home/craftyclown/logs/rclone.log \
   --umask 002 \
   --vfs-read-chunk-size 128M \
   --vfs-read-chunk-size-limit off \
   --rc
ExecStop=/bin/fusermount -uz /home/craftyclown/Skull
Restart=on-failure
User=craftyclown
Group=craftyclown

[Install]
WantedBy=multi-user.target
3

Yes you could add and tweak --tpslimit=X, however keep in mind API limits so google dont block you for few hours. ( Iam not sure whats default tpslimit … maybe @ncw can give you more info )

Also you could download test rclone beta with new command --drive-pacer-min-sleep=10ms (default) https://github.com/ncw/rclone/issues/2880

With those two settings you could play and get the best performance, eg if you set tpslimit to 10 or 20 you may get.

Test that ncw did:

4

You could try to increase the checkers. I got good results with --checkers 128 for sync and copy commends. However I haven’t tested it on google drive.

5

Sync and copy yes but i dont think checkers do anything for mount.

6

So if you are using Plex as well, it really doesn’t do well with small files and music as it open and closes them multiple times per second when scanning.

I haven’t ever used Lidarr or Airsonic, but you’d need to grab a debug log of a slow scan to see what it’s doing.

If it is the same to what Plex is doing, you’d probably want to use the cache backend for small music files as that handles that scenario a bit better. You won’t be able to tune out of that scenario with VFS as you’ll always have some latency to open and close a cloud file if there isn’t anything local.