What is the problem you are having with rclone?
I am trying to achieve something very similar to how most users use rclone + plex. I am trying to run a calibre docker container, with rclone and gdrive (shared drives) + crypt. I previously have not been using or looking at the --cache* flags, and am wondering if that would help. Like those outlined here: https://rclone.org/commands/rclone_mount/#file-caching
Currently what happens if I mount the docker containers on top of the rclone mounts: https://pastebin.com/mDCTXFF3
And while the log is pretty clear, I am just wondering more about performance, if minimal or writes is the better way to go? If write locks the filesystem and creates the full file size prior to writing, this would potentially mean I need a disk space as large as I can possibly get fr the db, correct?
As I am expecting the size of the metadata.db file to grow in excess and the entire Calibre Library configuration to grow over 200GB in size. I noticed with minimal enabled, I am constantly getting inotify errors when running tail.
What is your rclone version (output from rclone version
)
rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 20.04 LTS 64bit
Which cloud storage system are you using? (eg Google Drive)
Google Drive Shared Drives
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
[Unit]
Description=RClone Service
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/andrew/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount gdriveebooks-calibre: /mnt/rclone/gcalibre \
--allow-other \
--buffer-size 256M \
--dir-cache-time 1000h \
--log-level INFO \
--log-file /var/log/rclone.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--rc \
--rc-addr 127.0.0.1:5577
ExecStop=/bin/fusermount -uz /mnt/rclone/gcalibre
Restart=on-failure
User=andrew
Group=andrew
[Install]
WantedBy=multi-user.target
A log from the command with the -vv
flag (eg output from rclone -vv copy /tmp remote:tmp
)
020/05/12 04:39:14 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:14 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2020/05/12 04:39:14 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:14 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:14 ERROR : config/app.db: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2020/05/12 04:39:15 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:15 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2020/05/12 04:39:15 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:15 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
So after changing the vfs-cache-mode
to writes
, it is fully working, minimal didn't do it. However, I am still seeing loads of errors/slowness. The cache folder is placed on a 200GB Virtual Hard Drive attached to a VM, and that VHD is hosted on a 960 Pro NVME ssd.
The updated mount and log file:
# cat rclone-ebooks-calibre.service
[Unit]
Description=RClone Service
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/andrew/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount gdriveebooks-calibre: /mnt/rclone/gcalibre \
--allow-other \
--buffer-size 256M \
--cache-dir /mnt/calibre/.cache/ \
--dir-cache-time 1000h \
--log-level INFO \
--log-file /var/log/rclone.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--vfs-cache-mode writes \
--rc \
--rc-addr 127.0.0.1:5577
ExecStop=/bin/fusermount -uz /mnt/rclone/gcalibre
Restart=on-failure
User=andrew
Group=andrew
[Install]
WantedBy=multi-user.target
2020/05/12 08:06:32 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:35 INFO : config/Calibre Library/calibre_test_case_sensitivity.txt: Copied (new)
2020/05/12 08:06:35 INFO : config/Calibre Library/calibre_test_case_sensitivity.txt: Removed from cache
2020/05/12 08:06:38 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:39 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:41 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:42 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:45 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:46 ERROR : config/Calibre Library/metadata.db-journal(0xc00207c640): failed to find cache file: object not found
2020/05/12 08:06:46 ERROR : config/Calibre Library/metadata.db-journal(0xc00207c640): RWFileHandle.Release error: failed to find cache file: object not found
2020/05/12 08:06:46 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:48 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:49 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:49 ERROR : config/Calibre Library/metadata.db-journal(0xc00207ce00): failed to find cache file: object not found
2020/05/12 08:06:49 ERROR : config/Calibre Library/metadata.db-journal(0xc00207ce00): RWFileHandle.Release error: failed to find cache file: object not found
2020/05/12 08:06:51 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:52 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:54 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:55 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:00 INFO : config/.config/calibre/gui.py.json: Copied (new)
2020/05/12 08:07:05 INFO : config/.config/calibre/global.py.json: Copied (replaced existing)
2020/05/12 08:07:07 INFO : config/.config/calibre/gui.json: Copied (new)
2020/05/12 08:07:07 INFO : Cleaned the cache: objects 24 (was 24), total size 384.319k (was 49.329k)
2020/05/12 08:07:11 INFO : config/Calibre Library/metadata.db: Copied (new)
2020/05/12 08:07:14 INFO : config/Calibre Library/calibre_test_case_sensitivity.txt: Copied (new)
2020/05/12 08:07:15 INFO : config/Calibre Library/calibre_test_case_sensitivity.txt: Removed from cache
2020/05/12 08:07:18 INFO : config/.config/calibre/global.py.json: Copied (replaced existing)
2020/05/12 08:07:20 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:07:21 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:24 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:07:25 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:28 INFO : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:07:30 INFO : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:31 INFO : config/.config/calibre/gui.json: Copied (replaced existing)
2020/05/12 08:07:33 INFO : config/Calibre Library/metadata.db: Copied (replaced existing)
2020/05/12 08:08:07 INFO : Cleaned the cache: objects 24 (was 24), total size 384.815k (was 384.319k)
2020/05/12 08:08:13 INFO : config/.config/QtProject.conf.rbJyMA: Removed from cache
2020/05/12 08:08:13 ERROR : config/.config/QtProject.conf.rbJyMA(0xc00207d680): failed to find cache file: object not found
2020/05/12 08:08:13 ERROR : config/.config/QtProject.conf.rbJyMA(0xc00207d680): RWFileHandle.Release error: failed to find cache file: object not found
2020/05/12 08:08:23 INFO : config/.config/QtProject.conf.qlfsvS: Copied (new)
2020/05/12 08:08:23 INFO : config/.config/QtProject.conf.qlfsvS: Renamed in cache
2020/05/12 08:08:24 INFO : config/.config/QtProject.conf.qlfsvS: Moved (server side)
2020/05/12 08:08:26 INFO : config/.config/QtProject.conf.lock: Copied (new)
2020/05/12 08:08:27 INFO : config/.config/QtProject.conf.lock: Removed from cache
2020/05/12 08:09:07 INFO : Cleaned the cache: objects 25 (was 25), total size 385.304k (was 384.815k)
2020/05/12 08:10:07 INFO : Cleaned the cache: objects 25 (was 25), total size 385.304k (was 385.304k)
Is there a way to cache only specific files and exclude others potentially?