Calibre + rclone

What is the problem you are having with rclone?

I am trying to achieve something very similar to how most users use rclone + plex. I am trying to run a calibre docker container, with rclone and gdrive (shared drives) + crypt. I previously have not been using or looking at the --cache* flags, and am wondering if that would help. Like those outlined here: https://rclone.org/commands/rclone_mount/#file-caching

Currently what happens if I mount the docker containers on top of the rclone mounts: https://pastebin.com/mDCTXFF3

And while the log is pretty clear, I am just wondering more about performance, if minimal or writes is the better way to go? If write locks the filesystem and creates the full file size prior to writing, this would potentially mean I need a disk space as large as I can possibly get fr the db, correct?

As I am expecting the size of the metadata.db file to grow in excess and the entire Calibre Library configuration to grow over 200GB in size. I noticed with minimal enabled, I am constantly getting inotify errors when running tail.

What is your rclone version (output from rclone version)

rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 LTS 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive Shared Drives

The command you were trying to run (eg rclone copy /tmp remote:tmp)

[Unit]
Description=RClone Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/andrew/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount gdriveebooks-calibre: /mnt/rclone/gcalibre \
--allow-other \
--buffer-size 256M \
--dir-cache-time 1000h \
--log-level INFO \
--log-file /var/log/rclone.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--rc \
--rc-addr 127.0.0.1:5577

ExecStop=/bin/fusermount -uz /mnt/rclone/gcalibre
Restart=on-failure
User=andrew
Group=andrew

[Install]
WantedBy=multi-user.target

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

020/05/12 04:39:14 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:14 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2020/05/12 04:39:14 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:14 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:14 ERROR : config/app.db: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2020/05/12 04:39:15 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:15 ERROR : config/.config/calibre/global.py.json: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2020/05/12 04:39:15 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/05/12 04:39:15 ERROR : config/app.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal

So after changing the vfs-cache-mode to writes, it is fully working, minimal didn't do it. However, I am still seeing loads of errors/slowness. The cache folder is placed on a 200GB Virtual Hard Drive attached to a VM, and that VHD is hosted on a 960 Pro NVME ssd.

The updated mount and log file:

# cat rclone-ebooks-calibre.service
[Unit]
Description=RClone Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/andrew/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount gdriveebooks-calibre: /mnt/rclone/gcalibre \
--allow-other \
--buffer-size 256M \
--cache-dir /mnt/calibre/.cache/ \
--dir-cache-time 1000h \
--log-level INFO \
--log-file /var/log/rclone.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--vfs-cache-mode writes \
--rc \
--rc-addr 127.0.0.1:5577

ExecStop=/bin/fusermount -uz /mnt/rclone/gcalibre
Restart=on-failure
User=andrew
Group=andrew

[Install]
WantedBy=multi-user.target
2020/05/12 08:06:32 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:35 INFO  : config/Calibre Library/calibre_test_case_sensitivity.txt: Copied (new)
2020/05/12 08:06:35 INFO  : config/Calibre Library/calibre_test_case_sensitivity.txt: Removed from cache
2020/05/12 08:06:38 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:39 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:41 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:42 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:45 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:46 ERROR : config/Calibre Library/metadata.db-journal(0xc00207c640): failed to find cache file: object not found
2020/05/12 08:06:46 ERROR : config/Calibre Library/metadata.db-journal(0xc00207c640): RWFileHandle.Release error: failed to find cache file: object not found
2020/05/12 08:06:46 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:48 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:49 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:49 ERROR : config/Calibre Library/metadata.db-journal(0xc00207ce00): failed to find cache file: object not found
2020/05/12 08:06:49 ERROR : config/Calibre Library/metadata.db-journal(0xc00207ce00): RWFileHandle.Release error: failed to find cache file: object not found
2020/05/12 08:06:51 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:52 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:06:54 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:06:55 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:00 INFO  : config/.config/calibre/gui.py.json: Copied (new)
2020/05/12 08:07:05 INFO  : config/.config/calibre/global.py.json: Copied (replaced existing)
2020/05/12 08:07:07 INFO  : config/.config/calibre/gui.json: Copied (new)
2020/05/12 08:07:07 INFO  : Cleaned the cache: objects 24 (was 24), total size 384.319k (was 49.329k)
2020/05/12 08:07:11 INFO  : config/Calibre Library/metadata.db: Copied (new)
2020/05/12 08:07:14 INFO  : config/Calibre Library/calibre_test_case_sensitivity.txt: Copied (new)
2020/05/12 08:07:15 INFO  : config/Calibre Library/calibre_test_case_sensitivity.txt: Removed from cache
2020/05/12 08:07:18 INFO  : config/.config/calibre/global.py.json: Copied (replaced existing)
2020/05/12 08:07:20 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:07:21 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:24 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:07:25 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:28 INFO  : config/Calibre Library/metadata.db-journal: Copied (new)
2020/05/12 08:07:30 INFO  : config/Calibre Library/metadata.db-journal: Removed from cache
2020/05/12 08:07:31 INFO  : config/.config/calibre/gui.json: Copied (replaced existing)
2020/05/12 08:07:33 INFO  : config/Calibre Library/metadata.db: Copied (replaced existing)
2020/05/12 08:08:07 INFO  : Cleaned the cache: objects 24 (was 24), total size 384.815k (was 384.319k)
2020/05/12 08:08:13 INFO  : config/.config/QtProject.conf.rbJyMA: Removed from cache
2020/05/12 08:08:13 ERROR : config/.config/QtProject.conf.rbJyMA(0xc00207d680): failed to find cache file: object not found
2020/05/12 08:08:13 ERROR : config/.config/QtProject.conf.rbJyMA(0xc00207d680): RWFileHandle.Release error: failed to find cache file: object not found
2020/05/12 08:08:23 INFO  : config/.config/QtProject.conf.qlfsvS: Copied (new)
2020/05/12 08:08:23 INFO  : config/.config/QtProject.conf.qlfsvS: Renamed in cache
2020/05/12 08:08:24 INFO  : config/.config/QtProject.conf.qlfsvS: Moved (server side)
2020/05/12 08:08:26 INFO  : config/.config/QtProject.conf.lock: Copied (new)
2020/05/12 08:08:27 INFO  : config/.config/QtProject.conf.lock: Removed from cache
2020/05/12 08:09:07 INFO  : Cleaned the cache: objects 25 (was 25), total size 385.304k (was 384.815k)
2020/05/12 08:10:07 INFO  : Cleaned the cache: objects 25 (was 25), total size 385.304k (was 385.304k)

Is there a way to cache only specific files and exclude others potentially?

Nope.

I don't use rclone to write locally as I use a local file system and use mergerfs to combine the two and I upload every night. There are a number of changes coming to the VFS cache that would help for this but it's not quite there yet. You could try something like my solution or just wait a bit until it is released if you want to use the cached writing items.

Hm, is there some better setup here then?
With running the apps in Docker, I am constantly hitting the limitations of the file system, that I didn't previously see in a traditional setting.

As far as I can see, with the use of docker (calibre and plex) need performance tuning when using rclone as the filesystem, but I can't find a perfect medium between the two (traditional vs containerized). As I am having to manipulate open file allowances, and other things that I have not traditionally had to do.

this is how i do calibre and rclone, maybe it will spark ideas.

i run calibre application from a rclone crypted mount.
the calibre program and its database are in the cloud.
and my collection of books are in that mount.

this is my script

rclone mount calibrebooks: b:\mount\calibrebooks\ --vfs-cache-mode=writes --file-perms=0777
b:\mount\calibrebooks\calibre-portable.exe

@asdffdsa
That appears to be exactly what I am doing. But I just see a very slow writes, and application instability when using the docker container. I am running off of linux vs Windows.

@Animosity022 when using mergerfs, can you specify at a file level to exist versus a block device?

What I mean by this, you have it setup like so:

/CalibreLibrary (local drive)
./Book1 (localdrive)
./Book2(gdrive)
./Book3(gdrive)
./metadata.db (localdrive)

The unfortunate piece of this is that there is a folder /CalibreLibrary which contains both the database and then folders of the ebooks. That is where I am struggling to try and find separation. Is there a list of the upcoming changes to vfs? Is it available on Github?

for me and my use case, i am not using docker for calibre.
calibre is a portable app and light on resources.

not sure what the need for complex docker with simple calibre?

"very slow writes", perhaps that could be a gdrive issue.
IMHO, i find on this forum, that so many problems are due to gdrive and it quirks in limits of transactions per second and api bans and whatnot.
there is always a solution tho but docker is yet another layer of complexity.

mergerfs uses mount points not files as it combines mounts.

I don't use dockers as it's over complex for my setup as I have no need for makings things portable or in a container.

So I am having some issues on excluding a specific .cache/ folder in the root of the directory, but not the future .cache folders in subsequent directories.

/usr/bin/rclone move /mnt/calibre/ gdrivecalibre-crypt: -v -Pv --exclude-from /mnt/calibre/excludes-move --include-from /mnt/calibre/includes-move --filter-from /mnt/calibre/filter-from-move --user-agent meappp  --max-transfer 700G --dry-run | grep -i 'Excluded'
2020/05/15 23:26:31 ERROR : Using --filter is recommended instead of both --include and --exclude as the order they are parsed in is indeterminate
2020/05/15 23:26:31 DEBUG : rclone: Version "v1.51.0" starting with parameters ["/usr/bin/rclone" "move" "/mnt/calibre/" "gdrivecalibre-crypt:" "-v" "-Pv" "--exclude-from" "/mnt/calibre/excludes-move" "--include-from" "/mnt/calibre/includes-move" "--filter-from" "/mnt/calibre/filter-from-move" "--user-agent" "meappp" "--max-transfer" "700G" "--dry-run"]
2020/05/15 23:26:31 DEBUG : Using config file from "/home/andrew/.config/rclone/rclone.conf"
2020-05-15 23:26:32 DEBUG : includes-move: Excluded
2020-05-15 23:26:32 DEBUG : .cache: Excluded
2020-05-15 23:26:32 DEBUG : filter-from-move: Excluded
2020-05-15 23:26:32 DEBUG : excludes-move: Excluded
2020-05-15 23:26:32 DEBUG : includes-copy: Excluded
2020-05-15 23:26:32 DEBUG : excludes-copy: Excluded
2020-05-15 23:26:33 DEBUG : .local: Excluded
2020-05-15 23:26:33 DEBUG : excludes: Excluded
2020-05-15 23:26:33 DEBUG : Calibre Library: Excluded
2020-05-15 23:26:33 DEBUG : .config: Excluded
2020-05-15 23:26:33 DEBUG : .cache: Excluded
2020-05-15 23:26:33 DEBUG : config/.config: Excluded
2020-05-15 23:26:33 DEBUG : config/thinclient_drives: Excluded
2020/05/15 23:26:33 INFO  :

You can see what I mean in the last two lines.

I have created some exclude, include and filter from files:

$ grep * excludes-move includes-move filter-from-move
excludes-move:.cache/
excludes-move:config/Calibre\ Library/metadata.db
excludes-move:'config/Calibre Library/metadata.db'
excludes-move:config/Calibre\ Library/metadata_db_prefs_backup.json
filter-from-move:- config/Calibre\ Library/metadata.db
includes-copy:config/.cache**
includes-copy:config/Calibre Library/**
includes-move:config/.cache**
includes-move:config/Calibre Library**
excludes-move:config/Calibre\ Library/metadata.db
excludes-move:'config/Calibre Library/metadata.db'
excludes-move:config/Calibre\ Library/metadata_db_prefs_backup.json
includes-move:config/.cache**
includes-move:config/Calibre Library**
filter-from-move:- config/Calibre\ Library/metadata.db

There is a few issues here, you can notice that the metadata.db is not being excluded in the move. Which will cause problems

If I then edit the includes-move file like so:

$ cat includes-move
config/.cache**
config/Calibre Library/

It excludes everything. Which is problematic.

I also have tried to make it simpler by just making a .cache excludes:

**/metadata.db
**/metadata_db_prefs_backup.json
['./.cache/**/*', '!./config/.cache/**/*']

And apparently something like this last line doesn't work. :confused:

This is the problem... Instead of using both --include-from and --exclude-from change them all to use --filter-from flag

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.