Rclone confusing Calibre via Sugarsync in a weird way

Help and Support
1

Hello, my first post here, and quite new to Linux at home, hope not to say something too stupid, and, I am not sure it is a problem with Rclone but I need to start from here, hoping there is some limitation I don't understand.

What is the problem you are having with rclone?

For those who do not know it, Calibre is a library management software.
In a nutshell, the problem I have is that Calibre on Linux is seeing all but the newest 30 books (not one more, not one less, and exactly the last ones added), when it is using Rclone to load the library.

So, I have since long time a Sugarsync account that I use to keep my Calibre library in the cloud. So far it has always worked fine when sharing it between multiple Windows computers.
Now, I have set up this Fedora 39 Workstation on my older pc (i7-4770K, 16GB RAM) and installed calibre 7.0 there, same version I have on Windows.

Since Sugarsync does not have a Linux client, I am using rclone to keep things synced.
I have directory "Biblioteca di calibre" in Sugarsync, and I am trying to sync it with /home/marco/Biblioteca di calibre.
There are about 500 directories in there and a total of 2.5GB of files in total (in subdirectories).
I used "rclone mount" to mount the sugarsync directory to the path where calibre looks for its files.
Issuing an ls -lrt there, I can see the files I expect to see (see pictures attached), so I assume the remote configuration and the mount command work fine.
For sure it looks like that, for example:

EDIT: I am not allowed to post images

Barring the different order an characted size, in general I can't spot anything missing.

However, as I said, if I run the calibre program I can see all but the most recent ones... that is, the latest 30 that I have added are visible in calibre for windows but not in calibre for linux.

EDIT: I am not allowed to post images

What is puzzling me so much is that the files are there! For example, one of the "missing books" as you can see from the pictures is "La cena" by Herman Koch. You can see here that the file is actually seen in the Linux mount:

one_of_the_missing
one_of_the_missing736×142 32.9 KB

True, the directory structure has some spaces and things that can confuse rclone and or Linux, but this happens not only for those 30 books, but for all of them, and anyway if only some files are impacted, why only the 30 most recently added?

Run the command 'rclone version' and share the full output of the command.

rclone 1.64.2

  • os/version: fedora 39 (64 bit)
  • os/kernel: 6.5.11-300.fc39.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Sugarsync

The command you were trying to run (eg rclone copy /tmp remote:tmp)

root@localhost-live:/etc/systemd/system# su - marco
marco@localhost-live:~$ mount Biblioteca\ di\ calibre/

Please run 'rclone config redacted' and share the full output. If you get

marco@localhost-live:~$ rclone config redacted
[calibre_lib]
type = sugarsync
username = 
refresh_token = XXX
authorization = XXX
authorization_expiry = 2023-11-19T05:57:21-08:00
user = XXX
root_id = XXX
deleted_id = XXX

[google-drive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = 
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

N/A or don't know how to get what

I put this in my /etc/fstab:

calibre_lib:Biblioteca\040di\040calibre /home/marco/Biblioteca\040di\040calibre rclone rw,noauto,vfs_cache_mode=full 0 0

(Btw I tried to set it to automount, and I made the system un-bootable and had to rescue it)
With this, the mount command works.
I can also mount the disk with

rclone mount calibre_lib:Biblioteca\ di\ calibre /home/marco/Biblioteca\ di\ calibre --vfs-cache-mode full

The result is the same: I can see the books, but not the most recent ones.
I really have no idea what can "confuse" my two calibre instances... It could be a problem in calibre, true, but I really can't see why.
What could be causing this issue?

Thanks!

2

I made some progress myself. Looks like I didn't inspect the directories carefully enough.
Actually Calibre has a feature to export all books, plugins and preferences from an instance to the other. It produces a 2.6GB file of my library, that I can copy over and import to Linux, to have a "local" version offline of the library.

If I do export and import, Calibre is able to see the complete library correctly, so it doesn't seem to be a problem in the library itself.

But, most important, looking at the directories side by side I could see some important differences:

sync_error
sync_error1842×1000 242 KB

As you can see in the picture, first: longer directory names gets truncated: you can see one example in the picture but I spotted others.
And if this looks to be a limitation: not good for me but that can make sense, the second issue is that one file is incompletely copied! :open_mouth:
The metadata file, which is the "core" of the library (and I think contains data about them all) is smaller in the rcloned directory!
How can this be? This looks to be a bug, or a serious limitation by rclone+sugarsync.

Can you help me figuring out why this happens?

Thanks

3

If you forget Calibre software problems could you distil some summary what is rclone issue?

Is your rclone mount missing files or folders? so for example you have different results when you run:

rclone ls calibre_lib:someFolder

vs

ls -l "~/Biblioteca di calibre/someFolder"
4

Hello Kapitainsky, thanks for helping.

I found some differencese as I said, but in part I was wrong.
So, I confirm that for file metadata.db and metadata_db_prefs* sizes differ between what I see in Windows and what I see in Linux.
At first I thougth that also directory names were getting truncated, but I was wrong: won't enter the details, but when comparing Windows and Linux version of the directory (and not the export that Calibre creates) the directory names look the same, so it's only a couple of files getting copied differently for a reason I ignore.

As per your specific question, I had to slightly change the ls command, as the ~ was not getting resolved correctly in your suggestion, so here's the result:

marco@localhost-live:/$ ls -l "~/Biblioteca di calibre/Richard Dawkins"
ls: impossibile accedere a '~/Biblioteca di calibre/Richard Dawkins': File o directory non esistente
marco@localhost-live:/$ ls -l ~/Biblioteca\ di\ calibre/Richard\ Dawkins
totale 0
drwxr-xr-x. 1 marco marco 0  1 gen  2000 'Il gene egoista (1978)'
drwxr-xr-x. 1 marco marco 0  1 gen  2000 'L'\''orologiaio cieco (1979)'
marco@localhost-live:/$

Whereas if I use the other command I get:

marco@localhost-live:/$ rclone ls calibre_lib:Biblioteca\ di\ calibre/Richard\ Dawkins
  1321498 L'orologiaio cieco (1979)/L'Orologiaio Cieco - Richard Dawkins.epub
   181275 L'orologiaio cieco (1979)/cover.jpg
     6697 L'orologiaio cieco (1979)/metadata.opf
  1497184 Il gene egoista (1978)/Il gene egoista - Richard Dawkins.epub
   237441 Il gene egoista (1978)/cover.jpg
     5883 Il gene egoista (1978)/metadata.opf

To compare all files I re-ran the ls command recursively:

marco@localhost-live:/$ ls -lR ~/Biblioteca\ di\ calibre/Richard\ Dawkins
'/home/marco/Biblioteca di calibre/Richard Dawkins':
totale 0
drwxr-xr-x. 1 marco marco 0  1 gen  2000 'Il gene egoista (1978)'
drwxr-xr-x. 1 marco marco 0  1 gen  2000 'L'\''orologiaio cieco (1979)'

'/home/marco/Biblioteca di calibre/Richard Dawkins/Il gene egoista (1978)':
totale 1701
-rw-r--r--. 1 marco marco  237441 15 nov 23.20  cover.jpg
-rw-r--r--. 1 marco marco 1497184 18 nov 19.42 'Il gene egoista - Richard Dawkins.epub'
-rw-r--r--. 1 marco marco    5883 18 nov 20.47  metadata.opf

'/home/marco/Biblioteca di calibre/Richard Dawkins/L'\''orologiaio cieco (1979)':
totale 1476
-rw-r--r--. 1 marco marco  181275 15 nov 23.20  cover.jpg
-rw-r--r--. 1 marco marco 1321498 18 nov 19.48 "L'Orologiaio Cieco - Richard Dawkins.epub"
-rw-r--r--. 1 marco marco    6697 18 nov 20.37  metadata.opf

Also metadata files in the main directory have the same values with the two commands... The problem is that they differ from their source system counterpart:

marco@localhost-live:/$ ls -l ~/Biblioteca\ di\ calibre/
totale 6071
[...]
-rw-r--r--. 1 marco marco 6193152 19 nov 16.20  metadata.db
-rw-r--r--. 1 marco marco   23116 19 nov 16.25  metadata_db_prefs_backup.json

marco@localhost-live:/$ rclone ls calibre_lib:Biblioteca\ di\ calibre
  6193152 metadata.db
    23116 metadata_db_prefs_backup.json
5

Sorry I do not understand.

You mean they are the same?

What do you mean by "differ from their source system counterpart?"

Source of

~/Biblioteca\ di\ calibre/

are files which resides on your mount in:

calibre_lib:Biblioteca\ di\ calibre

6

Uh, ok, let me see if I can explain me clearer.
So, first of all keep in mind that this entire "library" has been built over the years on my Windows Calibre, eventually moved in the cloud when I purchased a Sugarsync subscription.
So, with source, technically it should be the files in the cloud, alright?
But also, in my previous post I was considering "source" to be what I have in my Windows pc. It might not be 100% correct since now the cloud is source to both Win and Linux, but:

  • Cloud files where originally the Windows files loaded into the cloud by Sugarsync client
  • I keep using Windows Calibre, saving and reading cloud files without any hassle and with no problems, so I think what it "sees" on local windows pc is correct.

Edit: to be even clearer, the workflow I have is this: I work on Calibre Windows (need to), I close the software; metadata files get modified; SS agent uploads modified files to SS cloud; Rclone downloads new files to my Linux client.
So, when I say they differ from their source counterpart I mean that if I issue the ls -l or an rclone ls on linux I see the same data, and in particular, the same filesize, but if I compare it with the file in my Windows system, for those "metadata.db" files the size are different.
Can't really tell what Sugarsync "sees" because you can't have the exact size there.

My guess (might be wrong) that the "correct" size is the one on Windows and that Sugarsync has the same size, but rclone sees somehow an incorrect size on the cloud source and therefore retrieves an incomplete file.

See the picture here:

comparison
comparison1150×393 47.8 KB

As you can see, metadata.db size is different between Linux and Windows. Sugarsync client is constantly copying what I modify on Windows up to the cloud, and it should be therefore synced to Linux, but it is not.

As for the different timestamp, please note that the different time we have in Linux cannot explain the missing books (because for some reason Rclone is slow to update the Linux machine), because those 30 books have been added several days ago, not this evening.

I hope my issue is clearer now. :slight_smile:

7

I would start with simple test. Close Calibre software on Windows and on Linux.

Close any rclone mount.

Copy Calibre library from Windows to Linux - over SMB or even using USB memory stick - it is only 2.5 GB so should be quick.

Open Calibre program on Linux - does it work now as expected and sees all books?

8

Ok.
Calibre has this "export library and settings" feature that allows you to move an entire archive: it creates 3 files (which I believe are zip files or so) and then allows you to import them on another calibre. Doing so, everything shows up correctly on Calibre for Linux.

I'll grab am usb stick and do the "move" test right now.

9

Do not use any export features.

This test purpose is to "replace" rclone with manual copy. You do not use any export/import when syncing via Sugarsync using rclone...

10

Yes, yes, I know that it was our purpose. I was just adding it "on top", let's say.
I have now tried the lift&shift approach.
Copied Windows folder over to an usb stick (fs: exFAT), mounted in Fedora and then opened it on Calibre: all of the books are there, including the ones missing with rclone.

I also tried to copy the library from the usb stick in a subdirectory of my home (/home/marco/Documenti/Biblioteca di calibre), and again: all expected books show up when loading the library from calibre

1 Like
11

We have one thing less to worry about then:)

Test 2.

Close all Calibre. Trigger sync to cloud from Windows.

on Linux let's copy cloud content to some TMP folder.

rclone copy calibre_lib:Library /path/to/TMP

Now when using this library (from TMP) does Calibre Linux shows all books?

12

There's a daemon on Windows that keeps windows file synced, but I made sure it showed "all files are updated". I then issued this:

marco@localhost-live:~$ mkdir TMPLib
marco@localhost-live:~$ rclone copy calibre_lib:Biblioteca\ di\ calibre ~/TMPLib
<3>ERROR : Cixin Liu/Il problema dei tre corpi (1750)/metadata.opf: Failed to copy: failed to open source object: HTTP error 401 (401 Unauthorized): invalid token
<3>ERROR : Christopher Paolini/La forchetta, la strega, il drago (1249)/La forchetta, la strega, il dra - Christopher Paolini.epub: Failed to copy: failed to open source object: HTTP error 401 (401 Unauthorized): invalid token
<3>ERROR : Christopher Paolini/La forchetta, la strega, il drago (1249)/cover.jpg: Failed to copy: failed to open source object: HTTP error 401 (401 Unauthorized): invalid token
<3>ERROR : Attempt 1/3 failed with 3 errors and: failed to open source object: HTTP error 401 (401 Unauthorized): invalid token
<3>ERROR : Attempt 2/3 succeeded
marco@localhost-live:~$ mkdir TMPLib2
marco@localhost-live:~$ rclone copy calibre_lib:Biblioteca\ di\ calibre ~/TMPLib2
marco@localhost-live:~$

I don't know the reason of that invalid token issue (does it expires periodically and I was just "lucky"?), anyway to stay on the safe side I did the copy twice even if the second attempt showed as successful.

In both cases, alas, Calibre does not show all the books.
It only shows the ones that are also shown when using the "mounted" library

13

Btw, "data" of the missing books (that is, the subdirectory with the ebook files) is correctly copied both in rclone copied and in rclone mounted libraries, so it appears the issue is just in how that metadata.db is synced by rclone.

To test this, I put in TMPLib (which as I said, was missing latest books) the metadata.db that I had on my usb stick copied from Windows (which as I said, showed all the books).

Result --> Calibre shows all the books as it should.

The issue is in how metadata.db gets copied by rclone

1 Like
14

Now it is rclone or sugar sync…

What about to try other cloud? You can create free onedrive account. It is 5GB so more than enough for this library.

15

I would rather say it is a (suspected) rclone bug...
Sugarsync works fine while sending the same files to other Windows machines (I say Windows because Sugarsync only has clients for Win&Mac, and I don't own a Mac). I have set it up successfully in the past sharing my library between home and work computers, and Sugarsync had no issues with that.
Shall I raise a bug issue to the developers?

I mean... I can try, for the sake of testing, if it can be useful.
But I'd rather not to, in the long term.
I chose Sugarsync because it's the service that has the feature that I want at the cost I am willing to pay. For example, it can sync directories all over my pc, without the need to put a shortcut in a "synced" directory, like for example Dropbox does.

I tried other, for example I tried Koofr but it has a nasty feature where it messes with my synchronized directories: it creates an hidden directories called .SyncTrash, but that messes up Calibre own organization, so it's a no no. (But, metadata.db got copied correctly and all books where visible in Linux).

I don't like onedrive agent on Windows, I have it at work and it often nags me with unwanted notifications or problems.

Yes, I will have to come up with a different cloud provider if there is no other solution, but I hope to have a transition to linux less painful as possible. It's a shame it is not like that. :frowning:
Anyway, since it looks like a bug, I am still hoping the devs will solve it one day or another. Thanks for your help anyway!

Does it make sense to flag it as a bug or raise this to the attention of the devs? If so, how do I do this?

16

You can always report bug on github but at the moment what you want to report? Ideally you would have reproducible scenario others can replicate and test. I doubt anybody is going to install Calibre, create books library and test. What we need is the issue narrowed down to rclone behaviour.

Do not install any agents, onedrive software etc. Just create onedrive account and use rclone to test.

17

One more test would be easy and maybe conclusive.

All Calibre software not running.

Calculate sha1 of metadata.db - I do not use windows - you have to google how.

Make sure lib was synced to the cloud.

On Linux rclone copy calibre_lib:Library /path/to/TMP and check what is sha1 of metadata.db:

sha1sum /path/to/TMP/metadata.db

Is it the same on Windows and Linux?

18

Well, it "is" reproducible, if I just provide the faulty metadata.db file.
Since it has different size when copied to linux, one don't even need to launch calibre for that. It is sufficient to see if the file is copied exactly as it was.
Other than that, I cannot do. If the devs don't want to fix the issue I can't make them to. :slight_smile: What I can do is to provide the file that apparently is not copied correctly.

(Maybe there is a "magic sequence" of characters that make rclone crash??)

No no, I know how to make the test. :slight_smile:

What I was saying is that to use it as a long term solution, I would have to use onedrive agent on Windows, and I don't like it and how it works.
Also, I could and will use my cloud sharing service for other data, not just this library so the 5GB might not be sufficient. If possible, I would like to use Sugarsync.

If it turns out Rclone is not 100% compatible with Sugarsync or it is buggy, then so be it, I will either try other cloning solution on Linux, or other cloud services, right now I'm focused on having this combination work. :slight_smile:

I will do the sha1 test now and update you.

19

Here's the result:

Linux:

marco@localhost-live:~$ rclone copy calibre_lib:Biblioteca\ di\ calibre ~/TMPLib
marco@localhost-live:~/TMPLib$ sha1sum ./metadata.db 
261f13434ebe5fd68cd1643e6d5c82c8b3cb6efc  ./metadata.db
marco@localhost-live:~/TMPLib$

Windows:

D:\Biblioteca di calibre>CertUtil -hashfile metadata.db SHA1
SHA1 hash di metadata.db:
03e1985ac4c097fd323a9efdc2cd7e1d7f5c4165
CertUtil: - Esecuzione comando hashfile riuscita.

D:\Biblioteca di calibre>

Definitely incorrectly copied...

20

:open_mouth: :open_mouth: :open_mouth:

kapitainsky, the problem is definitely metadata.db, but maybe it is not rclone fault after all!!!
I did this one more additional test: on Linux, I logged to my Sugarsync account web page (I mean, with my browser) and downloaded from the web that metadata.db.
Well... it's size and checksum is exactly the same that rclone retrieves!!!

I don't know how technically rclone is getting files from sugarsync, but if it is the same "technology" that browsers use to download files, maybe it is THAT what is messed up...