Shared libraries with the same name are hidden when mounting seafile

rolf · August 26, 2021, 7:57pm

What is the problem you are having with rclone?

Setup

Seafile installation with user A and B
Additional user Backup
Both user have a library called "Mobile" which they share with user "backup"

To Backup the files out of the Seafile installation I'd like to use rclone to mount the Seafile shares as user Backup.

Problem

When mounting Seafile share with rclone only one "Mobile" library is available.

Further Analysis

Authenticated as user Backup:

In the Seafile web interface both shared "Mobile" libraries are listen in a table with the owner (e.g. A or B).
Connected through WebDAV both "Mobile" libraries are listed, the the ID of the user:
- Mobile-xxyyzz
- Mobile-xxyyzz

What is your rclone version (output from `rclone version`)

rclone v1.56.0

os/version: debian 10.10 (64 bit)
os/kernel: 4.19.0-17-amd64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.16.5
go/linking: static
go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Seafile

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

$ rclone mount files:/ /home/myuser/mnt

The rclone config contents with secrets removed.

[files]
type = seafile
url = <url>
user = <user>
pass = <password>

A log from the command with the `-vv` flag

Output after the following commands:

cd /home/myuser/mnt
ls

Output:

$ rclone -vv mount files:/ /home/myuser/mnt
2021/08/26 21:52:43 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "-vv" "mount" "files:/" "/home/myuser/mnt"]
2021/08/26 21:52:43 DEBUG : Creating backend with remote "files:/"
2021/08/26 21:52:43 DEBUG : Using config file from "/home/myuser/.config/rclone/rclone.conf"
2021/08/26 21:52:43 DEBUG : Seafile server version 8.0.5
2021/08/26 21:52:43 DEBUG : fs cache: renaming cache item "files:/" to be canonical "files:"
2021/08/26 21:52:43 INFO  : seafile root: poll-interval is not supported by this remote
2021/08/26 21:52:43 DEBUG : seafile root: Mounting on "/home/myuser/mnt"
2021/08/26 21:52:43 DEBUG : : Root: 
2021/08/26 21:52:43 DEBUG : : >Root: node=/, err=<nil>
2021/08/26 21:52:46 DEBUG : /: Attr: 
2021/08/26 21:52:46 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:47 DEBUG : /: Lookup: name=".git"
2021/08/26 21:52:47 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/08/26 21:52:47 DEBUG : /: Lookup: name=".git"
2021/08/26 21:52:47 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/08/26 21:52:47 DEBUG : /: Lookup: name="HEAD"
2021/08/26 21:52:47 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/08/26 21:52:48 DEBUG : /: Attr: 
2021/08/26 21:52:48 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:48 DEBUG : /: ReadDirAll: 
2021/08/26 21:52:48 DEBUG : /: >ReadDirAll: item=7, err=<nil>
2021/08/26 21:52:48 DEBUG : /: Lookup: name="Books"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=Books/, err=<nil>
2021/08/26 21:52:48 DEBUG : Books/: Attr: 
2021/08/26 21:52:48 DEBUG : Books/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:48 DEBUG : /: Lookup: name="Common"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=Common/, err=<nil>
2021/08/26 21:52:48 DEBUG : Common/: Attr: 
2021/08/26 21:52:48 DEBUG : Common/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:48 DEBUG : /: Lookup: name="Memories"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=Memories/, err=<nil>
2021/08/26 21:52:48 DEBUG : Memories/: Attr: 
2021/08/26 21:52:48 DEBUG : Memories/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:48 DEBUG : /: Lookup: name="Mobile"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=Mobile/, err=<nil>
2021/08/26 21:52:48 DEBUG : Mobile/: Attr: 
2021/08/26 21:52:48 DEBUG : Mobile/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:48 DEBUG : /: Lookup: name="Music"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=Music/, err=<nil>
2021/08/26 21:52:48 DEBUG : Music/: Attr: 
2021/08/26 21:52:48 DEBUG : Music/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:48 DEBUG : /: Attr: 
2021/08/26 21:52:48 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/08/26 21:52:48 DEBUG : /: Lookup: name=".git"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/08/26 21:52:48 DEBUG : /: Lookup: name=".git"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/08/26 21:52:48 DEBUG : /: Lookup: name="HEAD"
2021/08/26 21:52:48 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory

asdffdsa · August 26, 2021, 8:06pm

hello and welcome to the forum,

    Mobile-xxyyzz
    Mobile-xxyyzz

the two Mobile-xxyyzz folders, they have the exact same spelling or is each folder name unique?

if a folder has duplicate file names, rclone mount will display only one of them.
it has to emulate a local file system, which is not designed to handle that.

might be eaiser to test using rclone lsd

rolf · August 26, 2021, 8:51pm

Hi, thank you.

No, sorry, the folder have not exactly the same name. The xxyyzz is the User ID (I think):

Mobile-63fddc
Mobile-c559d5

I checked you the same with rclone lsd:

$ rclone lsd files:/

           0 2021-08-18 21:25:39        -1 Books
           0 2021-08-18 21:25:23        -1 Common
           0 2021-08-18 21:25:46        -1 Memories
 10752757984 2021-08-24 19:25:19        -1 Mobile
           0 2021-08-23 18:19:56        -1 Mobile
           0 2021-08-18 21:25:51        -1 Music

Two Mobile entries, but don't see the difference (except the size)

I setup a WebDAV to access the Seafile. Almost the same result:

rclone lsd files-webdav:/
          -1 2021-08-26 22:35:12        -1 Books
          -1 2021-08-26 22:35:12        -1 Common
          -1 2021-08-26 22:35:12        -1 Memories
          -1 2021-08-26 22:35:12        -1 Mobile-63fddc
          -1 2021-08-26 22:35:12        -1 Mobile-c559d5
          -1 2021-08-26 22:35:12        -1 Music

Then rclone mount files-webdav:/ /home/myuser/mnt:

ls
Books  Common  Memories  Mobile-63fddc  Mobile-c559d5  Music

So it seams that the problem is only with Seafile when using rclone mount.

No, I don't have the pro edition. It's (currently) only a private installation.

My first try was using webdav:

Mount seafile using webdav
Use restic to backup the files mounted through webdav

But I was using davfs2 which has some caching issues reported in the syslog

Then I saw that rclone has a Seafile backend which is (maybe) faster than WebDAV.

asdffdsa · August 26, 2021, 8:54pm

that rclone lsd files:/ does show two folders with the exact same name.
rclone mount will show only one of them.

0752757984 2021-08-24 19:25:19        -1 Mobile
           0 2021-08-23 18:19:56        -1 Mobile

i noticed that seafile server supports s3, which is the one i would try to use.
with rclone, seafile and seafile over webdav do not support hash to verify file transfers and does not support mod-time.
s3 supports both

rolf · August 26, 2021, 9:05pm

From the perspective as SW developer it makes absolutely sense. Two folders can't have the same (Who want to do that?). But can be confusing for some users.
And of course, sharing folder is a feature of Seafile and can't be applied directly to a file system.

It's not a big deal for me, because I can do a workaround (e.g. mobile1, mobile2).

I'll check out the s3, maybe it's an option.

jwink3101 · August 26, 2021, 9:05pm

I am not an expert at restic or seafile so take this with a grain of salt.

Restic uses the modtime of a file to avoid having to read, hash, and chunk every file every time. But since WebDAV and Seafile rclone remotes do not support ModTime, you are likely going to really mess up restic as it goes.

Furthermore, I thought seafile already does its own chunking. Why not just use rclone to backup or sync (if you use sync, I'd still do --backup-dir until you are certain). Or if you really want to make it complicated, call restic on the files directly.

Just something to think about. I may be totally off base.

asdffdsa · August 26, 2021, 9:08pm

sure, many new rcloners run into that problem once, never twice

as this is not a rclone bug, perhaps change the category from suspected bug to help and support.

rolf · August 26, 2021, 9:36pm

I never said my solution is the best And yes, what I've seen is using restic in combination with WebDAV mounts not the best solution.

Seafile does chunking. But that's a little bit annoying for me. Recovery should be as simple as possible, without setup a Seafile and mysql server first. That's why I try to backup the actual files, not the seafile chunks.

And chunking chunks with restic again is also not my intention.

So I'll rethink my backup strategy. But thank you for your inputs.

asdffdsa · August 26, 2021, 9:43pm

hmm,

rclone lsd files:/ returns folders with the same name
whereas rclone lsd files-webdav:/ returns the same folders with unique names.

can you post the output of
rclone lsd files:/ --dump=headers --retries=1 --low-level-retries=1 -vv

rolf · August 27, 2021, 6:50am

Here is the output. Don't know if it helps:

$  rclone lsd files:/ --dump=headers --retries=1 --low-level-retries=1 -vv
2021/08/27 08:43:26 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "lsd" "files:/" "--dump=headers" "--retries=1" "--low-level-retries=1" "-vv"]
2021/08/27 08:43:26 DEBUG : Creating backend with remote "files:/"
2021/08/27 08:43:26 DEBUG : Using config file from "/home/sysadmin/.config/rclone/rclone.conf"
2021/08/27 08:43:26 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/08/27 08:43:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/27 08:43:26 DEBUG : HTTP REQUEST (req 0xc00040bd00)
2021/08/27 08:43:26 DEBUG : GET /api2/server-info/ HTTP/1.1
Host: files.example.com
User-Agent: rclone/v1.56.0
Accept-Encoding: gzip

2021/08/27 08:43:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/27 08:43:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/27 08:43:26 DEBUG : HTTP RESPONSE (req 0xc00040bd00)
2021/08/27 08:43:26 DEBUG : HTTP/2.0 200 OK
Allow: GET, HEAD, OPTIONS
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Fri, 27 Aug 2021 06:43:26 GMT
Server: nginx/1.14.0 (Ubuntu)
Vary: Accept-Language, Cookie

2021/08/27 08:43:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/27 08:43:26 DEBUG : Seafile server version 8.0.5
2021/08/27 08:43:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/27 08:43:26 DEBUG : HTTP REQUEST (req 0xc00040a000)
2021/08/27 08:43:26 DEBUG : POST /api2/auth-token/ HTTP/1.1
Host: files.example.com
User-Agent: rclone/v1.56.0
Content-Length: 69
Content-Type: application/json
Accept-Encoding: gzip

2021/08/27 08:43:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/27 08:43:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/27 08:43:26 DEBUG : HTTP RESPONSE (req 0xc00040a000)
2021/08/27 08:43:26 DEBUG : HTTP/2.0 200 OK
Allow: POST, OPTIONS
Content-Language: en
Content-Type: application/json
Date: Fri, 27 Aug 2021 06:43:26 GMT
Server: nginx/1.14.0 (Ubuntu)
Vary: Accept-Language, Cookie

2021/08/27 08:43:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/27 08:43:26 DEBUG : fs cache: renaming cache item "files:/" to be canonical "files:"
2021/08/27 08:43:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/27 08:43:26 DEBUG : HTTP REQUEST (req 0xc00071c000)
2021/08/27 08:43:26 DEBUG : GET /api2/repos/ HTTP/1.1
Host: files.example.com
User-Agent: rclone/v1.56.0
Authorization: XXXX
Accept-Encoding: gzip

2021/08/27 08:43:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/27 08:43:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/27 08:43:26 DEBUG : HTTP RESPONSE (req 0xc00071c000)
2021/08/27 08:43:26 DEBUG : HTTP/2.0 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Fri, 27 Aug 2021 06:43:26 GMT
Enable_encrypted_library: 0
Server: nginx/1.14.0 (Ubuntu)
Vary: Accept-Language, Cookie

2021/08/27 08:43:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
           0 2021-08-18 21:25:39        -1 Books
           0 2021-08-18 21:25:23        -1 Common
           0 2021-08-18 21:25:46        -1 Memories
 10752757984 2021-08-24 19:25:19        -1 Mobile
           0 2021-08-23 18:19:56        -1 Mobile
           0 2021-08-18 21:25:51        -1 Music
2021/08/27 08:43:26 DEBUG : 4 go routines active

system · October 27, 2021, 2:50am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.