Rclone mount with compatible s3: all folders (except root folder) missing

murgen · December 30, 2022, 10:22am

What is the problem you are having with rclone?

I'm mounting a s3 compatible bucket (iDrive e2) with rclone.
Only the root folder is showing with rclone lsd, but it's empty (even if it's not)

rclone lsd idrive-e2-navidrome:navidrome-music -R
0 2022-12-30 10:02:54        -1 Music

But the weird thing is that all folders are listed when i run rclone lsf idrive-e2-navidrome:navidrome-music -R -d
I updated rclone but still not showing anything.
Note: It was working just fine yesterday, I didn't do anything in the meantime, except updating rclone once I realized nothing was mounted except the root folder, empty.

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.61.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.13.0-1027-oracle (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

iDrive e2 (s3 compatible)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

I run it through a systemctl service

[Unit]
Description=rclone mount
Documentation=http://rclone.org/docs/
After=network.service 
StartLimitInterval=500
StartLimitBurst=5

[Service]
Type=notify
Restart=on-failure
RestartSec=5
ExecStart=rclone mount idrive-e2-navidrome:navidrome-music /home/ubuntu/docker/navidrome/data/music \
	--log-level INFO \
	--rc \
	--rc-no-auth \
	--allow-other \
	--buffer-size 3G \
	--dir-cache-time 4h \
	--read-only \
	--vfs-cache-mode full \
	--vfs-fast-fingerprint \
	--vfs-cache-poll-interval 4h\
	--vfs-cache-max-size 170G \
	--vfs-cache-max-age 9999h \
	--vfs-read-ahead 128M \
	--vfs-read-chunk-size 16M \
	--vfs-read-chunk-size-limit 128M \
	--exclude="Kara/**" \
	--exclude="Music/Divers/**" \
	--exclude="*.mpc" \
	--exclude="*.mpg" \
	--exclude="*.avi" \
	--exclude="*.mpeg" \
	--exclude="*.zip" \
	--exclude="*.rar" \
	--exclude="*.mkv"
ExecStop=/bin/fusermount -uz /home/ubuntu/docker/navidrome/data/music
ExecStartPost=docker-compose -f /home/ubuntu/docker/navidrome/docker-compose.yml up -d 
ExecStopPost=docker-compose -f /home/ubuntu/docker/navidrome/docker-compose.yml down

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[idrive-e2-navidrome]
type = s3
provider = Other
access_key_id = REDACTED
secret_access_key = REDACTED
endpoint = o0v1.fra.idrivee2-20.com
acl = private

A log from the command with the `-vv` flag

2022/12/30 10:17:59 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "mount" "idrive-e2-navidrome:navidrome-music" "/home/ubuntu/docker/navidrome/data/music" "-vv" "--rc" "--rc-no-auth" "--allow-other" "--buffer-size" "3G" "--dir-cache-time" "4h" "--read-only" "--vfs-cache-mode" "full" "--vfs-fast-fingerprint" "--vfs-cache-poll-interval" "4h" "--vfs-cache-max-size" "170G" "--vfs-cache-max-age" "9999h" "--vfs-read-ahead" "128M" "--vfs-read-chunk-size" "16M" "--vfs-read-chunk-size-limit" "128M" "--exclude=Kara/**" "--exclude=Music/Divers/**" "--exclude=*.mpc" "--exclude=*.mpg" "--exclude=*.avi" "--exclude=*.mpeg" "--exclude=*.zip" "--exclude=*.rar" "--exclude=*.mkv"]
2022/12/30 10:17:59 NOTICE: Serving remote control on http://127.0.0.1:5572/
2022/12/30 10:17:59 DEBUG : Creating backend with remote "idrive-e2-navidrome:navidrome-music"
2022/12/30 10:17:59 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2022/12/30 10:17:59 INFO  : S3 bucket navidrome-music: poll-interval is not supported by this remote
2022/12/30 10:17:59 DEBUG : vfs cache: root is "/home/ubuntu/.cache/rclone"
2022/12/30 10:17:59 DEBUG : vfs cache: data root is "/home/ubuntu/.cache/rclone/vfs/idrive-e2-navidrome/navidrome-music"
2022/12/30 10:17:59 DEBUG : vfs cache: metadata root is "/home/ubuntu/.cache/rclone/vfsMeta/idrive-e2-navidrome/navidrome-music"
2022/12/30 10:17:59 DEBUG : Creating backend with remote "/home/ubuntu/.cache/rclone/vfs/idrive-e2-navidrome/navidrome-music"
2022/12/30 10:17:59 DEBUG : Creating backend with remote "/home/ubuntu/.cache/rclone/vfsMeta/idrive-e2-navidrome/navidrome-music"
2022/12/30 10:18:06 DEBUG : S3 bucket navidrome-music: Mounting on "/home/ubuntu/docker/navidrome/data/music"
2022/12/30 10:18:06 DEBUG : : Root: 
2022/12/30 10:18:06 DEBUG : : >Root: node=/, err=<nil>
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Chinese Man/2009 - The Groove Sessions Vol.2/11 - Our Time (feat. Franco).m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Infected Mushroom/the silent doof/01 - Merlin (Global Cut Mix).mp3 not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Korpiklaani/2003 - Spirit of The Forest/03 - God of Wind.mp3 not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/Ghibli/_Compilations/[2014-07-16] Studio Ghibli Miyazaki Hayao & Joe Hisaishi Soundtrack Box/D10 Gake no Ue no Ponyo/02 - Umi no Okaasan.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Faunts/2009 - Feel.Love.Thinking.Of/10 - Explain.mp3 not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Jónsi/MANH(A)TTAN Original Score From the Television Series/1.01 - Manhattan Main Title.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Wu-Tang Clan/2013 - The Essential Wu-Tang Clan/1.09 - Gravel Pit.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Final Fantasy/Final Fantasy 15/[2016-11-29] FINAL FANTASY XV ULTIMATE COLLECTOR'S EDITION SPECIAL SOUNDTRACK/3.14 - Winds of Change.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Final Fantasy/Final Fantasy 7/[2020-09-24] SQUARE ENIX JAZZ -FINAL FANTASY VII- at Billboard Live TOKYO/1.03 - LIVE- Cait Sith's Theme Jazz Arrangement.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Lunar/[1995-02-22] LUNAR ~ETERNAL BLUE/08 - Seitou Pentagulia.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/FictionJunction/2004 - FJY - akatsuki no kuruma/03 - akatsuki no kuruma ~acoustic version.mp3 not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/Azumanga Daioh/[2002-10-02] TRIBUTE TO AZUMANGA-DAIOH/07 - moi moi.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/Kaizoku Oujo (Fena Pirate Princess)/[2021-12-08] TV Animation Kaizoku Oujo ORIGINAL SOUNDTRACK/1.02 - Umi to Shinju (TV-Edit Japanese Ver).m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/The Promised Neverland/[2021-02-24] THE PROMISED NEVERLAND Season 1&2 Original Soundtrack/2.11 - Nat King Cool.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/Steins;Gate/[2018-09-19] TV Animation STEINS;GATE 0 Original Soundtrack/1.05 - Hiyajou Maho.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Edge of Eternity OST/71 - Song of Eternity.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Radiohead/2007 - In Rainbows/1.03 - Nude.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/The Legend of Zelda/_Fan Albums/Rozen - 2019 - Children of Termina/16 - Song of Healing.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/86 EIGHTY-SIX/[2021-10-27] 86 EIGHTY-SIX REARRANGE & OUTTRACKS CD/10 - Giad Renpou.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/Gundam/MOBILE SUIT GUNDAM UNICORN/[2010-03-10] MOBILE SUIT GUNDAM UNICORN ORIGINAL SOUNDTRACK/21 - H@R0.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/Sword Art Online/[2016-01-27] Sword Art Online Music Collection/04.22 - you are not alone #3.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Bonobo/2013 - The North Borders/14 - Heaven.m4a not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Orbital/1996 - In Sides/07 - Out There Somewhere Part 1.mp3 not removed, freed 0 bytes
2022/12/30 10:18:06 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Half-Life/[2004-11-16] Half-Life 2/19 - Lab Practicum.m4a not removed, freed 0 bytes
[...same for lots of files]
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Zoxea/1999 - À mon tour d'briller/03 - Rap, musique que j'aime.m4a not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Chrono Series/_Fan Albums/Materia Collective - 2019 - EPOCH- A Tribute to Chrono Trigger/24 - Jolly Ol' Spekkio.m4a not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Denpa Men/The Denpa Men 3 - The Rise of Digitoll (3DS Gamerip)/19 - ELEGY.mp3 not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Final Fantasy/Final Fantasy 15/[2019] Final Fantasy XV Episode Ardyn Original Soundtrack (GameRip)/16 - Resist your Fate.mp3 not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Ys/Ys 3/[2005-09-23] Ys -THE OATH IN FELGHANA- ORIGINAL SOUND TRACK/2.04 - Sealed Time.m4a not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Rurutia/2006 - Chorion/04 - mizu keshiku,  hoshi mouyou.mp3 not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/pg.lost/pg.lost - yes i am/05 - the kind heart of lanigon.mp3 not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Animes/Shingetsutan Tsukihime/[2003-12-10] Moonlit archives Shingetsutan Tsukihime Original Sound Track 1/25 - Rinne no Hate ni... (ON AIR Ver.).m4a not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Another Eden/[2022-04-08] ANOTHER EDEN Concert Album for fragile time/1.16 Delicate Frozen Floret.m4a not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Caroline/2006 - Murmurs/05 - Everylittlething.mp3 not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Musique generale/Mikuni Shimokawa/1999 - 2000EXPRESS (single)/01 - 2000EXPRESS.m4a not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/LIVE A LIVE/[2012-05-02] LIVE A LIVE Original Soundtrack/01 - Live-A-Live.m4a not removed, freed 0 bytes
2022/12/30 10:18:08 DEBUG : vfs cache RemoveNotInUse (maxAge=35996400000000000, emptyOnly=false): item Music/Soundtracks/Games/Metal Gear/2006.12.22 - Metal Gear Solid Portable Ops Original Soundtrack/26 - Before Dawn.m4a not removed, freed 0 bytes
2022/12/30 10:18:08 INFO  : vfs cache: cleaned: objects 62919 (was 62919) in use 0, to upload 0, uploading 0, total size 169.994Gi (was 169.994Gi)

ncw · December 30, 2022, 11:28pm

Can you run this with -vv --dump bodies and post the output please? The log of the mount isn't very useful debugging this problem.

murgen · December 31, 2022, 6:48am

Here:

rclone lsd idrive-e2-navidrome:navidrome-music -vv --dump bodies
2022/12/31 06:47:32 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "lsd" "idrive-e2-navidrome:navidrome-music" "-vv" "--dump" "bodies"]
2022/12/31 06:47:32 DEBUG : Creating backend with remote "idrive-e2-navidrome:navidrome-music"
2022/12/31 06:47:32 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2022/12/31 06:47:32 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/12/31 06:47:32 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/12/31 06:47:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/12/31 06:47:32 DEBUG : HTTP REQUEST (req 0x400085a200)
2022/12/31 06:47:32 DEBUG : GET /navidrome-music?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: o0v1.fra.idrivee2-20.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221231T064732Z
Accept-Encoding: gzip

2022/12/31 06:47:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/12/31 06:47:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/12/31 06:47:32 DEBUG : HTTP RESPONSE (req 0x400085a200)
2022/12/31 06:47:32 DEBUG : HTTP/1.1 200 OK
Content-Length: 358
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 31 Dec 2022 06:47:32 GMT
Server: nginx
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 1735CDEC77269697
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>navidrome-music</Name><Prefix></Prefix><KeyCount>1</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>Music/</Prefix></CommonPrefixes><EncodingType>url</EncodingType></ListBucketResult>
2022/12/31 06:47:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
           0 2022-12-31 06:47:32        -1 Music
2022/12/31 06:47:32 DEBUG : 4 go routines active

ncw · December 31, 2022, 9:36am

That appeared to work.

What were you expecting to see? rclone lsd lists the folders in the root. Are there more folders in the root?

If you do

rclone lsf idrive-e2-navidrome:navidrome-music/Music -vv

Do you see what you expect?

Does the output of

rclone lsf -R idrive-e2-navidrome:navidrome-music

differ from

rclone lsf -R --disable ListR idrive-e2-navidrome:navidrome-music

(as in are there things missing - I don't care about the order).

murgen · December 31, 2022, 9:52am

Yep you're right, but actually the command that bugged me was rclone lsd idrive-e2-navidrome:navidrome-music -R
I expect it to list all the directories, not just Music

here's the output:

rclone lsd idrive-e2-navidrome:navidrome-music -R -vv --dump bodies
2022/12/31 09:46:08 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "lsd" "idrive-e2-navidrome:navidrome-music" "-R" "-vv" "--dump" "bodies"]
2022/12/31 09:46:08 DEBUG : Creating backend with remote "idrive-e2-navidrome:navidrome-music"
2022/12/31 09:46:08 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2022/12/31 09:46:08 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/12/31 09:46:08 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/12/31 09:46:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/12/31 09:46:08 DEBUG : HTTP REQUEST (req 0x40008d6100)
2022/12/31 09:46:08 DEBUG : GET /navidrome-music?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: o0v1.fra.idrivee2-20.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221231T094608Z
Accept-Encoding: gzip

2022/12/31 09:46:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/12/31 09:46:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/12/31 09:46:08 DEBUG : HTTP RESPONSE (req 0x40008d6100)
2022/12/31 09:46:08 DEBUG : HTTP/1.1 200 OK
Content-Length: 358
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 31 Dec 2022 09:46:08 GMT
Server: nginx
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 1735D7AB77346544
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>navidrome-music</Name><Prefix></Prefix><KeyCount>1</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>Music/</Prefix></CommonPrefixes><EncodingType>url</EncodingType></ListBucketResult>
2022/12/31 09:46:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
           0 2022-12-31 09:46:08        -1 Music
2022/12/31 09:46:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/12/31 09:46:08 DEBUG : HTTP REQUEST (req 0x40004b6400)
2022/12/31 09:46:08 DEBUG : GET /navidrome-music?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix=Music%2F HTTP/1.1
Host: o0v1.fra.idrivee2-20.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221231T094608Z
Accept-Encoding: gzip

2022/12/31 09:46:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/12/31 09:46:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/12/31 09:46:08 DEBUG : HTTP RESPONSE (req 0x40004b6400)
2022/12/31 09:46:08 DEBUG : HTTP/1.1 200 OK
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Sat, 31 Dec 2022 09:46:08 GMT
Server: nginx
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 1735D7AB785965CA
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>navidrome-music</Name><Prefix>Music/</Prefix><KeyCount>8</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>Music/Asian+Music/</Key><LastModified>2022-09-21T06:25:48.078Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Music/Classique/</Key><LastModified>2022-09-21T06:25:58.048Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Music/Danse/</Key><LastModified>2022-09-21T06:26:03.427Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Music/Divers/</Key><LastModified>2022-09-21T06:26:09.578Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Music/Kara/</Key><LastModified>2022-09-21T06:26:09.919Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Music/Musique+generale/</Key><LastModified>2022-09-21T06:26:09.919Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Music/Soundtracks/</Key><LastModified>2022-09-21T06:33:19.992Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Music/_navidrome_playlists/</Key><LastModified>2022-11-23T17:23:09.503Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>945bb878c26a11ecb21c3cecef7cca14945bb878c26a11ecb21c3cecef7cca14</ID><DisplayName>o0v1.fra.idrivee2-20.com</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><EncodingType>url</EncodingType></ListBucketResult>
2022/12/31 09:46:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/12/31 09:46:08 DEBUG : 4 go routines active

and here, it seems to list in the ListBucketResult xml at least the 1st level of directories. It's not if I run this command without --dump bodies

rclone lsd idrive-e2-navidrome:navidrome-music -R -vv
2022/12/31 09:44:16 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "lsd" "idrive-e2-navidrome:navidrome-music" "-R" "-vv"]
2022/12/31 09:44:16 DEBUG : Creating backend with remote "idrive-e2-navidrome:navidrome-music"
2022/12/31 09:44:16 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
           0 2022-12-31 09:44:16        -1 Music
2022/12/31 09:44:16 DEBUG : 4 go routines active

Like I said in my first post, lsf is working fine.

Actually, the real issue is that even if lsf is listing all the files, I still can't see them mounted:

ls -alsh ~/docker/navidrome/data/music/Music/
total 0
0 drwxrwxr-x 1 ubuntu ubuntu 0 Dec 31 09:42 .
0 drwxrwxr-x 1 ubuntu ubuntu 0 Dec 31 09:42 ..

rclone lsf -R idrive-e2-navidrome:navidrome-music ouputs all the files that are supposed to be mounted

rclone lsf -R --disable ListR idrive-e2-navidrome:navidrome-music outputs that:

rclone lsf -R --disable ListR idrive-e2-navidrome:navidrome-music
Music/

ncw · December 31, 2022, 10:33am

I'm not sure what is causing lsd vs lsf differences - they should be the same as they use the same code.

However since you see the files with lsf lets work on the mount.

Not seeing any files is usually a consequence of mounting the mount as a different user to you and not using the --allow-other flag.

You appear to be using the allow other flag, but is it enabled in /etc/fuse.conf ?

murgen · December 31, 2022, 1:46pm

cat /etc/fuse.conf
# /etc/fuse.conf - Configuration file for Filesystem in Userspace (FUSE)

# Set the maximum number of FUSE mounts allowed to non-root users.
# The default is 1000.
#mount_max = 1000

# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other

ncw · December 31, 2022, 7:02pm

Try mounting a local directory instead of S3 and see if that works. Put the absolute path to an existing directory instead of idrive-e2-navidrome:navidrome-music

That should tell us whether this is a mount problem or an S3 problem

murgen · January 1, 2023, 8:56am

It's working just fine with a local directory

Btw, happy new year

ncw · January 1, 2023, 9:38am

Ok! We need to find a command which replicates the problem without mount.

What did this one do? Did it replicate the problem? This is the closest to how mount works.

murgen · January 2, 2023, 12:38pm

Yep it did replicate the problem. If I understand correctly, it should list files recursively, but only the root directory is listed.

ncw · January 3, 2023, 9:15am

Great. That will make it much easier to debug.

Can you now run

rclone lsf -R --disable ListR idrive-e2-navidrome:navidrome-music -vv --dump bodies

And post the output please?

murgen · January 8, 2023, 10:05am

Sorry for my late reply, I had a busy week.
I just tried, and for reason unknown, it seems to be correctly mounted now o_O
I can list and browse pretty much every folders and subfolders, so I guess the issue was on server side ?
Anyway, thanks for your help and have a nice day !

system · February 7, 2023, 10:05am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.