Mount randomly missing directories

Suspected Bug
#1

What is the problem you are having with rclone?

rclone mount not showing all directories, this is at random with random root directories.
Verifying with ls path | wc -l which will show a different count when compared to rclone lsd | wc -l or against the same mount on another server.

Having this issue with multiple versions of rclone beta against multiple version of min.io.

Tried the mount without VFS cache and issue is still there. Currently running a mount with --drive-disable-http2 in the service file but too early to conclude if that helps.

What is your rclone version (output from rclone version)

rclone v1.54.0-beta.4977.a94e4d803

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Min.io / s3 compatible

The command you were trying to run (eg rclone copy /tmp remote:tmp)

ExecStart=/usr/bin/rclone mount \
        --config /home/USER/.config/rclone/rclone.conf \
        --async-read=false \
        --read-only \
        --allow-other \
        --dir-cache-time=6000h \
        --timeout 10m \
        --rc \
        --rc-no-auth \
        --fast-list \
        --low-level-retries 2 \
        --vfs-read-ahead 128M \
        --vfs-read-chunk-size-limit 2G \
        --cache-dir=/home/USER/.config/rclone/.vfscache \
        --vfs-cache-mode=full \
        --vfs-cache-max-age=24h \
        --vfs-cache-max-size=100G \
        --umask=002 \
        --log-level=DEBUG \
        --log-file=LOGLocation \
        minio:media /home/USER/mount

The rclone config contents with secrets removed.

[storage]
type = s3
provider = Minio
env_auth = false
region = eu
access_key_id = X
secret_access_key =X
endpoint = http://localhost
chunk_size = 50M
upload_concurrency = 16
list-chunk = 100000
disable_http2 = true

A log from the command with the -vv flag

Not showing up in the logs
#2

This means that rclone will only scan for new directories every 6000h since s3 doesn't support the change notify protocol (yet).

Either set that down to something much lower (5 minutes say) or scan manually using the remote control interface and vfs/refresh or vfs/forget

#3

I should have added it in the original post but the issue occurs while the mount is active and happens regardless 6000h or 1h dir-cache-time.

As an example ls ~/mount/Movies | wc -l gives me 13058 at the moment. At random times that number can be much lower into the 2000 suddenly. This is not remedied by using vfs/refresh or vfs/forget, those are run whenever new media is added to s3.

#4

That is a big difference.

Can you see if using rclone lsd on the movies directory does the same thing? Sometimes showing the wrong number of items.

Are items being added to the directory while the listing is going on? I wonder if that is the problem? Maybe you could experiment?

#5

You may also be hitting this issue in Minio: https://github.com/minio/minio/pull/11136

This was introduced with the build after 27th October and fixed very recently (15 days).

#6

@ncw rclone lsd shows the correct directory count, it's only the mount which does it.

@darthShadow Could be the issue as I run both the 27/10 build as a later one from end of dec. I've upgraded the December cluster to the latest version of minio so lets see.

Didn't connect the issue back to a possible regression in minio but hope it's solved.

#7

That looks very likely the problem.

As an aside, rclone uses the ListObjects v1 API which uses an actual key as the continuation parameter, whereas ListObjects v2 API uses an opaque "continuation parameter". This would almost certainly help in the case the bucket is being altered while rclone is listing it.

Worth switching do you think @darthShadow ? Did we discuss this already - I think we may have done!

#8

Yeah, we should definitely consider switching to it since per our discussion, it will result in smaller responses, even if nothing else improves, which is a positive gain for something that needs to be done eventually anyway.

However, I have been too busy at my work recently to spend any time on it or even respond to other posts or issues, as you may have noticed, and I am not sure when I will get some time to work on it.

The only reason I posted on this issue was that I faced this exact issue recently and spent quite a lot of time debugging the actual issue and then verifying the fix which is why I wanted to spare anyone else the headache for it.

1 Like
#9

After upgrading minio to latest and some testing, I now get random listings even in rclone lsd.

:~$ rclone lsd rmount:media/Movies | wc -l
10449
:~$ rclone lsd rmount:media/Movies | wc -l
10472
:~$ rclone lsd rmount:media/Movies | wc -l
13065
:~$ rclone lsd rmount:media/Movies | wc -l
10465

And a descrepancy again between mount and rclone lsd. What is strange though is that the files work fine when playing from Plex, if you scan though it will remove the missing media.

:~$ ls ~/mount/Movies | wc -l
1791
:~$ rclone lsd rmount:media/Movies | wc -l
13065

Edit: After more testing using mc ls it shows the same behaviour. For now I assume this is a issue with minio.

1 Like
#10

After some more testing this is an issue with minio, seems to be a long standing issue since they updated the list caching. While I thought it would be fixed by now, downgrading to a minio version from 27/10 has fixed it.

#11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.