rclone mount not showing all directories, this is at random with random root directories.
Verifying with ls path | wc -l which will show a different count when compared to rclone lsd | wc -l or against the same mount on another server.
Having this issue with multiple versions of rclone beta against multiple version of min.io.
Tried the mount without VFS cache and issue is still there. Currently running a mount with --drive-disable-http2 in the service file but too early to conclude if that helps.
What is your rclone version (output from rclone version)
rclone v1.54.0-beta.4977.a94e4d803
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Debian 10, 64 bit
Which cloud storage system are you using? (eg Google Drive)
I should have added it in the original post but the issue occurs while the mount is active and happens regardless 6000h or 1h dir-cache-time.
As an example ls ~/mount/Movies | wc -l gives me 13058 at the moment. At random times that number can be much lower into the 2000 suddenly. This is not remedied by using vfs/refresh or vfs/forget, those are run whenever new media is added to s3.
@ncw rclone lsd shows the correct directory count, it's only the mount which does it.
@darthShadow Could be the issue as I run both the 27/10 build as a later one from end of dec. I've upgraded the December cluster to the latest version of minio so lets see.
Didn't connect the issue back to a possible regression in minio but hope it's solved.
As an aside, rclone uses the ListObjects v1 API which uses an actual key as the continuation parameter, whereas ListObjects v2 API uses an opaque "continuation parameter". This would almost certainly help in the case the bucket is being altered while rclone is listing it.
Worth switching do you think @darthShadow ? Did we discuss this already - I think we may have done!
Yeah, we should definitely consider switching to it since per our discussion, it will result in smaller responses, even if nothing else improves, which is a positive gain for something that needs to be done eventually anyway.
However, I have been too busy at my work recently to spend any time on it or even respond to other posts or issues, as you may have noticed, and I am not sure when I will get some time to work on it.
The only reason I posted on this issue was that I faced this exact issue recently and spent quite a lot of time debugging the actual issue and then verifying the fix which is why I wanted to spare anyone else the headache for it.
And a descrepancy again between mount and rclone lsd. What is strange though is that the files work fine when playing from Plex, if you scan though it will remove the missing media.
After some more testing this is an issue with minio, seems to be a long standing issue since they updated the list caching. While I thought it would be fixed by now, downgrading to a minio version from 27/10 has fixed it.