Only 999 files show up when using mount on Windows

Suspected Bug
1

What is the problem you are having with rclone?

When using mount on an S3 bucket, folders that hold over 1000 files only show 999 in Windows explorer.

What is your rclone version (output from rclone version)

rclone v1.50.2
rclone v1.50.2-086-ga186284b-beta

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64bit
Windows Server 2016

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount bucket:bucketname X: --config="C:\Users\Username.config\rclone\rclone.conf"

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

The log is very large so I had to split it up between 2 in pastebin.
Logs have been updated to add the space back in.
https://pastebin.com/7c14SQSe
https://pastebin.com/uMs7uEeu

2

welcome to the forum,

i think it is premature to call this a bug. perhaps a question.

i use wasabi, which is a s3 compatible and i have no such problem

i created a .cmd batch file
For /L %%i in (1,1,2000) do fsutil file createnew test%%i.txt 1

and copied the 2000 files to wasabi

and use the this to mount
C:\data\rclone\scripts\rclone.exe mount wasabieast2:testnumfiles2000 b:\mount\testnumfiles2000\

and all 2000 files are there

image

if i do a dir B:\mount\testnumfiles2000\, i get
2000 File(s) 2,000 bytes

so if you do a dir x:\ what do you get?

3

Thanks for replying so quickly.

I followed your process and produced the same result as you, 2000 files showing in explorer.

I copied the files from my computer using the aws cli into this folder "/To Required/LAS/8-November-2019-Depot" and could only see 1000 files in this folder.

After a number of test's removing all possibilities I think I have solved the problem but it still seems like a bug.
I copied the files into this folder "/ToRequired/LAS/8-November-2019-Depot" and I can now see all 2000 files.

Seems like the space in the folder name was causing this problem.

In the original logs I replaced a company name with Required and did not leave a space in the name.

4

glad to help,

yeah, spaces are evil, i never, ever use them.

5

1000 files is the number of files rclone lists in a chunk which makes me think that this might be the same problem as https://github.com/rclone/rclone/issues/3799

So can you try this beta to see if it fixes it?

https://beta.rclone.org/branch/v1.50.2-085-g2dddcc52-fix-3799-s3-nextmarker-beta/

6

good news, the beta fixed the bug!

i was able to replicate the bug by creating 2000 files with a space in the file name.
For /L %%i in (1,1,2000) do fsutil file createnew "test %%i.txt" 1

then i did two mounts, pointing to the same folder, using two different versions of rclone.

rclone-v1.50.1.exe mount wasabieast2:testnumfiles2000 b:\mount\rclone-v1.50.1\
and
rclone-v1.50.2-085-g2dddcc52-fix-3799-s3-nextmarker-beta-windows-amd64.exe mount wasabieast2:testnumfiles2000 b:\mount\1.50.2-085-g2dddcc52-fix-3799-s3-nextmarker-beta-windows-amd64\

and then copied the files to one of the the mounts

image
image934×308 14 KB

7

Excellent! Thanks for testing.

I've merged the fix to master now which means it will be in the latest beta in 15-30 mins and released in v1.51