Rclone ls listing files twice

What is the problem you are having with rclone?

"rclone ls" from a S3 object storage listed files twice

We have a bucket with a few millions of objects and we were listing them to make sure that we copied everything that we wanted (and we keep it for future reference).

It has 4 directories with a bit less than 1 million files.

We did "rclone ls" and then we were doing diff between the source and the destination. We saw that the number of files didn't match. After a closer look we saw that some files were listed twice. I could do "grep dir1/file_name.bin list_of_files.txt" and it would appear twice.

I'm sure that:
a) only one rclone generated list_of_files.txt (not two rclones with >>)
b) it was a section (or multiple?) of many files listed twice, not only one file
c) no other rclone/changes in the bucket during the listing

Because it takes time to do the list but specially some money to do this big list on S3 we haven't tried to reproduce it. We had another set of tools that we wrote before we knew rclone and we used them instead (done in Python) for the verification (which I like, in case that rclone had a bug at copy time and listing time and a file was missing from the copy and the list).

I don't expect a fix for this with this vague description, only sharing this in case that someone else finds a similar problem would know that the person is not the first one. Because all the files were listed is not a problem of missing files (unless someone only compares the total number of files and will be confused).

What is your rclone version (output from rclone version)

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Red Hat Enterprise Linux Server release 7.7 (Maipo)

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls bucket
(I don't have the exact command)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Not available :frowning:

I'm pretty sure this is fixed in the latest beta - can you give that a go?

Hi nick,
I'll pay attention on our next "rclone ls" - sadly it's a bucket with many files and we are avoiding to do big listing if we don't need them there :slight_smile:
Thanks very much! We will keep our eyes opened

1 Like