Rclone --max-size option doesn't work correctly with aws s3

Suspected Bug
1

What is the problem you are having with rclone?

Rclone ls and opy --max-size option doesn't work correctly with aws s3 bucket. When I use the option --max-size 2B it shows files that are much larger that 2 bytes

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.1

  • os/version: ubuntu 24.04 (64 bit)
  • os/kernel: 6.8.0-55-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.24.0
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

aws s3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls remote: --max-size 2B

The rclone config contents with secrets removed.

Results of the rclone config command:

Current remotes:

Name                 Type
====                 ====
aws                  s3

A log from the command with the -vv flag

The size of the log is huge, but basically it looks like that

2025/03/07 15:32:03 DEBUG : folder/subfolder/website.html: Excluded (Size Filter)
      304 folder/README.md

Note that README.md isn't excluded even though its size is 304.

2

Thank you for reporting!

I can confirm the same issue on local drive, so it is not S3 specific:

$ rclone ls . --max-size 2B
      276 kptsky.mouse-battery-check.log
     1250 kptsky.wakeup.scheduler.log

I think you could open a bug report on github. It definitely does not work as it should.

It worked in v1.67.0 and is broken since v1.68.0 including the latest beta.

Maybe you could do git-bisect? Here a high level how to:

git checkout master
git bisect start 
git bisect bad
git checkout v1.67.0
git bisect good

# now build rclone and test it
go build
./rclone ls . --max-size 2B

# if it worked
git bisect good

# if it failed
git bisect bad

# git will choose another commit
# go back to the go build step

# eventually git will stop and print the offending commit

# at the end
git bisect reset
3

Ahh well, sometime DIY is needed:)

4

imho,

this is a very serious bug that seems to affect all storage systems.
should to be fixed now and released as v1.69.2

5

I wish I knew how to fix it. I've poked around source code but do not understand what is going on here. Some hardcode golang expertise is needed here.

6

me too

the bug does not seem to affect --min-age and --max-age
tho, does affect --min-size and --max-size

rclone ls d:\zork -vv --max-age=1d   --dump=filters 
DEBUG : --max-age 1d to 2025-03-08 10:48:13.2575181 -0500 EST m=-86399.872531199
--- start filters ---
Last-modified date must be equal or greater than: 2025-03-08 10:48:13.2575181 -0500 EST m=-86399.872531199
--- File filter rules ---
--- Directory filter rules ---
--- end filters ---

rclone ls d:\zork -vv --min-size=1B --dump=filters
--- start filters ---
--- File filter rules ---
--- Directory filter rules ---
--- end filters ---

rclone ls d:\zork -vv --max-size=1B  --dump=filters 
--- start filters ---
--- File filter rules ---
--- Directory filter rules ---
--- end filters ---
8

I think --min-size and --max-size does not list anything in dumped filters and never was.

Maybe it is another issue?

9

This bug only affects --min-size and --max-size with a B suffix. It is proving a little tricky to fix but you can use a workaround --min-size 0.001k for --min-size 1B.

10

This could be documented and we could live without B. However this bug potentially affects existing scripts. Which is not good.

Also your workaround can be tricky for many people. Especially that for example 0.1K is 102B… not 100B.

I have not checked my scripts but I use B a lot. Regardless of size. I can display values in human readable format but to avoid accumulating errors during some more complex calculations B is the best choice to store values and use in rclone.

11

I've posted a fix for this on the GitHub issue

Testing appreciated, and also discussion on the future of the B suffix in rclone v2.0.

2 Likes
12

i did some quick testing, everything looks ok to me.

13

My tests show that it is fixed.

Not sure what your plan here is. IMO for completeness B suffix should be supported. In ideal world size value without suffix should represent B (not K as it is now) but it would be breaking change not bringing any specific value.

14

Yes that is what I think too. It would be a breaking change so rclone v2 - what do you think?

Also what do you think about an warning message for use of unsuffixed numbers which is almost always a mistake.

15

Short term it would be nice thing to have. Some mistakes could be avoided thx to it.

Most humans (excluding some geeks) always interpret value without specific unit as corresponding to the lowest unit available. So if we allow B it should be B.

Some Unix programs (rsync, df, du) for whatever historical reasons behave differently. But it is not really an argument for rclone to do the same.

Now I am thinking that bumping rclone version to 2.0 might be very good opportunity to get rid off many legacy things like B suffix one.

On the side note I would make sure that rclone 1.x should not "rclone selfupdate" to 2.0. Especially if there are breaking changes.

16

I keep meaning to make a page with things to change for any notional v2.0 so I made a start here: Breaking changes for v2 · rclone/rclone Wiki · GitHub

Anything you'd like to add? There are probably lots of weird things which could be done better!

1 Like
17

I've merged this fix to master now which means it will be in the latest beta in 15-30 minutes and released in v1.70

2 Likes