Does --no-traverse work differently when used with --fast-list

Help and Support
1

What is the problem you are having with rclone?

I was performing an rclone move and had been using the --fast-list and -no-traverse flag. I was wondering if using --no-traverse changes if I used it with vs without the --fast-list. Since --no-traverse only checks for the specified files and doesn't load the definition of all the files in the remote, I was wondering if using --fast-list speeds up that process.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

Which cloud storage system are you using? (eg Google Drive)

AWS S3

2

--fast-list will be used for the source remote I think.

If that is s3 then you'll get a speed up. If it is local then you won't since local doesn't support fast list.

3

Is there a way to know what flags are used for source remotes exclusively and what aren't? I came under the misconception that I could have used --fast-list for S3 which is my destination and I would like to be more aware on if/how I can catch myself. I may also be completely ignorant and missed obvious documentation.

4

hi,

based on the documentation, --fast-list will work with s3 as dest.
https://rclone.org/s3/#avoiding-get-requests-to-read-directory-listings

5

Got it, so would --no-traverse work differently based on whether --fast-list was used? Do you have any thoughts on my original question?

6

no idea, hard to take a general question and apply to a specific use-case.
maybe someone else knows.

too many variables, the source, the dest, the provider, the internet connection, latency, the flags, the values of the flags,
the config file, etc..

fwiw, for my given real world use-case, i run a few tests and then i know.

for example, even tho aws and wasabi are both s3,
the results are often very different, especially with large number of api calls.

7

i had already a test script for something like your case, so i tweaked it.
the source is local, a single dir with 100,000 files
the dest is s3, the exact same set of files.

rclone copy D:\files\100000 wasabi01:test100000 --no-traverse --fast-list --transfers=256 --checkers=256 -v --dry-run 
Elapsed time:        33.8s

rclone copy D:\files\100000 wasabi01:test100000 --no-traverse --transfers=256 --checkers=256 -v --dry-run
Elapsed time:        36.3s

rclone copy D:\files\100000 wasabi01:test100000 --fast-list --transfers=256 --checkers=256 -v --dry-run 
Elapsed time:        51.5s

rclone copy D:\files\100000 wasabi01:test100000 --transfers=256 --checkers=256 -v --dry-run 
Elapsed time:        54.9s
8

Nice test.

Note that fast list (despite the name) isn't always faster as it doesn't run in parallel whereas normal lists do.

The major effect here seems to be --no-traverse which is interesting.

No traverse is very efficient on s3 but hopeless on Google drive for example.

Anyway it is complicated enough that an actual test seems like the best way to evaluate.

You'd probably want to check how many API calls each one did too to get an idea of costs. --fast-list will help here and --no-traverse will hinder.

9

sure that you know this, but wanted to point out that,
wasabi, s3 clone known for hot storage, is different from aws s3.
--- does not charge for api calls.
--- does not charge for downloads.
--- does have a mandatory retention period

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.