No-traverse for Dummies

Can someone please explain the no traverse flag in a dummies format. I have read the doc descriptions and its still fuzzy for me with rclone clopy

examples would be great!

Firstly rclone works exactly the same with or without --no-traverse - it is entirely an optimization.

So the for dummies suggestion would be - try it with and without --no-traverse and use the faster method.

It will generally be faster if you are only copying (or moving) a small number of files into a folder with a large number of files.

If there are about the same number of files in the source and destination then don’t use --no-traverse.

I hope that helps a bit!

@ncw

it does

can you provide a slightly more technical answer?

Technically speaking…

Let’s say you have a destination with 6 files {a,b,c,d,e,f}.

If you are copying {a} to the destination then without no-traverse, rclone will load in the definitions for all the files {a,b,c,d,e,f} before discovering whether {a} needs to be uploaded. If you use --no-traverse rclone will just check {a} on the remote.

So why wouldn’t you use --no-traverse all the time?

If you are copying {a,b,c,d,e,f} to the destination, then rclone will check each file individually. This will take at least 6 transactions, wheras likely you could have got the listing for all the objects done in 1 listing.

So there are tradeoffs! The new sync method implemented in 1.36 makes --no-traverse less useful than it used to be, but it can still come in handy, especially if you are moving or copying files into a deep hierarchy.

I’d suggest having a ‘optimization’ page or even just enhancing some of the existing pages on the wiki that explains some of these flags. We now also have the ‘fast-list’ flag. All of these are not well understood but can have dramatic effects on the sync process. We see lots of people asking about them and I see lots of people just using them ‘because someone said it helps’. :slight_smile: The forum is great for these type of questions but the answers slowly get washed away with time. :eye:

Or perhaps a user editable wiki.

That is a great idea!

I have put docs about --fast-list in the relevant remotes pages. But --no-traverse doesn’t fit there.

Also I don’t know if I’ve explained that reading the mod time on swift & s3 uses an extra HTTP transaction, so --size-only may be a lot faster.

I’d love to have suggestions for text if you have some? I made a wiki page to write stuff if you have some suggestions.

I’d like to formalize suggestions into a new page on the website or existing pages where appropriate.

For just this minute I added the text for no-traverse (It is your explanation :slight_smile: as I thought it was well explained) but as I come across content in the forum, i’ll make it a habit to add it to a relevant section in the wiki with a link in the post to the wiki page. I see the same things discussed over and over again and it would be nice to have it somewhere.

An “Example Script Snippet” section might also prove worthy. One such which I just saw was how to check for existence of a running rclone before executing the next out of cron.