No-traverse for Dummies

jxer · June 26, 2017, 2:25am

Can someone please explain the no traverse flag in a dummies format. I have read the doc descriptions and its still fuzzy for me with rclone clopy

examples would be great!

ncw · June 28, 2017, 5:05pm

Firstly rclone works exactly the same with or without --no-traverse - it is entirely an optimization.

So the for dummies suggestion would be - try it with and without --no-traverse and use the faster method.

It will generally be faster if you are only copying (or moving) a small number of files into a folder with a large number of files.

If there are about the same number of files in the source and destination then don’t use --no-traverse.

I hope that helps a bit!

jxer · June 28, 2017, 5:27pm

@ncw

it does

can you provide a slightly more technical answer?

ncw · June 29, 2017, 12:47pm

Technically speaking…

Let’s say you have a destination with 6 files {a,b,c,d,e,f}.

If you are copying {a} to the destination then without no-traverse, rclone will load in the definitions for all the files {a,b,c,d,e,f} before discovering whether {a} needs to be uploaded. If you use --no-traverse rclone will just check {a} on the remote.

So why wouldn’t you use --no-traverse all the time?

If you are copying {a,b,c,d,e,f} to the destination, then rclone will check each file individually. This will take at least 6 transactions, wheras likely you could have got the listing for all the objects done in 1 listing.

So there are tradeoffs! The new sync method implemented in 1.36 makes --no-traverse less useful than it used to be, but it can still come in handy, especially if you are moving or copying files into a deep hierarchy.

calisro · July 6, 2017, 2:01pm

I’d suggest having a ‘optimization’ page or even just enhancing some of the existing pages on the wiki that explains some of these flags. We now also have the ‘fast-list’ flag. All of these are not well understood but can have dramatic effects on the sync process. We see lots of people asking about them and I see lots of people just using them ‘because someone said it helps’. The forum is great for these type of questions but the answers slowly get washed away with time.

Or perhaps a user editable wiki.

ncw · July 6, 2017, 2:08pm

That is a great idea!

I have put docs about --fast-list in the relevant remotes pages. But --no-traverse doesn’t fit there.

Also I don’t know if I’ve explained that reading the mod time on swift & s3 uses an extra HTTP transaction, so --size-only may be a lot faster.

I’d love to have suggestions for text if you have some? I made a wiki page to write stuff if you have some suggestions.

I’d like to formalize suggestions into a new page on the website or existing pages where appropriate.

calisro · July 6, 2017, 2:31pm

For just this minute I added the text for no-traverse (It is your explanation as I thought it was well explained) but as I come across content in the forum, i’ll make it a habit to add it to a relevant section in the wiki with a link in the post to the wiki page. I see the same things discussed over and over again and it would be nice to have it somewhere.

An “Example Script Snippet” section might also prove worthy. One such which I just saw was how to check for existence of a running rclone before executing the next out of cron.