Object key remote directory filter clarification

[Edward_Barker (Edward)]

I do not use object base remotes and therefore can't easily test this.

Is the following true?

Rclone uses directory filter matches to decide whether to recurse into
directories, and to optimise access to a remote, but only when the remote has
a concept of directories (eg local, Google Drive and Microsoft OneDrive
remotes). Neither directory optimisation nor recursion blocking is
performed for object key based remotes such as Amazon S3, BackBlaze,
Google Compute Storage or Swift.

Essentially on an object based remote I can see rclone could not use a directory filter for access optimisation, but does rclone still use it to block recursion into the directory or not?

The rest of the para is still very much draft around that issue.

[ncw (Nick Craig-Wood)]

@Edward_Barker

I do not use object base remotes and therefore can't easily test this.

The :memory: remote is an object remote. Testing command line stuff against it is hard though since it forgets everything on every command invocation!

@Edward_Barker

Rclone uses directory filter matches to decide whether to recurse into
directories, and to optimise access to a remote

So far so good

@Edward_Barker

but only when the remote has
a concept of directories (eg local, Google Drive and Microsoft OneDrive
remotes).

That is not entirely true

@Edward_Barker

Neither directory optimisation nor recursion blocking is
performed for object key based remotes such as Amazon S3, BackBlaze,
Google Compute Storage or Swift.

Nor is that.

The optimization we are talking about is not recursing into dir if rclone can prove to itself that it doesn't need to because the filters exclude everything in dir.

This is done for all remotes if not using the ListR (list recursive) primitive.

ListR will be used if the remote supports it and you specify --fast-list.

ListR will also be used for naturally recursive rclone calls, eg rclone ls, rclone lsf -R, rclone size etc.

ListR will not be used for sync/copy/move unless you specify --fast-list.

Which remotes support ListR is in the overview table - you'll see this is mostly the bucket based ones, but there are some exceptions like Google Drive.

Now that was a lot to take in - probably too much for this doc. Maybe something like

Rclone uses directory filter matches to decide whether to recurse into
directories, and to optimise access to a remote so it doesn't list directories it doesn't have to. This optimisation is used unless rclone is listing the remote recursively. Recursive lists are used for rclone ls and related commands or sync/copy/move with --fast-list. You can disable recursive lists with --disable ListR.

What do you think?

[Edward_Barker (Edward)]

@ncw

The :memory: remote is an object remote. Testing command line stuff against it is hard though since it forgets everything on every command invocation!

Found that problem! Testing directory filters in general is a pain.

These comments will be a great help once I have got through them.

Many thanks.