Practical Limit to Union remotes?

Hi Devs,

I would like to know if there is a practical limit to Union remotes? Specifically I'm looking at Google Drive remotes.

For example, if I have 400 remotes under one union with an eprand policy, will this slow down or cause any problems?

On this setup, I also have another question, what happens if one remote becomes unaccessable (e.g. due to a 403 error), will that be ignored or will the request be retried with another remote?

And another question, can I have a union which is made up of sub unions? Will this cause issues?
For example, one union for drive accounts which is attached a union to replace mergerfs.
e.g. media union -> local media folder, union of drive remotes

Operations where it has to scan all 400 drives will be slow I think! Like reading the union.

If you were using rclone copy to the union then the initial upload would fail but the high level retry would choose a different drive.

If you were using a mount I don't think it would choose a different union (unless you used the new VFS branch).

The union isn't really designed to drop failing upstreams at the moment.

It should work. I haven't tried it though!

Thanks for the answers.

If you were using a mount I don't think it would choose a different union (unless you used the new VFS branch).

For reference at the moment I am using the new VFS branch (so I can use the full cache functionality).

Operations where it has to scan all 400 drives will be slow I think! Like reading the union.

Perhaps I misunderstood, but doesn't eprand union pick a random remote on each request? So how would 400 remotes make reading slow in this case?

When you list the union it will list all the upstreams to show you the union of the files won't it?

As part of doing an upload unless you are careful it will be listing the union.

It might be worth doing some experiments with a small number of remotes and -vv --dump headers to see the requests going out

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.