Advantage of new union remote?

[BinsonBuzz]

Yes, that would work perfectly.

I still think moves from location-0 to location-1 should be parked and considered in phase 2, as I think the existing rclone move is good enough.

[jrock]

Yes I would be quite happy to manage that part manually as well if it is going to be too onerous to build initially.

[ncw (Nick Craig-Wood)]

If I delete a file (from the highest priority remote, whose file is the one that is visible) then also delete that filename from other remotes that have the delete option enabled.

I see - thank you for explaining.

Looking at your updated schema I think moves look good.

• Updated files: If the file is opened and saved, or replaced with a file of a larger/smaller size (same filename):

1. If the original file exists only in location-0: just overwrite the original file in location-0
2. If the original file exists only in location-1: delete the file in location-1 and write it to location-0
3. If the original file exists in both locations: delete the file from location-1 and overwrite the file in location-0.

I think my preference here would be not to delete the file in step 2 or step 3, but rely on the rclone move to replace the file properly. This means that if anything goes wrong you still have the old file.

[jrock]

Yes that would work, agree it is safer. And slightly less latency at runtime too.

[ncw (Nick Craig-Wood)]

Yes that would work, agree it is safer. And slightly less latency at runtime too.

Could one of you please make a clean copy of the design so far into a new issue on github (link to this thread too) then we can have a think about implementation

[jrock]

I will have a go at that.

[BinsonBuzz]

Brilliant! @ncw Thanks for adding this to the backlog as it would solve my biggest outstanding issue, which is ditching unionfs as the extra IO from lack of hard link support (and I also suspect just moving files from other local paths to location-0) is killing my server setup

[jrock]

Github issue created. Sorry it took a few days, got busy.

[ncw (Nick Craig-Wood)]

Thanks for doing that

[distinguished-git]

mergerfs allows “random” selection of branches for search-type actions.
i.e., access, getattr, getxattr, ioctl, listxattr, open, readlink

In theory, this would allow a user to:

1. create three rclone remotes, each authenticating with a different account;
2. pool them together with mergerfs, using category.search=rand;
3. and have most Read-Only API calls spread among the three accounts’ API quotas

If successful, this could effectively triple the standard API quota, and thus triple rclone performance in an API-bottleneck situation.

Is this possible with the rclone Union remote?

N.B. it’s not even clear that this would work properly with mergerfs, but maybe it could work with rclone Union.

Thank you for your attention and hard work!

[calisro (Rob)]

It isn’t possible currently. That bring said, the only way you’d be able to use multiple users to mitigate quota issues against one gdrive is to share it or use a team drive. Both of those have their own restrictions/ negatives as well.

I suppose you could use one account with multiple client IDs to mitigate client ID API only restrictions but I think most people are quota restricted.

[ookla-ariel-ride]

How could this be done with mergerfs? Do you have a working config?

[xiaolei_0125 (xiaolei)]

Is there any new progress about unionfs?

[Animosity022]

Please don’t necrobump old topics. Closing.