Rclone Sync with multiple destinations

Help and Support
1

Hey guys,

I have been using Rclone for years now and em very thankful for it and for this forum for the help I've received.

I am not having an issue with Rclone as much as I have a question, which is why I haven't copied my config file and provided the other info.

Question:
I have a server at my house with a substantial amount of data on it. It's approaching 50TBs and I can see a time when it gets to 100TB+. The server already has the capacity.

I have been using Rclone to upload this data to Google Drive and it's been working great. Recently though they have killed their unlimited plans (or more correctly, they have started enforcing the stated data limits) so I will be switching to another provider like Dropbox or Onedrive. This is really beyond the scope of this discussion. The point is that many of these cloud providers will give you data in chunks. For example, Onedrive will give you storage in 25TB Sharepoint sites. This means that if you have 70TB of data, you'll get three 25TB Sharepoint sites. Thats 3 separate destinations.

So my question is: Is it possible to sync one source folder (on my server) with multiple destination folders such that if a file is in any one of those 3 destinations it will consider it synced?

I realize that a solution might be to run 3 separate sync scripts but would like to avoid that because, as an example, I have a base level folder with a bunch of files in it thats about 30TB and would have to arbitrarily split that up just to make it sync.

I'm happy to take advice and provide any other info thats needed.

Thanks in advance for the help. it's much appreciated.
JM

2

You can try the Rclone Union option How To Use Rclone Union

1 Like
3

Agree with @leodsouza. The union is way to go:

I use it myself in exactly scenario you described - stitching together multiple Onedrive sites. It works flawlessly and create perfect illusion that I have one big remote - instead of few independent ones.

1 Like
4

Thank you very much for your response.

So if I have a large quantity of files at home that I want to push to OneDrive, I assume I would create a union remote with all of the individual OneDrive sites.

Then, would it be as easy as running in rclone sync command from the local home folder to the combined union remote? How does it know when to start putting files in the second, third, or fourth OneDrive site?

5

Read policies part of the documentation - you can define how drives are selected.

6

Thanks. I'll check that out. can you please post your command here so I can get an idea of how you use it?

7 
[BIG_CLOUD_DISK]
type = union
upstreams = onedrive1:u onedrive2:u onedrive3:u
action_policy = epall
create_policy = mfs
search_policy = ff
cache_time = 120
1 Like
8

I read the policies and am still a little confused.

Let's say I have a folder called "/Stuff" locally that contains several hundred files (Not no subfolders). The size of this folder is 60TB.

lets also say that I create a union remote with the same configuration as yours. this union contains 3 one drive sites, just as yours does.

I run an "rclone sync Local:Stuff union:"

Will it fill up one one drive site and then create the same /Stuff folder in the second one drive site to put the rest of the files? Or will it create a /Stuff folder in each one drive site and split the files equally?

Thanks in advance for your help!

9

This is how it will work - and this was my objective with below policies:

action_policy = epall
create_policy = mfs
search_policy = ff

create is based on free space - put new file where is the most free space.

1 Like
10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.