The Sync command can't disable server-side-copy?

Help and Support
1

Hi, I want to sync GdriveA to GdriveB, and I have set those folders in GdriveA share to GdriveB and add to my drive in GdriveB
But the service-side-copy has 750GB Limited, so I want use my unlimited bandwidth server to continue sync (need to disable service-side-copy)
So I tried to use:

/usr/bin/rclone sync gd-B:file gd-B:Sync_file --disable --max-transfer 750G --config=/root/.config/rclone/rclone.conf -P -vvvv

the command successful run but I found it still use server-side-copy mode in the log

So I'm so curious about is that --disable flag just set for copy or move command? Sync command can't use the flag or may i set the wrong way?

2

Even the normal up/down has the same 750 GB limit. Doesn't really matter if your server has unlimited bandwidth or not. Server-side copies will simply reduce the amount taken with no bandwidth utilized on the server.

3

Thanks for ur respond. I know the normal mode had the 750gb limited. But I want to use server side copy and normal mode to transfer more data (1.5tb/day) to reduce time.
Good news is i found the right way to disable server side copy in the sync command.
The right command is
/usr/bin/rclone sync --disable copy gd-B:file gd-B:Sync_file --max-transfer 750G --config=/root/.config/rclone/rclone.conf -P -vvvv

1 Like
4

I think you are still misunderstanding.
It is 750GB/day no matter what method of copying you use. You can't use both methods to get 1.5TB in a day. The only way to get more data up in a day on Gdrive is to use multiple accounts some way.

And that flag you put in disables all copies, not just server-side copies.
server-side copies are by default disabled on Gdrive, so it won't get used unless you enable it yourself.
That said, if you have it enabled in config and still want to overload that flag in the command, then you can use:
--drive-server-side across-configs false

5

Very Thanks!
Seems I really misunderstand the 750G limited,I used to think it's to distinguish with the server side copy and local upload.

6

There is a difference, but only in the sense that the server-side copy quota is assumed to be less. Exactly how much is not really known since google does not tell us. Some have told me 100G but my experience is that it seems to be at least a few hundred G, and there may also be other factors involved like how much load google has that day or how many files are being worked on ect. More research is needed.

So in short: the limit is still 750G
But it still will make sense to continue with a local->remote transfer after server-side stops working, because that probably won't be able to do 750G a day on it's own.
So your idea was right, but for the wrong reason :smiley:

7

The server-side copy has the same 750 GB limit as long as all the permissions are fine. I have copied multiple hundreds of terabytes server side and I have yet to hit a lower limit

8

Limits:

  • ~100GB Day server side copy
    -750 GB Day local to gdrive
9

It seems very inconsistent to me - with varying results. I've definitely seen a lot of stalls after a while, usually around 200-300GB (resulting in 403 errors) where it simply won't progress anymore and produced 100% errors - yet local->Gdrive works fine to continue on with. Pretty sure it has nothing do to with permissions as all the data there is from the same user and sent via the same channel.

There was some very recent fixes in the code for server-side though. Since then it seems like it has been better in general? I just don't know the details of what was fixed and if that is relevant.

10

The stalls were fixed for 1.49. Any beta after the fix has worked perfectly fine for me with no stalls and the same 750 GB limit.

11

Interesting! I haven't really stress-tested it with a large amount of data since 1.49, I've just noticed no stalls so far. If it now works as reliably as local copy that is fan-fucking-tastic :smiley:

Just out of curiosity, do you know specifically what the problem was that got fixed?

12
1 Like
13

How do you get hundred of TB

14

Presumably he means distributed over several days.

15

Not really, you can copy it in a couple of hours with service accounts.

16

Wait, hang on a minute ... are you saying the quotas are tied to the authorization and not the drive itself ?? I assume there is nothing special about service accounts over Oauth in that regard.

If so then I have fundamentally misunderstood something critical. Please provide more detail!

17

Yep, the quotas are per authorization.

Its easier with Team Drives. Normal GDrive is limited because currently there is no easy way to share it with service accounts and have it work without issues.

18

Wow, this is a massive revelation. Thank you for letting me know.
Can those SAs all be created on the same google user then? Or do you need some separate accounts to do that? Any other limitations I need to be aware of if I try this?

19

SAs are separate from google users. You can create as many SAs as you would like but I think you are limited to 100 per google cloud project. You can obviously create more projects as you like.

20

Hmmm yea, I first assumed they were made in the google API where you create the Oath but I guess when I think about it that option doesn't exist there.

So do you need any paid access to Gcloud or is just a free user fine?