Best gclone "fork"?

Off Topic
1

As per the subject. I have to move a ton of data in a hurry (enough said) and I'm trying to decide which "fork" to use;

I'm considering using a fork because the original one (which would normally be my preference) is years old and sync'ed to rclone since v1.51:

From some github searching, it seems the most up-to-date but still with a reasonable number of "stars" (meaning other users have demonstrated an interest on it) is the one from "dogbutcat":

So, anyone using it? If not, which one are you using?

TIA!

2

i wrote some lines of code in python to rotate the sa ids. (as rclone command line option)

1 Like
3

Thanks for chiming in. Unfortunately I don't have the time for that, I need something ready-made and as reliable as possible, and that's why I chose gclone. My only doubt so far is regarding which fork to use.

4

Additional question to @benba: I'm creating a service account following the procedure here: Google drive

I'm having a lot of trouble on step " 2. Allowing API access to example.com Google Drive": when I access admin.google.com and click on "Security", it lists no "Show more" option, much less "Advanced settings"... and searching for "Manage API client access" with the search bar also doesn't show any such options.

This is when logged with normal "Admin" account, not "Super Admin" -- does it have to be the latter?

What can I be doing wrong?

5

What limit are you trying to bypass?

1 Like
6

Discussions about bypassing limits are strongly discouraged here on the forum, and with good reason. So, I'm not really trying to bypass any limits :slight_smile: If this is somehow relevant to my questions, please send a PM and I will be happy to explain.

7

I'm just trying to understand the challenge you are facing. Limits for the various cloud providers are discussed regularly here, especially the undocumented ones. It is often those that will render a service less usefull or even entirely useless.

I am not aware of any limits that can be extended by rotating service accounts, when it comes to downloading from google drive.

In other words, if one service account reaches a download limit, any other service account will all reach it at the same time. Because the limit reached belongs to the drive it itself, not to the service account.

Other errors like rateLimitExceeded are easily avoided by using the --fast-list parameter.

When it comes to uploading, Google blocked the old "upload 750GB to a shared drive per service account per day" possibility earlier this year.

Anyway, just chipping in with my experience - I don't find the statement seen at the dogbutcat project to be true any longer:

Quotas SAs CAN bypass:

  • Google 'copy/upload' quota (750GB/account/day)
  • Google 'download' quota (10TB/account/day)

Google Service Accounts - docs.saltbox.dev uses the sa-gen utility to create service accounts - it could perhaps be tried out as an alternative to the admin.google.com way.

As can be seen the saltbox instructions were updated due to the service account restructuring at Google's side also, your mileage might vary.

1 Like
8

Thanks for the detailed, thoughtful post about those limits. And you are right, we are just discussing them, for educational purposes only -- no one here is trying to bypass anything! :grin:

Some immediate comments:

I am not aware of any limits that can be extended by rotating service accounts, when it comes to downloading from google drive.

In other words, if one service account reaches a download limit, any other service account will all reach it at the same time. Because the limit reached belongs to the drive it itself, not to the service account.

I'm a novice re: service accounts, but I'm used to Drive's "under the table" limiting -- and I've seen some weird crap regarding that, eg sometimes just changing the user agent is enough to speed things up (not bypass any limits, mind you).

Other errors like rateLimitExceeded are easily avoided by using the --fast-list parameter.

I can't really use this here because I'm on a dirtree with over 13M files -- and it seems --fast-list forces rclone to scan the whole tree before beginning to copy anything. So I'm forced to manage that by the traditional means of using the various --tps-* options. But proceeding that way, this is not being a problem right now.

Here's the last "progress" output of the transfer I was forced to terminate yesterday, to try and optimize it further:

2023/12/23 08:56:54 INFO  :
Transferred:        2.696 TiB / 4.901 TiB, 55%, 9.669 MiB/s, ETA 2d18h26m
Errors:                 0
Checks:            180485 / 180485, 100%
Transferred:         3443 / 13491, 26%
Elapsed time:  3d8h50m1.0s

If you do the math, the above boils down to ~820GB/day, so even without any SA 'optimizations', Google was already allowing me to upload more than these 750GB. My hope is that by adding SAs to the mix, I will be able to optimize it further and meet the deadline for my account removal.

Google Service Accounts - docs.saltbox.dev uses the sa-gen utility to create service accounts - it could perhaps be tried out as an alternative to the admin.google.com way.

As can be seen the saltbox instructions were updated due to the service account restructuring at Google's side also, your mileage might vary.

Thank youi for this! I hadn't heard of saltbox and sa-gen, perhaps they will help me get over the SA creation issues I'm now experiencing.

9

Excellent :slight_smile:

No, the transfer was still under 750GB/day (it was at 674GB/day to be precise) - it is just that it was biting into the quota for day #4 at this time. So it would face no problems unless it reached 3.000 TiB before the new quota for day #5 was added :slight_smile:

10

Not sure how you came to that 674GB/day figure, as using 4 for the number of days we find 2.696 * 1024 / 4 = ~707GiB/Day... but anyway, OK, point granted -- it's still less than 750GiB/Day.

But then how do you explain the numbers of my latest 2 parallels runs?

In one machine:

2023/12/24 10:02:02 ERROR : REDACTED: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

2023/12/24 10:01:53 INFO  : 
Transferred:      753.616 GiB / 2.103 TiB, 35%, 13.059 MiB/s, ETA 1d6h29m
Errors:                 1 (retrying may help)
Checks:               806 / 806, 100%
Transferred:          486 / 1500, 32%
Elapsed time:    2h51m0.6s

And in the other:

2023/12/24 13:02:02 ERROR : REDACTED: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2023/12/24 10:14:35 INFO  : 
Transferred:      102.508 GiB / 520.480 GiB, 20%, 9.396 MiB/s, ETA 12h39m11s
Errors:                 1 (retrying may help)
Checks:            147264 / 147264, 100%
Transferred:          300 / 10336, 3%
Elapsed time:    3h17m1.2s

Both rclone copy commands where from the same source remote (using exactly the same Google credentials), so I believe the in these 3h17m (~14% of a day) they managed to copy 753.616+102.508= 856.124 GiBs... had they also consumed part of "day 2" quota? If so, why did they stop at just 850GiB and not 2*750= 1500GiB?

I'm not trying to be argumentative, I swear -- It's just that these Google quota numbers never made much sense to me, and you seem to have given them more thought than I and could perhaps explain it in a way that made sense.

11

A guess: 2*750-856 = 644GB had already been used beforehand in other upload sessions.

If you spend a few days uploading more than 750GB every day and watch the logs, you will get a notion of when precisely the daily upload quota is reset - that will be the time when the rate limit errors stops being added to the log, and the transfers are able to continue.

That will make things a lot clearer. It used to be 06.00 CEST for me.

12

This is possible. I haven't kept full logs of the transfer sessions in that period (things are rather chaotic) but it's certainly possible it happened that way.

But then it would mean Google "saves" the unused quota from the previous day and "adds" it to the 750GB quota available for the current day, which seems pretty unintuitive to me. Or else that these "extra" 856-750= 106GB was what the transfers managed to consume from the "previous" (at the time, actually current) day's quota before the clock "turned over" and Google reset the quota back to 750GB... Oh heck, this is starting to give me a solid headache.

This I was able to determine from my logs: today, errors stopped by 08:45 GMT-3 almost to the second, which is quite a few hours and change later than 06:00 CEST. So I guess Google has different reset times per zone/server/datacenter/whatever.

13

I'm also a new user and I can confirm that the admin menu doesn't have those options anymore. So I'm blocked.