If you use rclone lsjson or rclone lsf to show the object IDs you'll see that the shortcut IDs look different. Rclone synthesizes these out of the ID of the shortcut and the ID of the thing it is pointing to. So you can tell them apart like that.
Yes, I can see that now. Thanks!
1 Like
I just found out that --bwlimit doesn't actually work in my case (server-side copy). I "transferred" 750GB in less than 15 minutes. Is this a bug or perhaps a known limitation?
I'm now using --drive-stop-on-upload-limit instead and a daily cron job. The cron job produces 1000+ fatal errors of the form
2020/06/18 11:13:38 ERROR : [...]: Failed to Mkdir: couldn't list directory: context canceled
but exits with code 0 so I guess everything is fine? 
Its a known limitation since you're not actually transferring data ona server side move. Yes you'd need --drive-stop-on-upload-limit in that case.
Can try transferring with one or a few files and then share the full debug log? I'm guessing though it may be your config. root_folder_id leave that empty and see if it fixes the issue. What is its current value?
To be clear, the copy operation seems to work just fine. My issue is just with (what appears to be) excessive error reporting when the limit is reached.
The root_folder_id of my destination drive was set automatically to some random looking string. I'd rather not delete it. How is it generated?
The root_folder_id of my source drive is appDataFolder.
You can save its current value before setting to "". You haven't shared a debug log but I don't have errors like that in my logs.
Here's a debug log of me attempting to copy a directory with a handful of files. Some of the files were copied successfully. Unfortunately, I don't get the
errors in this case. Does my log look more like what you usually see?
2020/06/18 14:11:00 DEBUG : rclone: Version "v1.52.1-061-gfb06427c-beta" starting with parameters ["rclone" "copy" "drive-with-appfolder:6rhjg0d0a21poa4d90bbr9utqg/i775bqpf0lpt3k5i7gokv6rl6o/a4643tl3elsppd350u1tfrsuaiqprshe1k8d4uas3g58farmlaurka4iddvdh3ao3lcnlr6730ml8" "drive-without-appfolder:rclone/6rhjg0d0a21poa4d90bbr9utqg/i775bqpf0lpt3k5i7gokv6rl6o/a4643tl3elsppd350u1tfrsuaiqprshe1k8d4uas3g58farmlaurka4iddvdh3ao3lcnlr6730ml8" "--create-empty-src-dirs" "--fast-list" "--links" "--no-traverse" "--drive-stop-on-upload-limit" "--log-file" "copy-errors.log" "--log-level" "DEBUG"]
2020/06/18 14:11:00 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/06/18 14:11:05 DEBUG : Google drive root 'rclone/6rhjg0d0a21poa4d90bbr9utqg/i775bqpf0lpt3k5i7gokv6rl6o/a4643tl3elsppd350u1tfrsuaiqprshe1k8d4uas3g58farmlaurka4iddvdh3ao3lcnlr6730ml8': Waiting for checks to finish
2020/06/18 14:11:05 DEBUG : Google drive root 'rclone/6rhjg0d0a21poa4d90bbr9utqg/i775bqpf0lpt3k5i7gokv6rl6o/a4643tl3elsppd350u1tfrsuaiqprshe1k8d4uas3g58farmlaurka4iddvdh3ao3lcnlr6730ml8': Waiting for transfers to finish
2020/06/18 14:11:07 ERROR : Google drive root 'rclone/6rhjg0d0a21poa4d90bbr9utqg/i775bqpf0lpt3k5i7gokv6rl6o/a4643tl3elsppd350u1tfrsuaiqprshe1k8d4uas3g58farmlaurka4iddvdh3ao3lcnlr6730ml8': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2020/06/18 14:11:07 ERROR : mvrba10b5tu5ad98fgh42ji865eb0t4fomtqrs0jj4ee77otm9sqdmpn89u988q8u4f13tu7k6fpo: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2020/06/18 14:11:07 ERROR : Cancelling sync due to fatal error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2020/06/18 14:11:08 DEBUG : 5rdjd85alns8lljjbp8msrc8b5oofqs2k44vh2u2p73ktei1celqt18tu47uhn8lh2ki394ug54k2: MD5 = f9083490acb53094a5e4d267055bcac2 OK
2020/06/18 14:11:08 INFO : 5rdjd85alns8lljjbp8msrc8b5oofqs2k44vh2u2p73ktei1celqt18tu47uhn8lh2ki394ug54k2: Copied (server side copy)
2020/06/18 14:11:08 DEBUG : k1h6jlj1elmbbi1psofi5h9f0e6qbu3ree18uqqlle3uk61qooarbjhgiqu0a3vg7noum4dg8293u: MD5 = 54104b666f3d30f585315bfb2be98921 OK
2020/06/18 14:11:08 INFO : k1h6jlj1elmbbi1psofi5h9f0e6qbu3ree18uqqlle3uk61qooarbjhgiqu0a3vg7noum4dg8293u: Copied (server side copy)
2020/06/18 14:11:09 DEBUG : 8n2eoheef5ggiadu31qcr7ddg2ntpsran1nnh35egeuhimak6qqit4vbajilupko47rpg5qaf8n84: MD5 = 83d06a073ac6c8f982bcefa15a984c60 OK
2020/06/18 14:11:09 INFO : 8n2eoheef5ggiadu31qcr7ddg2ntpsran1nnh35egeuhimak6qqit4vbajilupko47rpg5qaf8n84: Copied (server side copy)
2020/06/18 14:11:09 ERROR : Fatal error received - not attempting retries
2020/06/18 14:11:09 INFO :
Transferred: 1.744M / 3.441 MBytes, 51%, 496.497 kBytes/s, ETA 3s
Errors: 1 (fatal error encountered)
Transferred: 3 / 4, 75%
Elapsed time: 3.5s
2020/06/18 14:11:09 DEBUG : 16 go routines active
2020/06/18 14:11:09 Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
This doesn't really say how it's generated! For my destination drive, I'm in the
Normally you will leave this blank and rclone will determine the correct root to use itself.
case. I tried deleting root_folder_id from my config entirely, and it was recreated with the exact same value, so I guess it's OK.
I dont see you using your own client ID in your config... That would help with your errors.
I'm using a service account, which comes with its own client ID. I've never experienced anything that suggests that I'm not using my own client ID.
A service account and a client ID are two different things.
You still need to create a client ID/secret regardless if you are using your own account for authentication or a service account.
If you haven't made a client ID/secret, you are using the default rclone one.
Honestly, it's been some time since I did the setup, but going to
https://console.developers.google.com/
and looking at Google Drive API metrics, if I filter out traffic for everything but my single OAuth client, I see a clear spike whenever I do anything with rclone which suggests to me that I'm using my own client.
To put it another way, I'd consider it exceedingly weird to have to put a client ID in rclone.conf when there's already a "client_id" field (with a value!) in the service account file.
The client ID in the service account JSON file is for tying the service account client ID back to the GSuite Delegation that you do.
For accessing the API, you need a client ID and secret.
That being said, if you do not have Client ID and Client Secret in your rclone.conf in the remote, it's using the default rclone API key.
Nifty.
I re-ran through making a new service account there is a change to the process it seems.
This was before making an account today.
My after now has this:
Which looks like there is an oAuth client ID/Secret generated automatically for the project that cannot be changed.
1 Like
I have a "service account client" under "OAuth 2.0 Client IDs" (like you, I assume). Where do I get the secret? Or are you suggesting that I don't actually need it anyway?
I think you are correct as something has definitely changed, which makes things much easier.
From what I can tell, no matter how many service accounts I make, it uses that automatically generated oAuth Client ID and no more. There has to be some magic in tying the service account back to the client ID/secret that happens behind the scenes as you have no way to change either.
I tested with no values for client ID /secret and I do see the API traffic coming in the right spot under that new Client ID that was created. I'd imagine things are working as expected.
That error is either quota related for the API being hammered or you hitting the upload quota. If you can't upload anymore, it's the latter.
Yes, it never made sense to me that I needed anything more than what's in the JSON file, but that's apparently something rather recent.
That's what I thought! This whole client ID/secret business was a bit of a detour.
It's me hitting the upload quota.
I guess my point was simply that I'm seeing a lot of errors that are really not errors, in my copy operation. Perhaps "--drive-stop-on-upload-limit" should turn these ERROR messages into INFO messages?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.