Using copy to upload to Google Drive Fails

Help and Support
1

What is the problem you are having with rclone?

I'm using rclone on windows 11 x64 to copy a folder to Google Drive, using an encrypted remote. Everything seems to work as normal, i.e. the file begins uploading... but then when it reaches 100% it starts throwing "error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded" errors, rclone retries a few times and fails then restarts the entire upload again.

Run the command 'rclone version' and share the full output of the command.

rclone --version
rclone v1.63.0

  • os/version: Microsoft Windows 11 Enterprise 21H2 (64 bit)
  • os/kernel: 10.0.22000.2176 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.5
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive(via encrypted remote)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vv -P --transfers 3 --drive-chunk-size 32M 'C:\Users\Name\Storage\' GDStorage:

The rclone config contents with secrets removed.

[GDStorage]
type = crypt
remote = GD:Work/Storage
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

A log from the command with the -vv flag

2023-07-11 19:46:21 DEBUG : FILE1: Need to transfer - File not found at Destination
2023-07-11 19:46:21 DEBUG : Encrypted drive 'GDStorage:': Waiting for checks to finish
2023-07-11 19:46:21 DEBUG : Encrypted drive 'GDStorage:': Waiting for transfers to finish
2023-07-11 19:46:23 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:46:23 DEBUG : pacer: Rate limited, increasing sleep to 1.172115132s
2023-07-11 19:46:23 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:46:23 DEBUG : pacer: Rate limited, increasing sleep to 2.074724446s
2023-07-11 19:46:23 DEBUG : Super FILE1: Received error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded - low level retry 1/10
2023-07-11 19:46:23 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:46:23 DEBUG : pacer: Rate limited, increasing sleep to 4.996637418s
2023-07-11 19:46:24 DEBUG : pacer: Reducing sleep to 0s
2023-07-11 19:46:24 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:46:24 DEBUG : pacer: Rate limited, increasing sleep to 1.586873538s
2023-07-11 19:46:24 DEBUG : pacer: Reducing sleep to 0s
2023-07-11 19:46:24 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:46:24 DEBUG : pacer: Rate limited, increasing sleep to 1.573217611s
2023-07-11 19:46:24 DEBUG : FILE2: Received error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded - low level retry 2/10


SNIP 


2023-07-11 19:54:03 DEBUG : pacer: low level retry 6/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:03 DEBUG : pacer: Rate limited, increasing sleep to 1.292221985s
2023-07-11 19:54:04 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:04 DEBUG : pacer: Rate limited, increasing sleep to 2.945051374s
2023-07-11 19:54:05 DEBUG : vc2krul63m0l5c7rtm0bt7b3hl125fnen58u2susbl31a71ibgh42jkkgilbfn5phqud1p76vgma8/um4nqe4jvd7qi3vvq07ehrklk82887ov9utfql8ugqfd269ntsdhepds8b9tsf603694vjjbljnq2gm20ci1v2kv0nnsfa4rngkbqbhg53kbrmss9dsq6mfdm50256bk: Sending chunk 134217728 length 33554432
2023-07-11 19:54:08 DEBUG : i0ctncqgn73tkblt2q7rc5524l792me0ba4k2roi69ums76oe9ng/5k3pc2rts3l7cq4vhf5rbj3tnhpmtq2adc21t8jem3itb217ofjcp4gdtl7gvv2hk4uq043lvvhmj12gkof7oqd3v7k5q7tpehh2070: Sending chunk 2684354560 length 4981902
2023-07-11 19:54:08 DEBUG : pacer: low level retry 7/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:08 DEBUG : pacer: Rate limited, increasing sleep to 4.057865358s
2023-07-11 19:54:09 DEBUG : pacer: Reducing sleep to 0s
2023-07-11 19:54:11 DEBUG : j1t8rt6tfstonarqhvijv2gsnj81ml94fd8d87co1gtr92a7hl90/ggmp53gepenm70a7bn3doqt493a3hhkc045undsk2gofqa034bb3lnmo23r0g5vccovp830lso7fdgcs9g74en4uunfl036fv3n912dbke74es9sgb4r47ttk3naath1: Sending chunk 436207616 length 29723621
2023-07-11 19:54:11 DEBUG : i0ctncqgn73tkblt2q7rc5524l792me0ba4k2roi69ums76oe9ng/5k3pc2rts3l7cq4vhf5rbj3tnhpmtq2adc21t8jem3itb217ofjcp4gdtl7gvv2hk4uq043lvvhmj12gkof7oqd3v7k5q7tpehh2070: Sending chunk 2684354560 length 4981902
2023-07-11 19:54:11 DEBUG : vc2krul63m0l5c7rtm0bt7b3hl125fnen58u2susbl31a71ibgh42jkkgilbfn5phqud1p76vgma8/um4nqe4jvd7qi3vvq07ehrklk82887ov9utfql8ugqfd269ntsdhepds8b9tsf603694vjjbljnq2gm20ci1v2kv0nnsfa4rngkbqbhg53kbrmss9dsq6mfdm50256bk: Sending chunk 167772160 length 33554432
2023-07-11 19:54:12 DEBUG : pacer: low level retry 8/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:12 DEBUG : pacer: Rate limited, increasing sleep to 1.020076843s
2023-07-11 19:54:12 DEBUG : i0ctncqgn73tkblt2q7rc5524l792me0ba4k2roi69ums76oe9ng/5k3pc2rts3l7cq4vhf5rbj3tnhpmtq2adc21t8jem3itb217ofjcp4gdtl7gvv2hk4uq043lvvhmj12gkof7oqd3v7k5q7tpehh2070: Sending chunk 2684354560 length 4981902
2023-07-11 19:54:12 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:12 DEBUG : pacer: Rate limited, increasing sleep to 2.087705773s
2023-07-11 19:54:13 DEBUG : i0ctncqgn73tkblt2q7rc5524l792me0ba4k2roi69ums76oe9ng/5k3pc2rts3l7cq4vhf5rbj3tnhpmtq2adc21t8jem3itb217ofjcp4gdtl7gvv2hk4uq043lvvhmj12gkof7oqd3v7k5q7tpehh2070: Sending chunk 2684354560 length 4981902
2023-07-11 19:54:13 DEBUG : pacer: low level retry 10/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:13 DEBUG : pacer: Rate limited, increasing sleep to 4.06098594s
2023-07-11 19:54:13 DEBUG : FILE2: Received error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded - low level retry 1/10
2023-07-11 19:54:14 DEBUG : pacer: low level retry 9/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:14 DEBUG : pacer: Rate limited, increasing sleep to 8.560747669s
2023-07-11 19:54:15 DEBUG : pacer: Reducing sleep to 0s
2023-07-11 19:54:15 DEBUG : j1t8rt6tfstonarqhvijv2gsnj81ml94fd8d87co1gtr92a7hl90/ggmp53gepenm70a7bn3doqt493a3hhkc045undsk2gofqa034bb3lnmo23r0g5vccovp830lso7fdgcs9g74en4uunfl036fv3n912dbke74es9sgb4r47ttk3naath1: Sending chunk 436207616 length 29723621
2023-07-11 19:54:15 DEBUG : vc2krul63m0l5c7rtm0bt7b3hl125fnen58u2susbl31a71ibgh42jkkgilbfn5phqud1p76vgma8/um4nqe4jvd7qi3vvq07ehrklk82887ov9utfql8ugqfd269ntsdhepds8b9tsf603694vjjbljnq2gm20ci1v2kv0nnsfa4rngkbqbhg53kbrmss9dsq6mfdm50256bk: Sending chunk 201326592 length 33554432
2023-07-11 19:54:15 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2023-07-11 19:54:15 DEBUG : pacer: Rate limited, increasing sleep to 1.088307089s
Transferred:        7.393 GiB / 146.280 GiB, 5%, 5.983 MiB/s, ETA 6h36m10s
Transferred:            1 / 165, 1%
Elapsed time:      7m54.7s
Transferring:
 * FILE:  6% /2.959Gi, 4.764Mi/s, 9m54s
 * FILE2:  0% /2.504Gi, 0/s, -
 * FILE3:100% /444.238Mi, 124.728Ki/s, -
2

for GD:, did you follow this guide?
It is strongly recommended to use your own client ID as the default rclone ID is heavily used

can you post the redacted config for GD

3

I should have included that info, but yes. I have set my own client id.

5 
[GD]
type = drive
client_id = XXXXX.apps.googleusercontent.com
client_secret = XXXXXXX
scope = drive
token = {"access_token":"XXXXXXX","token_type":"Bearer","refresh_token":"XXXXXXXX","expiry":"2023-07-11T20:13:46.730066117-04:00"}
root_folder_id = XXXXXXX
6

looks like you hit one of the gdrive limits, often just have to wait until the limit is reset.

i think we have enough basic info for others to provide more info.

7

Ah. Ok. I'll give it a day or so and if its still happening I'll try to gather more info and post it here.

Thanks

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.