--drive-stop-on-BLAH-limit not stopping immediately

What is the problem you are having with rclone?

rclone move hangs for over 10 minutes even though I'm running both --drive-stop-on-download-limit and --drive-stop-on-upload-limit. I assume it is iterating through the rest of the files it could've moved at this time, but why not just error out right away?

I have confirmed that I am over my upload quota with that service account.

I think it might have something to do with 'x64tools.zip' being flagged as a virus by Google. Other than that this is over my head. Unless this is the expected result?

You can download the same file here:
https://www.nirsoft.net/packages/x64tools.zip

I can not download that file directly from Google Drive via my browser either. Even though I am not at my download limit, and although I'm telling it to ignore the virus scan. It fails.

I was able to successfully copy it out of Google Drive File Stream. So I can bypass it for now. What gives though?

What is your rclone version (output from rclone version)

rclone v1.54.0
- os/arch: linux/amd64
- go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial

Which cloud storage system are you using? (eg Google Drive)

drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move \
/mnt/gdrive/ \
tcrypt00: \
-vvvP \
--drive-stop-on-upload-limit \
--drive-stop-on-download-limit \
--max-transfer 749G \
--delete-empty-src-dirs \
--exclude-from /opt/scripts/tmp/rclone_exclude \
--exclude-from /opt/scripts/encrypt_exclude

The rclone config contents with secrets removed.

A log from the command with the -vv flag

I actually went with the -vvvP flag. It's rather short anyways.

2021/03/13 14:12:16 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "move" "/mnt/gdrive/" "tcrypt00:" "-vvvP" "--drive-stop-on-upload-limit" "--drive-stop-on-download-limit" "--max-transfer" "749G" "--delete-empty-src-dirs" "--exclude-from" "/opt/scripts/tmp/rclone_exclude" "--exclude-from" "/opt/scripts/encrypt_exclude"]
2021/03/13 14:12:16 DEBUG : Creating backend with remote "/mnt/gdrive/"
2021/03/13 14:12:16 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/03/13 14:12:16 DEBUG : Creating backend with remote "tcrypt00:"
2021/03/13 14:12:16 DEBUG : Creating backend with remote "tdrive00:/.obfuscated"
2021/03/13 14:12:17 DEBUG : fs cache: renaming cache item "tdrive00:/.obfuscated" to be canonical "tdrive00:.obfuscated"
2021-03-13 14:12:17 DEBUG : .obfuscated: Excluded
2021-03-13 14:12:18 DEBUG : Documents/My Documents: Excluded
2021-03-13 14:12:18 DEBUG : Documents/Resumes & Cover Letters: Excluded
2021-03-13 14:12:18 DEBUG : Documents/School: Excluded
2021-03-13 14:12:18 DEBUG : Programs/uTorrent.exe: Excluded
2021-03-13 14:12:18 DEBUG : Documents/Google Admin Downloads/drive_logs_1575049632858.xlsx: Excluded
2021-03-13 14:12:20 DEBUG : .recycle/sonarr/Big Brother Canada/Season 05/Big Brother Canada - S05E26 - Live Eviction #11 - HDTV-720p.mkv.partial~: Excluded
2021-03-13 14:12:20 DEBUG : .recycle/sonarr/Big Brother Canada/Season 07/Big Brother Canada - S07E22 - Episode 22 - HDTV-720p.mkv.partial~: Excluded
2021-03-13 14:12:21 DEBUG : .recycle/sonarr/LEGO Masters (US)/Season 01/LEGO Masters (US) - S01E06 - Need for Speed + Super-Bridges - WEBDL-1080p.mkv.partial~: Excluded
2021-03-13 14:12:23 DEBUG : .recycle/sonarr/The Good Doctor/Season 03/The Good Doctor - S03E13 - Sex and Death - WEBDL-1080p.mkv.partial~: Excluded
2021-03-13 14:12:25 ERROR : Google drive root '.obfuscated': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:12:25 ERROR : Projects/Google-Photos-Managercache/SeleniumProfile/Crashpad/metadata: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:12:25 ERROR : Projects/Google-Photos-Managercache/SeleniumProfile/Crashpad/metadata: Not deleting source as copy failed: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:12:25 ERROR : Cancelling sync due to fatal error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:12:25 ERROR : Google drive root '.obfuscated': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:12:25 ERROR : Projects/Google-Photos-Managercache/SeleniumProfile/Crashpad/settings.dat: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:12:25 ERROR : Projects/Google-Photos-Managercache/SeleniumProfile/Crashpad/settings.dat: Not deleting source as copy failed: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:12:26 ERROR : Projects/Gamerr/venv/Scripts/Activate.ps1: Failed to copy: couldn't list directory: context canceled
2021-03-13 14:12:26 ERROR : Projects/Gamerr/venv/Scripts/Activate.ps1: Not deleting source as copy failed: couldn't list directory: context canceled
2021-03-13 14:12:27 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 1/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:12:33 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 2/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:12:40 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 3/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:12:47 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 4/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:12:53 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 5/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:13:00 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 6/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:13:08 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 7/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:13:14 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 8/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:13:20 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 9/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:13:27 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopening on read failure after 0 bytes: retry 10/10: read /mnt/gdrive/Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: input/output error
2021-03-13 14:13:27 DEBUG : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Reopen failed after 0 bytes read: failed to reopen: too many retries
2021-03-13 14:13:27 ERROR : Google drive root '.obfuscated': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:13:27 ERROR : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-03-13 14:13:27 ERROR : Documents/Google Drive (REDACTED@gmail.com)/x64tools.zip: Not deleting source as copy failed: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
Transferred:           152 / 7.314 MBytes, 0%, 2 Bytes/s, ETA 5w5d6h25m35s
Errors:                 4 (fatal error encountered)
Checks:                 4 / 4, 100%
Transferred:            0 / 23, 0%
Elapsed time:     13m47.6s
2021/03/13 14:25:57 INFO  :
Transferred:           152 / 7.314 MBytes, 0%, 2 Bytes/s, ETA 5w5d6h25m35s
Errors:                 4 (fatal error encountered)
Checks:                 4 / 4, 100%
Transferred:            0 / 23, 0%
Elapsed time:     13m47.6s

2021/03/13 14:25:57 DEBUG : 2 go routines active
2021/03/13 14:25:57 Failed to move with 4 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

nirtools is great set of tools, but can be a pain, as its tools are always getting flags as trojan/virus/etc...
tho that does not look like the problem.

rclone about tcrypt00: -vvvP output

2021/03/13 15:20:45 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "about" "tcrypt00:" "-vvvP"]
2021/03/13 15:20:45 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/03/13 15:20:45 DEBUG : Creating backend with remote "tcrypt00:"
2021/03/13 15:20:45 DEBUG : Creating backend with remote "tdrive00:/.obfuscated"
2021/03/13 15:20:46 DEBUG : fs cache: renaming cache item "tdrive00:/.obfuscated" to be canonical "tdrive00:.obfuscated"
2021-03-13 15:20:46 DEBUG : Google drive root '.obfuscated': read info from team drive "Team Drive"
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         1.3s
2021/03/13 15:20:46 DEBUG : 6 go routines active

Not quite sure what that is supposed to tell me. Using a different service account (ie. tcrypt01) uploads fine. So I do not doubt this is a quota limit problem.

no worries, we have many gdrive experts, one should stop by soon....

Excellent. Oh and I should mention that you are correct about the fact that it was a virus flagged file is likely irrelevant. I removed that file, and it still hangs on other files. This time for over 20 minutes.

nirsoft is so good, it has to be flagged, a small price to pay...

i just setup a new laptop for myself, made the mistake of copying nirsoft from my server.
i forgot to make an exclusion rule for it.
the anti-virus freaked out over and over

perhaps @VBB could help with this.

Too true. It was a necessary part of my tech toolkit. This is just another reason to encrypt my cloud contents.
Don't worry Google, I just know better than you do.

with nirsoft, the fact that --drive-acknowledge-abuse is needed
not sure if that is an insult or a badge of honor :upside_down_face:

Ahhhh I knew a saw a flag for this somewhere. Thanks for that. Will have to add that to my flags.

I saw it hang that way for over an hour once now.

Could you try this beta to see if it helps?

v1.55.0-beta.5216.e556df071.fix-backend-cancel-context on branch fix-backend-cancel-context

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.