Download quata is exceeded

ANAZ_A · February 15, 2023, 11:57pm

What is the problem you are having with rclone?

iam using rclone in colab for transfering files
when i transfer files from one gdrive to another gdrive. it shows "download quata exceeded, please check what applications might be downloading a lot of large files from gdrive and allow one day for the download quata to reset". daily limit of google drive is 750gb. but this problem shows up even when less than 30gb complete. folder contains 7549 files. all the problems in destination gdrive. how can i avoid checking files? it checks the entire files in the folder while transfering and thats when this problem occurs. checking act as downloading.
copying still continues. but when i play a file in destination drive, it shows download quota exceeded

can you suggest any method for avoiding this problem?

Run the command 'rclone version' and share the full output of the command.

Rclone: v1.61.1

Which cloud storage system are you using? (eg Google Drive)

google drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

!rclone copy "/content/drives/gs/movie" "/content/drives/ge/movie"  --stats-one-line -vv --stats 5s --tpslimit 25 --tpslimit-burst 10 --bwlimit 38M --max-transfer 600G

!rclone copy "/content/drives/gs/movie" "/content/drives/ge/movie" --stats-one-line -vv --stats 5s --drive-server-side-across-configs

The rclone config contents with secrets removed.

[gs]
type = drive
scope = drive
token = {"access_token":"ya29.a0AVvZVsoowwSxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx163","token_type":"Bearer","refresh_token":"1//06UFhyoM2oBVdCgYIARAAGAYSNwF-L9Irin6s-1MuEE8PmDkReWNy6VG6T_L_oYEeqLnvVThDha6MIdf-aRwMVHE3ZQCHYafWNWE","expiry":"2023-02-06T03:39:58.335128+05:30"}
team_drive = 0ADdOaxxxxxxxxxxx
root_folder_id = 

[ge]
type = drive
scope = drive
token = {"access_token":"ya29.a0AVvZVsqo9-IZxQ-gtVxxxxxxxxxxxxxxxxxxxxxxxwLFv-45q5fAOp3pD6bA0DNNtXQc9mVEr5c5g-rUlDa2_A4vtr0woaQEsjf0KZLx2ZEj9H9hduuZ1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx24+05:30"}
team_drive =

A log from the command with the `-vv` flag

2023/02/15 23:03:57 INFO  : Starting bandwidth limiter at 38Mi Byte/s
2023/02/15 23:03:57 INFO  : Starting transaction limiter: max 25 transactions/s with burst 10
2023/02/15 23:03:57 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "/content/drives/gs/movie" "/content/drives/ge/movie" "--stats-one-line" "-vv" "--stats" "5s" "--checkers" "0" "--tpslimit" "25" "--tpslimit-burst" "10" "--bwlimit" "38M" "--max-transfer" "600G"]
2023/02/15 23:03:57 DEBUG : Creating backend with remote "/content/drives/gs/movie"
2023/02/15 23:03:57 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/02/15 23:03:58 DEBUG : Creating backend with remote "/content/drives/ge/movie"
2023/02/15 23:04:03 INFO  :           0 B / 0 B, -, 0 B/s, ETA -
2023/02/15 23:04:05 DEBUG : (18+) КАРТИНА КРАСОТЫ _ PICTURE of BEAUTY (2017).Эротика.Драма.(Великобритания).mp4: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/02/15 23:04:05 DEBUG : (18+) КАРТИНА КРАСОТЫ _ PICTURE of BEAUTY (2017).Эротика.Драма.(Великобритания).mp4: Unchanged skipping
2023/02/15 23:04:05 DEBUG : ...ЖЕНА В ОТПУСКЕ...ЛЮБОВНИЦА В ГОРОДЕ(1980)LA MOGLIE IN VACANZA  L'AMANTE IN CITTA.................................mp4: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/02/15 23:04:05 DEBUG : ...ЖЕНА В ОТПУСКЕ...ЛЮБОВНИЦА В ГОРОДЕ(1980)LA MOGLIE IN VACANZA  L'AMANTE IN CITTA.................................mp4: Unchanged skipping
2023/02/15 23:04:05 DEBUG : 1020.mp4: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/02/15 23:04:25 ERROR : Tubi/Watch End of Summer (1997) - Free Movies - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:25 ERROR : Tubi/Watch End of Summer (1997) - Free Movies - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch End of Summer (1997) - Free Movies - Tubi.ts: input/output error
2023/02/15 23:04:25 DEBUG : Tubi/Watch End of Summer (1997) - Free Movies - Tubi.ts: md5 differ
2023/02/15 23:04:25 DEBUG : Tubi/Watch Enemies- A Love Story (1989) - Free Movies - Tubi.ts: Modification times differ by 764h38m9.339s: 2022-12-21 04:15:29.678 +0000 UTC, 2023-01-22 00:53:39.017 +0000 UTC
2023/02/15 23:04:25 ERROR : Tubi/Watch Enemies- A Love Story (1989) - Free Movies - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:25 ERROR : Tubi/Watch Enemies- A Love Story (1989) - Free Movies - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Enemies- A Love Story (1989) - Free Movies - Tubi.ts: input/output error
2023/02/15 23:04:25 DEBUG : Tubi/Watch Enemies- A Love Story (1989) - Free Movies - Tubi.ts: md5 differ
2023/02/15 23:04:25 DEBUG : Tubi/Watch Fall (1997) - Free Movies - Tubi.ts: Modification times differ by 693h38m40.998s: 2022-12-23 23:21:24.919 +0000 UTC, 2023-01-21 21:00:05.917 +0000 UTC
2023/02/15 23:04:26 ERROR : Tubi/Watch Fall (1997) - Free Movies - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:26 ERROR : Tubi/Watch Fall (1997) - Free Movies - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Fall (1997) - Free Movies - Tubi.ts: input/output error
2023/02/15 23:04:26 DEBUG : Tubi/Watch Fall (1997) - Free Movies - Tubi.ts: md5 differ
2023/02/15 23:04:26 DEBUG : Tubi/Watch Fear of Water (2017) - Free Movies - Tubi.ts: Modification times differ by 694h42m33.281s: 2022-12-23 22:23:53.636 +0000 UTC, 2023-01-21 21:06:26.917 +0000 UTC
2023/02/15 23:04:26 ERROR : Tubi/Watch Fear of Water (2017) - Free Movies - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:26 ERROR : Tubi/Watch Fear of Water (2017) - Free Movies - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Fear of Water (2017) - Free Movies - Tubi.ts: input/output error
2023/02/15 23:04:26 DEBUG : Tubi/Watch Fear of Water (2017) - Free Movies - Tubi.ts: md5 differ
2023/02/15 23:04:26 DEBUG : Tubi/Watch Fling (2008) - Free Movies - Tubi.ts: Modification times differ by 762h27m33.385s: 2022-12-21 04:40:19.403 +0000 UTC, 2023-01-21 23:07:52.788 +0000 UTC
2023/02/15 23:04:27 ERROR : Tubi/Watch Fling (2008) - Free Movies - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:27 ERROR : Tubi/Watch Fling (2008) - Free Movies - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Fling (2008) - Free Movies - Tubi.ts: input/output error
2023/02/15 23:04:27 DEBUG : Tubi/Watch Fling (2008) - Free Movies - Tubi.ts: md5 differ
2023/02/15 23:04:27 DEBUG : Tubi/Watch Flower (2017) - Free Movies - Tubi.ts: Modification times differ by 770h2m10.127s: 2022-12-21 01:42:06.843 +0000 UTC, 2023-01-22 03:44:16.97 +0000 UTC
2023/02/15 23:04:27 ERROR : Tubi/Watch Flower (2017) - Free Movies - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:27 ERROR : Tubi/Watch Flower (2017) - Free Movies - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Flower (2017) - Free Movies - Tubi.ts: input/output error
2023/02/15 23:04:27 DEBUG : Tubi/Watch Flower (2017) - Free Movies - Tubi.ts: md5 differ
2023/02/15 23:04:27 DEBUG : Tubi/Watch Flunk S01-E01 - First Kiss Free TV - Tubi.ts: Modification times differ by 769h36m16.275s: 2022-12-21 01:30:12.041 +0000 UTC, 2023-01-22 03:06:28.316 +0000 UTC
2023/02/15 23:04:27 ERROR : Tubi/Watch Flunk S01-E01 - First Kiss Free TV - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:27 ERROR : Tubi/Watch Flunk S01-E01 - First Kiss Free TV - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Flunk S01-E01 - First Kiss Free TV - Tubi.ts: input/output error
2023/02/15 23:04:27 DEBUG : Tubi/Watch Flunk S01-E01 - First Kiss Free TV - Tubi.ts: md5 differ
2023/02/15 23:04:27 DEBUG : Tubi/Watch Flunk S01-E02 - White Lies Free TV - Tubi.ts: Modification times differ by 769h30m37.728s: 2022-12-21 01:31:25.508 +0000 UTC, 2023-01-22 03:02:03.236 +0000 UTC
2023/02/15 23:04:28 ERROR : Tubi/Watch Flunk S01-E02 - White Lies Free TV - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:28 ERROR : Tubi/Watch Flunk S01-E02 - White Lies Free TV - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Flunk S01-E02 - White Lies Free TV - Tubi.ts: input/output error
2023/02/15 23:04:28 DEBUG : Tubi/Watch Flunk S01-E02 - White Lies Free TV - Tubi.ts: md5 differ
2023/02/15 23:04:28 DEBUG : Tubi/Watch Flunk S01-E03 - A Night to Remember Free TV - Tubi.ts: Modification times differ by 770h16m25.134s: 2022-12-21 01:32:43.821 +0000 UTC, 2023-01-22 03:49:08.955 +0000 UTC
2023/02/15 23:04:28 ERROR : Tubi/Watch Flunk S01-E03 - A Night to Remember Free TV - Tubi.ts: Failed to calculate src hash: hash: failed to read: context canceled
2023/02/15 23:04:28 ERROR : Tubi/Watch Flunk S01-E03 - A Night to Remember Free TV - Tubi.ts: Failed to calculate dst hash: hash: failed to read: read /content/drives/ge/movie/Tubi/Watch Flunk S01-E03 - A Night to Remember Free TV - Tubi.ts: input/output error
2023/02/15 23:04:28 DEBUG : Tubi/Watch Flunk S01-E03 - A Night to Remember Free TV - Tubi.ts: md5 differ
2023/02/15 23:04:28 DEBUG : Tubi/Watch Flunk S01-E04 - New Love Free TV - Tubi.ts: Modification times differ by 769h24m55.521s: 2022-12-21 01:33:41.901 +0000 UTC, 2023-01-22 02:58:37.422 +0000 UTC
2023/02/15 23:04:28 INFO  :   695.391 MiB / 34.154 GiB, 2%, 35.716 MiB/s, ETA 15m59s (xfr#0/37, chk#551/784)
2023/02/15 23:05:52 DEBUG : d/1276263598790.srt: Modification times differ by 5079h23m56.555s: 2022-06-27 22:21:04.951 +0000 UTC, 2023-01-25 13:45:01.506 +0000 UTC
2023/02/15 23:05:53 INFO  :     3.817 GiB / 415.336 GiB, 1%, 37.720 MiB/s, ETA 3h6m11s (xfr#3/569, chk#828/4987)
2023/02/15 23:05:58 INFO  :     4.003 GiB / 417.017 GiB, 1%, 37.809 MiB/s, ETA 3h6m25s (xfr#3/570, chk#828/5070)
2023/02/15 23:06:03 INFO  :     4.188 GiB / 418.732 GiB, 1%, 37.846 MiB/s, ETA 3h6m56s (xfr#3/571, chk#828/5152)
2023/02/15 23:06:08 INFO  :     4.374 GiB / 418.732 GiB, 1%, 37.923 MiB/s, ETA 3h6m28s (xfr#5/571, chk#828/5265)
2023/02/15 23:06:13 INFO  :     4.560 GiB / 428.166 GiB, 1%, 37.952 MiB/s, ETA 3h10m29s (xfr#5/580, chk#828/5440)
2023/02/15 23:06:18 INFO  :     4.746 GiB / 430.663 GiB, 1%, 37.959 MiB/s, ETA 3h11m29s (xfr#5/591, chk#828/5523)
^C

asdffdsa · February 16, 2023, 12:12am

hello and welcome to the forum,

checking does not download files. use api calls to do things like get a list of files in a folder.

the two command and the debug log, are local to local, not accessing gdrive?

some kind of network error?

ANAZ_A · February 16, 2023, 7:59am

Every time I try to copy I get this problem. This error came last 3 days. And after 24 hours I can't download from that drive. but there is no problem for uploading

Marcus1 · February 16, 2023, 8:57am

I must say upload and download quotas for Google Drives are different.

Upload is 750GB/day if I am not mistaken, and this is publicly disclosed info.

Download is about 10TB/day and this was not publicly disclosed last time I checked, the community guessed it by trial and error.

Then maybe depending on which flavor of Google Workspace you are using Google might adjust those quotas. Dunno.

Hope it helps to clarify a bit.

ANAZ_A · February 16, 2023, 9:13am

i know about upload and download limit of gdrive. what will i do to avoid download limit exceeded error?

Animosity022 · February 16, 2023, 12:15pm

You really don't.

The limits you are hitting are not documented so there isn't a work around.

You can ask Google what limit you are breaking (They won't say).

You have to just wait it out until you can.

ANAZ_A · February 16, 2023, 1:21pm

i only transfered 20 gb of data

Animosity022 · February 16, 2023, 1:24pm

I can't answer why you hit an undocumented limit that Google won't share or give you a reason for.

Feel free to ask them and see what you get.

ANAZ_A · February 16, 2023, 9:51pm

problem solved. i used --size-only flag

Animosity022 · February 16, 2023, 9:53pm

size-only has nothing do with that error.

Size only means instead of checking the md5sums on a file, it only compare if the size of a file matches and ignores it.

ANAZ_A · February 17, 2023, 5:30am

i dont know, its helped me. Because of this I was able to transfer 350gb without download limit.

iam using this notebook for mount drive and rclone. do you know the command for mounting rclone?

system · March 19, 2023, 5:30am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.