Fatal error encountered during rclone copy

Help and Support
1

What is the problem you are having with rclone?

When I try to copy a folder from Google Drive to Sharepoint, it copied for a while (24 hours), than crashed with a "fatal error encountered". The log (below) says that "Cancelling sync due to fatal error: quotaLimitReached: Quota limit reached"...I presume from the log that the quota limit was on Google Drive, but I am not sure. I am surprised that a quota limit would trigger a fatal error, rather than just thottling. I ran the copy operation again after 12 hours and got the same message (log below). Any suggestions to overcome these Quotas?
Suggested checkers? Transfers? Other flags?

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

  • os/version: Microsoft Windows 10 Enterprise 21H2 (64 bit)
  • os/kernel: 10.0.19044.2604 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Sharepoint

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -P "wnzlabgdrive:Crystallography" "zagottalab:wnzlab archives\Crystallography" --ignore-checksum --ignore-size --ignore-existing --drive-acknowledge-abuse --user-agent "ISV|rclone.org|rclone/v1.61.1" -vv --log-file rclonelog.txt

The rclone config contents with secrets removed.

Google Drive and Sharepoint configs, with the default ClientID and secret

Paste config here

A log from the command with the -vv flag

2023/03/12 09:25:13 DEBUG : pacer: Rate limited, increasing sleep to 2.671229245s
2023/03/12 09:25:13 DEBUG : pacer: Reducing sleep to 0s
2023/03/12 09:25:13 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.3: Starting multipart upload
2023/03/12 09:25:13 INFO  : Crystallography data/H2SegC/H2SegC_48/dtfind.log.2: Copied (new)
2023/03/12 09:25:13 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.5: Starting multipart upload
2023/03/12 09:25:14 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.3: Uploading segment 0/17658 size 17658
2023/03/12 09:25:14 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.5: Uploading segment 0/11079 size 11079
2023/03/12 09:25:15 INFO  : Crystallography data/H2SegC/H2SegC_48/dtfind.log.3: Copied (new)
2023/03/12 09:25:15 INFO  : Crystallography data/H2SegC/H2SegC_48/dtfind.log.5: Copied (new)
2023/03/12 09:25:15 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.6: Starting multipart upload
2023/03/12 09:25:15 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.4: Starting multipart upload
2023/03/12 09:25:15 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.8: Starting multipart upload
2023/03/12 09:25:15 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.7: Starting multipart upload
2023/03/12 09:25:16 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.6: Uploading segment 0/5167 size 5167
2023/03/12 09:25:16 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.4: Uploading segment 0/5142 size 5142
2023/03/12 09:25:16 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.8: Uploading segment 0/8125 size 8125
2023/03/12 09:25:16 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.log.7: Uploading segment 0/8122 size 8122
2023/03/12 09:25:16 INFO  : Crystallography data/H2SegC/H2SegC_48/dtfind.log.6: Copied (new)
2023/03/12 09:25:16 INFO  : Crystallography data/H2SegC/H2SegC_48/dtfind.log.4: Copied (new)
2023/03/12 09:25:17 INFO  : Crystallography data/H2SegC/H2SegC_48/dtfind.log.8: Copied (new)
2023/03/12 09:25:17 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/202264815644",
      "quota_limit": "defaultPerMinutePerProject",
      "quota_limit_value": "420000",
      "quota_location": "global",
      "quota_metric": "drive.googleapis.com/default",
      "service": "drive.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quota#requesting_higher_quota"
      }
    ]
  }
]
, rateLimitExceeded)
2023/03/12 09:25:17 DEBUG : pacer: Rate limited, increasing sleep to 1.781190783s
2023/03/12 09:25:17 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/202264815644",
      "quota_limit": "defaultPerMinutePerProject",
      "quota_limit_value": "420000",
      "quota_location": "global",
      "quota_metric": "drive.googleapis.com/default",
      "service": "drive.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quota#requesting_higher_quota"
      }
    ]
  }
]
, rateLimitExceeded)
2023/03/12 09:25:17 DEBUG : pacer: Rate limited, increasing sleep to 2.844016146s
2023/03/12 09:25:17 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/202264815644",
      "quota_limit": "defaultPerMinutePerProject",
      "quota_limit_value": "420000",
      "quota_location": "global",
      "quota_metric": "drive.googleapis.com/default",
      "service": "drive.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quota#requesting_higher_quota"
      }
    ]
  }
]
, rateLimitExceeded)
2023/03/12 09:25:17 DEBUG : pacer: Rate limited, increasing sleep to 4.652067194s
2023/03/12 09:25:17 INFO  : Crystallography data/H2SegC/H2SegC_48/dtfind.log.7: Copied (new)
2023/03/12 09:25:18 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/202264815644",
      "quota_limit": "defaultPerMinutePerProject",
      "quota_limit_value": "420000",
      "quota_location": "global",
      "quota_metric": "drive.googleapis.com/default",
      "service": "drive.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quota#requesting_higher_quota"
      }
    ]
  }
]
, rateLimitExceeded)
2023/03/12 09:25:18 DEBUG : pacer: Rate limited, increasing sleep to 8.082719795s
2023/03/12 09:25:20 DEBUG : pacer: Reducing sleep to 0s
2023/03/12 09:25:20 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.ref: Starting multipart upload
2023/03/12 09:25:21 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.ref: Uploading segment 0/16984679 size 10485760
2023/03/12 09:25:22 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.ref: Uploading segment 10485760/16984679 size 6498919
2023/03/12 09:25:23 DEBUG : Crystallography data/H2SegC/H2SegC_48/dtfind.ref: Cancelling multipart upload: quotaLimitReached: Quota limit reached
2023/03/12 09:25:23 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/202264815644",
      "quota_limit": "defaultPerMinutePerProject",
      "quota_limit_value": "420000",
      "quota_location": "global",
      "quota_metric": "drive.googleapis.com/default",
      "service": "drive.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quota#requesting_higher_quota"
      }
    ]
  }
]
, rateLimitExceeded)
2023/03/12 09:25:23 DEBUG : pacer: Rate limited, increasing sleep to 1.694729953s
2023/03/12 09:25:23 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/202264815644",
      "quota_limit": "defaultPerMinutePerProject",
      "quota_limit_value": "420000",
      "quota_location": "global",
      "quota_metric": "drive.googleapis.com/default",
      "service": "drive.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quota#requesting_higher_quota"
      }
    ]
  }
]
, rateLimitExceeded)
2023/03/12 09:25:23 DEBUG : pacer: Rate limited, increasing sleep to 2.677367128s
2023/03/12 09:25:23 ERROR : Crystallography data/H2SegC/H2SegC_48/dtfind.ref: Failed to copy: quotaLimitReached: Quota limit reached
2023/03/12 09:25:23 ERROR : Cancelling sync due to fatal error: quotaLimitReached: Quota limit reached
2023/03/12 09:25:23 DEBUG : pacer: Reducing sleep to 0s
2023/03/12 09:25:23 ERROR : Crystallography data/H2SegC/H2SegC_48/dtfind.ref.1: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ISMTWYiRDzOXCdL3gFAy-Zf6tkxvm9wr?alt=media": context canceled
2023/03/12 09:25:23 ERROR : Crystallography data/H2SegC/H2SegC_48/dtfind.ref.2: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/1IqmgOBn6k3m61Vja8w3Yndm5BlfrZbOs?alt=media": context canceled
2023/03/12 09:25:25 ERROR : Crystallography data/H2SegC/H2SegC_48/dtfind.ref.3: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/1J4NO2aD9ODwG9WQr-4UGRkUReFiXkqb3?alt=media": context canceled
2023/03/12 09:25:25 ERROR : Crystallography data/ALS20120924/Anne/HCN2+flav/A01/A01_proc/AutoMR_run_32_: error reading source directory: couldn't list directory: context canceled
2023/03/12 09:25:25 ERROR : Crystallography data/ALS20120924/Anne/HCN2+flav/A01/A01_proc/AutoMR_run_33_: error reading source directory: couldn't list directory: context canceled
2023/03/12 09:25:25 ERROR : Crystallography data/ALS20100728/t42/t42Proc/PDS/AutoBuild_run_2_: error reading source directory: couldn't list directory: context canceled
2023/03/12 09:25:25 ERROR : Crystallography data/RECYCLER/S-1-5-21-3919847096-3528151712-4222781828-1190: error reading source directory: couldn't list directory: context canceled
2023/03/12 09:25:25 ERROR : Crystallography data/ALS20120924/Anne/HCN2+flav/A01/A01_proc/AutoMR_run_27_: error reading source directory: couldn't list directory: context canceled
2023/03/12 09:25:25 ERROR : Crystallography data/ALS20120924/Anne/HCN2+flav/A01/A01_P4212/coot-molprobity: error reading source directory: couldn't list directory: context canceled
2023/03/12 09:25:25 DEBUG : OneDrive root 'wnzlab archives/Crystallography': Waiting for checks to finish
2023/03/12 09:25:25 DEBUG : OneDrive root 'wnzlab archives/Crystallography': Waiting for transfers to finish
2023/03/12 09:25:25 ERROR : Fatal error received - not attempting retries
2023/03/12 09:25:25 INFO  : 
Transferred:   	   30.411 MiB / 77.088 GiB, 0%, 994.507 KiB/s, ETA 22h34m7s
Errors:                10 (fatal error encountered)
Checks:             35229 / 35229, 100%
Transferred:          223 / 10231, 2%
Elapsed time:      2m56.0s

2023/03/12 09:25:25 DEBUG : 9 go routines active
2023/03/12 09:25:25 Failed to copy with 10 errors: last error was: quotaLimitReached: Quota limit reached
2

Google has upload and download quotas. Most likely you've hit one of them but you didn't share any details / log / version /etc so it's a guess.

3

Sorry...I accidently posted before finishing. I have edited my post with the log etc.

4

A plethora of posts on that:

5

Adding to /etc/hosts didn't work for me. The same error (see log below). Also, I am not using Hetzner. Could it be that I just have to wait some time for my quota to reset? 24 hours? Are there flags I should put on my copy to prevent going over my google drive quota?

2023/03/12 11:00:36 DEBUG : pacer: Rate limited, increasing sleep to 16.755967799s
2023/03/12 11:00:37 ERROR : Crystallography data/ALS20100520/ALS20100520/Trip8b_241-602x129/x129_1_050.x: Failed to copy: quotaLimitReached: Quota limit reached
2023/03/12 11:00:37 ERROR : Cancelling sync due to fatal error: quotaLimitReached: Quota limit reached
2023/03/12 11:00:37 NOTICE: Crystallography data/ALS20100520/ALS20100520/Trip8b_241-602x129/x129_1_046.x: Failed to cancel multipart upload: Delete "https://uwnetid.sharepoint.com/sites/og_zagottalab/_api/v2.0/drives/b!hxRTQyM7R0OJcwfgbzVxCUZY2qJaB0ZOh0Roll2yNXicGg17JO0CQpzcPxLbU0pP/items/01JKVHP4WODXV2HIK5BBCJIC5TAWIIFPZE/uploadSession?guid='acf9ebd4-d6fc-429d-9966-24a0b8bb091f'&overwrite=True&rename=False&dc=0&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvdXduZXRpZC5zaGFyZXBvaW50LmNvbUBmNmI2ZGQ1Yi1mMDJmLTQ0MWEtOTlhMC0xNjJhYzUwNjBiZDIiLCJpc3MiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAiLCJuYmYiOiIxNjc4NjQ0MDM2IiwiZXhwIjoiMTY3ODczMDQzNiIsImVuZHBvaW50dXJsIjoiUlBXS0J2dFh3MDczZnRZNEFnMWFBOWZGeUxLYlRieDBjL0RNZko4dzhiTT0iLCJlbmRwb2ludHVybExlbmd0aCI6IjI2NiIsImlzbG9vcGJhY2siOiJUcnVlIiwiY2lkIjoiTnpjeE5XRTNaak10Wm1WbE15MDBZek5rTFdKa1kyWXRabUkwWTJFd09XTXpOelk0IiwidmVyIjoiaGFzaGVkcHJvb2Z0b2tlbiIsInNpdGVpZCI6Ik5ETTFNekUwT0RjdE0ySXlNeTAwTXpRM0xUZzVOek10TURkbE1EWm1NelUzTVRBNSIsImFwcF9kaXNwbGF5bmFtZSI6InJjbG9uZSIsImdpdmVuX25hbWUiOiJXaWxsaWFtIiwiZmFtaWx5X25hbWUiOiJaYWdvdHRhIiwiYXBwaWQiOiJiMTU2NjVkOS1lZGE2LTQwOTItODUzOS0wZWVjMzc2YWZkNTkiLCJ0aWQiOiJmNmI2ZGQ1Yi1mMDJmLTQ0MWEtOTlhMC0xNjJhYzUwNjBiZDIiLCJ1cG4iOiJ6YWdvdHRhQHV3LmVkdSIsInB1aWQiOiIxMDAzQkZGRDg2MjhBMTAyIiwiY2FjaGVrZXkiOiIwaC5mfG1lbWJlcnNoaXB8MTAwM2JmZmQ4NjI4YTEwMkBsaXZlLmNvbSIsInNjcCI6Im15ZmlsZXMucmVhZCBhbGxmaWxlcy5yZWFkIG15ZmlsZXMud3JpdGUgYWxsZmlsZXMud3JpdGUgYWxsc2l0ZXMucmVhZCIsInR0IjoiMiIsInVzZVBlcnNpc3RlbnRDb29raWUiOm51bGwsImlwYWRkciI6IjIwLjE5MC4xNTQuMzIifQ.Qi9Mbm1UeDE1amQ0Z2t1S2NDTmw1SWlTUk5ydGNtTzlKdExsd0tTaGNoQT0": context canceled (upload failed due to: quotaLimitReached: Quota limit reached)
2023/03/12 11:00:37 ERROR : Crystallography data/ALS20100520/ALS20100520/Trip8b_241-602x129/x129_1_046.x: Failed to copy: quotaLimitReached: Quota limit reached
2023/03/12 11:00:37 DEBUG : pacer: Reducing sleep to 0s
2023/03/12 11:00:37 ERROR : Crystallography data/ALS20100520/ALS20100520/Trip8b_241-602x129/x129_1_049.x: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/1tq5u6rnVmHZM9MJXrIIOAqLqORz_r-Qo?alt=media": context canceled
2023/03/12 11:00:37 ERROR : Crystallography data/ALS20100728/x230-2/x230Proc: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:37 ERROR : Crystallography data/ALS20101021/y065/y065Proc/CCP4_DATABASE: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:37 ERROR : Crystallography data/ALS20120924/Anne/HCN2+flav/A01/Images: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:37 ERROR : Crystallography data/NSLS041404/spih-6_12: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:38 ERROR : Crystallography data/ALS20100520/H2SegCx90/proc/PDS/AutoSol_run_1_0/.TEMP: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:38 ERROR : Crystallography data/ALS20100728/t42/t42Proc/AutoBuild_run_2_/TEMP0/AutoBuild_run_3_/COMM: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:38 ERROR : Crystallography data/ALS20100520/ALS20100520/Trip8b_241-602x129/x129_1_048.x: Failed to copy: failed to open source object: open file failed: Get "https://www.googleapis.com/drive/v3/files/1to9hJwclxQLaZ7sBJk1Z2qmReoOkF3Y4?alt=media": context canceled
2023/03/12 11:00:38 ERROR : Crystallography data/ALS20100728/t26/AutoBuild_run_3_/TEMP0/PDS/AutoBuild_run_1_0: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:38 ERROR : Crystallography data/ALS20120924/Anne/HCN2+flav/A11/Images: error reading source directory: couldn't list directory: context canceled
2023/03/12 11:00:38 DEBUG : OneDrive root 'wnzlab archives/Crystallography': Waiting for checks to finish
2023/03/12 11:00:38 DEBUG : OneDrive root 'wnzlab archives/Crystallography': Waiting for transfers to finish
2023/03/12 11:00:38 ERROR : Fatal error received - not attempting retries
2023/03/12 11:00:38 INFO  : 
Transferred:   	    9.642 MiB / 58.133 GiB, 0%, 147.619 KiB/s, ETA 4d18h41m5s
Errors:                12 (fatal error encountered)
Checks:             31922 / 31922, 100%
Transferred:           80 / 5298, 2%
Elapsed time:       1m2.7s

2023/03/12 11:00:38 DEBUG : 21 go routines active
2023/03/12 11:00:38 Failed to copy with 12 errors: last error was: quotaLimitReached: Quota limit reached
6

No flags - if you hit the upload quota, you wait till it resets.

750GB per day.

7

I figured out the problem. I just go an email from "Sharepoint online is out of storage space". While my university tells me that I will have 25 TB of storage, in fact they only gave me 1 TB and I have to request the rest.

I am sorry, Animosity022 if I have wasted your time. It is good to know that there is a daily transfer limit of 750 GB on Google Drive. I will keep that in mind.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.