What is the problem you are having with rclone?
I am using rclone mount (see exact command below). I tried the command with just basic options (--daemon --vfs-cache-mode full --cache-dir --log-file --log-level) and subsequently with various combinations and values of the performance-related flags that follow. I have not tried the flags that are commented out, at the very bottom, but listed them out just in case you recommend I use some of them. I saw the same problem (getting rate-limited by Google) for all variations of flags I tried. I was hoping that using --fast-list and --no-seek, lowering --drive-pacer-burst from 8 to 1, increasing drive-pacer-min-sleep from 100ms to 200ms, increasing drive-chunk-size from 8Mi to 128Mi, reducing transfers from 4 to 1 and reducing --checkers from 8 to 1, I would not be rate-limited, even if latency increased.
/usr/bin/rclone mount "$RCLONE_REMOTE" "$LOCAL_MOUNT_DIR" \
--daemon \
--vfs-cache-mode full \
--config="$RCLONE_CONFIG_FILE" \
--cache-dir="$RCLONE_CACHE_DIR" \
--log-file="$LOG_FILE" \
--log-level "$LOG_LEVEL" \
--fast-list \
--no-seek \
--drive-pacer-burst 1 \
--drive-pacer-min-sleep 200ms \
--drive-chunk-size 128Mi \
--transfers 1 \
--checkers 1 \
--buffer-size 16Mi \
--vfs-read-ahead 4Gi \
--vfs-read-chunk-streams 1 \
--vfs-read-chunk-size 128Mi \
# --vfs-read-chunk-size-limit 512Mi \
# --tpslimit 1 \
# --tpslimit-burst 1 \
# --max-read-ahead 128Ki \
# --no-modtime \
# --vfs-fast-fingerprint \
# --no-checksum \
- Most of my files are 10/100 KB. Many are of order <10 MB. I just have one large file that is 2 GB
- I do not have/stream videos, nor have/run executables from the mount.
- I manually browse my mount (no scripts etc.)
- My usual (non-heavy) operations with Google Drive are uploading, opening, deleting, renaming and moving files, and creating, opening, renaming and moving folders. However, I have not done any uploads, deletes, renames or moves of files since I made the mount. I have not created, renamed or moved folders since I made the mount. I was getting rate limiting errors just casually navigating the folder structure and opening the KB and MB files.
Run the command 'rclone version' and share the full output of the command.
$ rclone --version
rclone v1.68.1
- os/version: debian 12.1 (64 bit)
- os/kernel: 6.1.0-26-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Google Drive (personal, non-team/shared)
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
/usr/bin/rclone mount "$RCLONE_REMOTE" "$LOCAL_MOUNT_DIR" \
--daemon \
--vfs-cache-mode full \
--config="$RCLONE_CONFIG_FILE" \
--cache-dir="$RCLONE_CACHE_DIR" \
--log-file="$LOG_FILE" \
--log-level "$LOG_LEVEL" \
--fast-list \
--no-seek \
--drive-pacer-burst 1 \
--drive-pacer-min-sleep 200ms \
--drive-chunk-size 128Mi \
--transfers 1 \
--checkers 1 \
--buffer-size 16Mi \
--vfs-read-ahead 4Gi \
--vfs-read-chunk-streams 1 \
--vfs-read-chunk-size 128Mi \
# --vfs-read-chunk-size-limit 512Mi \
# --tpslimit 1 \
# --tpslimit-burst 1 \
# --max-read-ahead 128Ki \
# --no-modtime \
# --vfs-fast-fingerprint \
# --no-checksum \
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
$ rclone config redacted
[gdrive-remote]
type = drive
scope = drive
token = XXX
team_drive =
A log from the command that you were trying to run with the -vv
flag
2024/10/26 15:34:29 INFO : vfs cache: cleaned: objects 25 (was 25) in use 0, to upload 0, uploading 0, total size 556.985Mi (was 556.985Mi)
2024/10/26 15:34:29 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"domain": "googleapis.com",
"metadata": {
"consumer": "projects/202264815644",
"quota_limit": "defaultPerMinutePerProject",
"quota_limit_value": "420000",
"quota_location": "global",
"quota_metric": "drive.googleapis.com/default",
"service": "drive.googleapis.com"
},
"reason": "RATE_LIMIT_EXCEEDED"
},
{
"@type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Request a higher quota limit.",
"url": "https://cloud.google.com/docs/quotas/help/request_increase"
}
]
}
]
, rateLimitExceeded)
2024/10/26 15:34:29 DEBUG : pacer: Rate limited, increasing sleep to 1.03296117s
2024/10/26 15:34:29 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"domain": "googleapis.com",
"metadata": {
"consumer": "projects/202264815644",
"quota_limit": "defaultPerMinutePerProject",
"quota_limit_value": "420000",
"quota_location": "global",
"quota_metric": "drive.googleapis.com/default",
"service": "drive.googleapis.com"
},
"reason": "RATE_LIMIT_EXCEEDED"
},
{
"@type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Request a higher quota limit.",
"url": "https://cloud.google.com/docs/quotas/help/request_increase"
}
]
}
]
, rateLimitExceeded)
2024/10/26 15:34:29 DEBUG : pacer: Rate limited, increasing sleep to 2.702414426s
2024/10/26 15:34:30 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:202264815644'.
Details:
[
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"domain": "googleapis.com",
"metadata": {
"consumer": "projects/202264815644",
"quota_limit": "defaultPerMinutePerProject",
"quota_limit_value": "420000",
"quota_location": "global",
"quota_metric": "drive.googleapis.com/default",
"service": "drive.googleapis.com"
},
"reason": "RATE_LIMIT_EXCEEDED"
},
{
"@type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Request a higher quota limit.",
"url": "https://cloud.google.com/docs/quotas/help/request_increase"
}
]
}
]
, rateLimitExceeded)
2024/10/26 15:34:30 DEBUG : pacer: Rate limited, increasing sleep to 4.713340704s
2024/10/26 15:34:33 DEBUG : pacer: Reducing sleep to 0s
2024/10/26 15:35:29 DEBUG : Google drive root '': Checking for changes on remote