Timeout occurred while trying to lock a resource, please reduce your request rate status code: 503

Michael_ArchTech · November 14, 2022, 8:51am

What is the problem you are having with rclone?

I want to copy the minio bucket to the target directory and I get the error:

ERROR: error reading source directory: RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
         status code: 503

Run the command 'rclone version' and share the full output of the command.

 rclone version
rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.15.9

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy minio:minb /mnt/minio_backup/

The rclone config contents with secrets removed.

[minio]
type = s3
env_auth = false
access_key_id = *******
secret_access_key = *******
region = us-east-1
endpoint = http://192.168.244.11:9000
location_constraint =.
server_side_encryption =.

A log from the command with the `-vv` flag

 rclone copy minio:minb /mnt/minio_backup/ -vv
2022/11/14 11:48:43 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "copy" "minio:minb" "/mnt/minio_backup/" "-vv"]
2022/11/14 11:48:43 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/11/14 11:48:43 DEBUG : Creating backend with remote "minio:minb"
2022/11/14 11:48:43 DEBUG : Creating backend with remote "/mnt/minio_backup/"
2022/11/14 11:48:57 DEBUG : pacer: low level retry 1/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 17276744CCA36533, host id: )
2022/11/14 11:48:57 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022/11/14 11:49:02 DEBUG : pacer: low level retry 2/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 17276745F6D50D50, host id: )
2022/11/14 11:49:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/11/14 11:49:07 DEBUG : pacer: low level retry 3/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 17276747210E15D1, host id: )
2022/11/14 11:49:07 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/11/14 11:49:12 DEBUG : pacer: low level retry 4/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 172767484B3D47FA, host id: )
2022/11/14 11:49:12 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022/11/14 11:49:17 DEBUG : pacer: low level retry 5/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 17276749756BBA31, host id: )
2022/11/14 11:49:17 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022/11/14 11:49:22 DEBUG : pacer: low level retry 6/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727674A9F9CCC64, host id: )
2022/11/14 11:49:22 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022/11/14 11:49:27 DEBUG : pacer: low level retry 7/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727674BC9D30731, host id: )
2022/11/14 11:49:27 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2022/11/14 11:49:32 DEBUG : pacer: low level retry 8/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727674CF407DF5B, host id: )
2022/11/14 11:49:32 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2022/11/14 11:49:37 DEBUG : pacer: low level retry 9/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727674E1E34AFCC, host id: )
2022/11/14 11:49:37 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/11/14 11:49:42 DEBUG : pacer: low level retry 10/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727674F486154C8, host id: )
2022/11/14 11:49:42 ERROR : : error reading source directory: RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727674F486154C8, host id:
2022/11/14 11:49:42 DEBUG : Local file system at /mnt/minio_backup/: Waiting for checks to finish
2022/11/14 11:49:42 DEBUG : Local file system at /mnt/minio_backup/: Waiting for transfers to finish
2022/11/14 11:49:42 INFO  : There was nothing to transfer
2022/11/14 11:49:42 ERROR : Attempt 1/3 failed with 1 errors and: RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727674F486154C8, host id:
2022/11/14 11:49:43 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:       1m0.0s

ncw · November 14, 2022, 8:56am

The minio instance is saying rclone is being too fast. Try adding --tpslimit 10. You'll have to experiment to find the largest number which dosn't give those errors.

Michael_ArchTech · November 14, 2022, 9:19am

I try, no change:

 rclone copy minio:minb /mnt/minio_backup/ --tpslimit 0.1 -vv
2022/11/14 12:13:11 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "c                                                                         opy" "minio:minb" "/mnt/minio_backup/" "--tpslimit" "0.1" "-vv"]
2022/11/14 12:13:11 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/11/14 12:13:11 INFO  : Starting HTTP transaction limiter: max 0.1 transactions/s with burs                                                                         t 1
2022/11/14 12:13:11 DEBUG : Creating backend with remote "minio:minb"
2022/11/14 12:13:11 DEBUG : Creating backend with remote "/mnt/minio_backup/"
2022/11/14 12:13:36 DEBUG : pacer: low level retry 1/10 (error RequestTimeout: A timeout occurred while trying to lock a resource, please reduce your request rate
        status code: 503, request id: 1727689D4CA4E08D, host id: )
2022/11/14 12:13:36 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022/11/14 12:13:42 DEBUG : pacer: Reducing sleep to 0s
2022/11/14 12:14:11 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:       1m0.0s

2022/11/14 12:15:11 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:       2m0.0s

2022/11/14 12:16:11 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:       3m0.0s

2022/11/14 12:17:11 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:       4m0.0s

ncw · November 14, 2022, 11:47am

I think the cluster must be broken in some way. Can you ask the admin?

Michael_ArchTech · November 15, 2022, 10:58am

Thanks for the advice. It's not a cluster, it's a sandalone server.

ncw · November 15, 2022, 6:04pm

I think the standalone server must be broken in some way then! Can you ask whoever administers it for help?

system · December 15, 2022, 6:04pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.