InternetArchive remote - shouldn't a folder be a single item? (and other issues)

I am trying to upload several hundred files to IA

What is the problem you are having with rclone?

2025/01/03 21:01:43 ERROR : 1201-gpl.tar.gz: Failed to copy: HTTP error 409 (409 Conflict) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>BucketAlreadyExists</Code><Message>The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.</Message><Resource /><RequestId>f3864b2d-9bc8-4c81-8c52-00ff45121012</RequestId></Error>"

This suggests that rclone was creating an item per file instead of an item for all the files I was uploading

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2
- os/version: unknown
- os/kernel: 5.13.x (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none

 $ rclone selfupdate
2025/01/03 21:08:25 NOTICE: rclone is up to date

Which cloud storage system are you using? (eg Google Drive)

Internet Archive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -v . IA:tp-link-gpl-source-2024

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[IA]
type = internetarchive
access_key_id = XXX
secret_access_key = XXX
disable_checksum = false

A log from the command that you were trying to run with the -vv flag

# Command re-running, I will update the log when possible - it *seems* to be working now

2025/01/03 21:19:00 INFO  :
Transferred:      185.016 MiB / 369.248 GiB, 0%, 1.432 MiB/s, ETA 3d1h18m
Checks:                 1 / 1, 100%
Transferred:            0 / 786, 0%
Elapsed time:       2m0.5s
Transferring:
 *                                      11.4.tar:  1% /2.575Gi, 356.452Ki/s, 2h4m3s
 *                               1201-gpl.tar.gz:  9% /403.831Mi, 332.860Ki/s, 18m40s
 *                                 150Router.tar:  5% /812.939Mi, 361.367Ki/s, 36m6s
 *                               2300-gpl.tar.gz: 15% /326.980Mi, 415.559Ki/s, 11m17s

now getting

2025/01/03 21:44:47 DEBUG : pacer: low level retry 6/10 (error HTTP error 503 (503 Slow Down) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><Resource>total_tasks_queued exceeds global_limit</Resource><RequestId>1a181eef-142a-45b9-b6ff-1352d97a7cab</RequestId></Error>")

Even with --transfers=1 it seems to be struggling - is there a --pacer setting I can use to slow it down further?

welcome to the forum,

might try tweaking

  • --tpslimit
  • --bwlimit

--low-level-retries=1
rclone will not continue to retry. so will lessen the load on iarchive and perhaps your account quota.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.