InternetArchive remote - shouldn't a folder be a single item? (and other issues)

I am trying to upload several hundred files to IA

What is the problem you are having with rclone?

2025/01/03 21:01:43 ERROR : 1201-gpl.tar.gz: Failed to copy: HTTP error 409 (409 Conflict) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>BucketAlreadyExists</Code><Message>The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.</Message><Resource /><RequestId>f3864b2d-9bc8-4c81-8c52-00ff45121012</RequestId></Error>"

This suggests that rclone was creating an item per file instead of an item for all the files I was uploading

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2
- os/version: unknown
- os/kernel: 5.13.x (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none

 $ rclone selfupdate
2025/01/03 21:08:25 NOTICE: rclone is up to date

Which cloud storage system are you using? (eg Google Drive)

Internet Archive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -v . IA:tp-link-gpl-source-2024

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[IA]
type = internetarchive
access_key_id = XXX
secret_access_key = XXX
disable_checksum = false

A log from the command that you were trying to run with the -vv flag

# Command re-running, I will update the log when possible - it *seems* to be working now

2025/01/03 21:19:00 INFO  :
Transferred:      185.016 MiB / 369.248 GiB, 0%, 1.432 MiB/s, ETA 3d1h18m
Checks:                 1 / 1, 100%
Transferred:            0 / 786, 0%
Elapsed time:       2m0.5s
Transferring:
 *                                      11.4.tar:  1% /2.575Gi, 356.452Ki/s, 2h4m3s
 *                               1201-gpl.tar.gz:  9% /403.831Mi, 332.860Ki/s, 18m40s
 *                                 150Router.tar:  5% /812.939Mi, 361.367Ki/s, 36m6s
 *                               2300-gpl.tar.gz: 15% /326.980Mi, 415.559Ki/s, 11m17s

now getting

2025/01/03 21:44:47 DEBUG : pacer: low level retry 6/10 (error HTTP error 503 (503 Slow Down) returned body: "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><Resource>total_tasks_queued exceeds global_limit</Resource><RequestId>1a181eef-142a-45b9-b6ff-1352d97a7cab</RequestId></Error>")

Even with --transfers=1 it seems to be struggling - is there a --pacer setting I can use to slow it down further?

welcome to the forum,

might try tweaking

  • --tpslimit
  • --bwlimit

--low-level-retries=1
rclone will not continue to retry. so will lessen the load on iarchive and perhaps your account quota.