Rclone can't upload on internetarchive

[lm2]

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

blocked by 403

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.53.3-DEV

• os/arch: linux/amd64
• go version: go1.15.9

i use the default one of LMDE5

Which cloud storage system are you using? (eg Google Drive)

internet archive / S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

taken from here

Add support for Internet Archive

Hello Regarding uploading to archive.org I tried this and it totally worked https://ia800702.us.archive.org/27/items/ia-pex/ia Now this library can be used to upload files to archive.org ia help A command line interface to Archive.org. usage: ia [--help | --version] ia [--config-file FILE] [--log | --debug] [--insecure] [--host HOST] []... options: -h, --help -v, --version -c, --config-file FILE Use FILE as config file. -l, --log Turn on logging [default: False]. …

Connecting to Internet Archive via S3 Keys

Hello, I would like to push datasets to Internet Archive and they allow access via s3-api like keys http://archive.org/help/abouts3.txt S3CMD works, Rclone does not. I am currently using the following settings and getting an incorrect key error. type = s3 env_auth = false access_key_id = xxx secret_access_key = xxx region = endpoint = s3.us.archive.org location_constraint = acl = server_side_encryption = storage_class = Failed to copy: InvalidAccessKeyId: The AWS Access Key Id you…

but because of discourse common rules (i hate discourse), i can't add over two or four links per first users post...

[archive]
type = s3
provider = Other
access_key_id = myid
secret_access_key = myaccesskey
endpoint = s3.us.archive.org

rclone  sync -P /tmp/1/ archive:/
2024-05-05 14:53:01 ERROR : testtt: Failed to copy: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:01 ERROR : S3 root: not deleting files as there were IO errors
2024-05-05 14:53:01 ERROR : S3 root: not deleting directories as there were IO errors
2024-05-05 14:53:01 ERROR : Attempt 1/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:02 ERROR : testtt: Failed to copy: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:02 ERROR : S3 root: not deleting files as there were IO errors
2024-05-05 14:53:02 ERROR : S3 root: not deleting directories as there were IO errors
2024-05-05 14:53:02 ERROR : Attempt 2/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:03 ERROR : testtt: Failed to copy: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:03 ERROR : S3 root: not deleting files as there were IO errors
2024-05-05 14:53:03 ERROR : S3 root: not deleting directories as there were IO errors
2024-05-05 14:53:03 ERROR : Attempt 3/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.3s
2024/05/05 14:53:03 Failed to sync: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: , host id: 

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

Paste config here

A log from the command that you were trying to run with the -vv flag

Paste  log here

here it is

rclone  sync -P /tmp/1/ archive-remote:/
2024-05-05 14:53:01 ERROR : testtt: Failed to copy: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:01 ERROR : S3 root: not deleting files as there were IO errors
2024-05-05 14:53:01 ERROR : S3 root: not deleting directories as there were IO errors
2024-05-05 14:53:01 ERROR : Attempt 1/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:02 ERROR : testtt: Failed to copy: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:02 ERROR : S3 root: not deleting files as there were IO errors
2024-05-05 14:53:02 ERROR : S3 root: not deleting directories as there were IO errors
2024-05-05 14:53:02 ERROR : Attempt 2/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:03 ERROR : testtt: Failed to copy: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
2024-05-05 14:53:03 ERROR : S3 root: not deleting files as there were IO errors
2024-05-05 14:53:03 ERROR : S3 root: not deleting directories as there were IO errors
2024-05-05 14:53:03 ERROR : Attempt 3/3 failed with 1 errors and: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: , host id:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.3s
2024/05/05 14:53:03 Failed to sync: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: , host id: 

rclone version
rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.15.9

---

after doing

sudo -v ; curl https://rclone.org/install.sh | sudo bash

rclone  sync -P /tmp/1/ archive:/
2024/05/05 15:08:11 ERROR : testtt: Failed to copy: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.
2024/05/05 15:08:11 ERROR : S3 root: not deleting files as there were IO errors
2024/05/05 15:08:11 ERROR : S3 root: not deleting directories as there were IO errors
2024/05/05 15:08:11 ERROR : Attempt 1/3 failed with 2 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.
2024/05/05 15:08:12 ERROR : testtt: Failed to copy: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.
2024/05/05 15:08:12 ERROR : S3 root: not deleting files as there were IO errors
2024/05/05 15:08:12 ERROR : S3 root: not deleting directories as there were IO errors
2024/05/05 15:08:12 ERROR : Attempt 2/3 failed with 2 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.
2024/05/05 15:08:14 ERROR : testtt: Failed to copy: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.
2024/05/05 15:08:14 ERROR : S3 root: not deleting files as there were IO errors
2024/05/05 15:08:14 ERROR : S3 root: not deleting directories as there were IO errors
2024/05/05 15:08:14 ERROR : Attempt 3/3 failed with 2 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:         3.6s
2024/05/05 15:08:14 Failed to sync with 2 errors: last error was: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutObjectInput.Key.

rclone version
rclone v1.66.0
- os/version: linuxmint 5 (64 bit)
- os/kernel: 5.10.0-12-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

edit discourse: hell i lovely love when my : demands a space to avoid an army of

[kapitainsky]

You are using ancient rclone version. Many years old one. I doubt anybody remembers what bugs it had. You won’t receive any support I am afraid. Unless maybe some other IT archeology fans here.

[lm2]

as you read the full post until bottom, i updated with the cli/sudo process, and posted also the most recent rclone config with output errors

[kapitainsky]

Sorry:) when I see such old version listed I consider reading further waste of time.

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

[archive]
type = s3
provider = Other
access_key_id = myid
secret_access_key = myaccesskey
endpoint = s3.us.archive.org

that remote does not look correct.
type = s3 should be type = internetarchive
provider = Other should be removed
endpoint = s3.us.archive.org is default value, can be removed.

---

@lm2

archive:/

the correct syntax for s3 remote is remote:bucket/dir

fwiw, run commands with -vv --retries=1
that will enable debug output and keep the log smaller.

[lm2]

thanks for your reply, but

rclone -vv  sync -P /tmp/1/ archive:/ 
2024/05/05 16:26:58 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "-vv" "sync" "-P" "/tmp/1/" "archive-sailfishos:/"]
2024/05/05 16:26:58 DEBUG : Creating backend with remote "/tmp/1/"
2024/05/05 16:26:58 DEBUG : Using config file from "/home/toshiba/.config/rclone/rclone.conf"
2024/05/05 16:26:58 DEBUG : fs cache: renaming cache item "/tmp/1/" to be canonical "/tmp/1"
2024/05/05 16:26:58 DEBUG : Creating backend with remote "archive-sailfishos:/"
2024/05/05 16:26:58 DEBUG : fs cache: renaming cache item "archive-sailfishos:/" to be canonical "archive-sailfishos:"
2024/05/05 16:26:58 DEBUG : test: Making directory
2024/05/05 16:26:58 DEBUG : testtt: Need to transfer - File not found at Destination
2024/05/05 16:26:58 DEBUG : Internet Archive root: Waiting for checks to finish
2024/05/05 16:26:58 DEBUG : Internet Archive root: Waiting for transfers to finish
2024/05/05 16:26:58 ERROR : testtt: Failed to copy: Put "s3.us.archive.org/testtt": unsupported protocol scheme ""
2024/05/05 16:26:58 ERROR : Internet Archive root: not deleting files as there were IO errors
2024/05/05 16:26:58 ERROR : Internet Archive root: not deleting directories as there were IO errors
2024/05/05 16:26:58 ERROR : Attempt 1/3 failed with 1 errors and: Put "s3.us.archive.org/testtt": unsupported protocol scheme ""
2024/05/05 16:26:58 DEBUG : test: Making directory
2024/05/05 16:26:58 DEBUG : testtt: Need to transfer - File not found at Destination
2024/05/05 16:26:58 DEBUG : Internet Archive root: Waiting for checks to finish
2024/05/05 16:26:58 DEBUG : Internet Archive root: Waiting for transfers to finish
2024/05/05 16:26:58 ERROR : testtt: Failed to copy: Put "s3.us.archive.org/testtt": unsupported protocol scheme ""
2024/05/05 16:26:58 ERROR : Internet Archive root: not deleting files as there were IO errors
2024/05/05 16:26:58 ERROR : Internet Archive root: not deleting directories as there were IO errors
2024/05/05 16:26:58 ERROR : Attempt 2/3 failed with 1 errors and: Put "s3.us.archive.org/testtt": unsupported protocol scheme ""
2024/05/05 16:26:58 DEBUG : test: Making directory
2024/05/05 16:26:58 DEBUG : testtt: Need to transfer - File not found at Destination
2024/05/05 16:26:58 DEBUG : Internet Archive root: Waiting for checks to finish
2024/05/05 16:26:58 DEBUG : Internet Archive root: Waiting for transfers to finish
2024/05/05 16:26:58 ERROR : testtt: Failed to copy: Put "s3.us.archive.org/testtt": unsupported protocol scheme ""
2024/05/05 16:26:58 ERROR : Internet Archive root: not deleting files as there were IO errors
2024/05/05 16:26:58 ERROR : Internet Archive root: not deleting directories as there were IO errors
2024/05/05 16:26:58 ERROR : Attempt 3/3 failed with 1 errors and: Put "s3.us.archive.org/testtt": unsupported protocol scheme ""
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s
2024/05/05 16:26:58 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2024/05/05 16:26:58 DEBUG : 5 go routines active
2024/05/05 16:26:58 Failed to sync: Put "s3.us.archive.org/testtt": unsupported protocol scheme ""

:/tmp$ rclone version
rclone v1.66.0
- os/version: linuxmint 5 (64 bit)
- os/kernel: 5.10.0-12-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

[archive]
type = internetarchive
#provider = Other
access_key_id = 123
secret_access_key = 456
endpoint = s3.us.archive.org

[asdffdsa (jojothehumanmonkey)]

@lm2

archive-sailfishos:

what is that, where is the config?

@lm2

archive-sailfishos:/

as i mentioned, up above, don't you need a bucket?

[lm2]

an error of modified log.
it was not supposed to happens here, but it's true that the real name is that one, not just "archive".
my mistake (but it doesnt solves the problem)

btw, wht do you mean by "a bucket"?

as rclone transmits files to the online storage, im trying to get a specific folder, transfered to the ":/" aka root of the remote drive..

[asdffdsa (jojothehumanmonkey)]

@lm2

btw, wht do you mean by "a bucket"?

A bucket is a container for objects
Items (things with details pages) get mapped to S3 Buckets

@lm2

":/" aka root of the remote drive

that might work with gdrive, onedrive.
with S3 remotes, must specify a bucket.

rclone copy d:\files\file.ext iarchive:bucketname -vv
2024/05/05 12:27:53 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "d:\\files\\file.ext" "iarchive:bucketname" "-vv"]
2024/05/05 12:27:53 DEBUG : Creating backend with remote "d:\\files\\file.ext"
2024/05/05 12:27:53 DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
2024/05/05 12:27:53 DEBUG : fs cache: adding new entry for parent of "d:\\files\\file.ext", "//?/d:/files"
2024/05/05 12:27:53 DEBUG : Creating backend with remote "iarchive:bucketname"
2024/05/05 12:27:56 DEBUG : file.ext: Need to transfer - File not found at Destination
2024/05/05 12:28:19 DEBUG : file.ext: Dst hash empty - aborting Src hash check
2024/05/05 12:28:19 INFO  : file.ext: Copied (new)

[lm2]

thank you vm, i didnt understood that far

so the right command looks like :

rclone command

 rclone sync -P /source/path archive:/mydestdirectory/ --metadata --header "X-Archive-Meta-Mediatype: audio" --header "X-Archive-Meta-title: Hey This Works" --header "X-Archive-Meta-Creator: It's Me"  --dump headers

rclone.conf sample

[archive]
type = internetarchive
access_key_id = 1234
secret_access_key = 5678

#where sync could be replaced by copy
pls not it might not work that good with older versions of rclone

hope it will help some persons....