How to add metadata in Internet Archive (Archive.org)?

oknchm · January 10, 2023, 8:05pm

What is the problem you are having with rclone?

I want to add metadata with my uploads to Internet Archive but they're still being set to the defaults or not showing up at all.

For example, I set the metadata for creator as test_name, mediatype as movies, and collection as test_collection but creator is not showing up, mediatype becomes data, and collection becomes community.

I tried Internet Archive's CLI and setting metadata works fine there but it's slower than rclone to me, so I would still prefer rclone.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

os/version: ubuntu 18.04 (64 bit)
os/kernel: 5.10.147+ (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.19.4
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive, Internet Archive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy -vv shareddrive1:"My Archive/video.ts" internet_archive:"test_upload_441" --metadata --metadata-set creator=test_name --metadata-set mediatype=movies --metadata-set collection=test_collection --internetarchive-wait-archive 10m0s

The rclone config contents with secrets removed.

[shareddrive1]
type = drive
scope = drive
token = {"access_token":"redacted","token_type":"Bearer","refresh_token":"redacted","expiry":"2023-01-07T18:16:14.669471057Z"}
team_drive = [redacted]
root_folder_id =

[internet_archive]
type = internetarchive
access_key_id = [redacted]
secret_access_key = [redacted]

A log from the command with the `-vv` flag

2023/01/10 19:46:55 DEBUG : MetadataUpload map[collection:test_collection creator:test_name mediatype:movies]
2023/01/10 19:46:55 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "-vv" "shareddrive1:My Archive/video.ts" "internet_archive:test_upload_441" "--metadata" "--metadata-set" "creator=test_name" "--metadata-set" "mediatype=movies" "--metadata-set" "collection=test_collection" "--internetarchive-wait-archive" "10m0s"]
2023/01/10 19:46:55 DEBUG : Creating backend with remote "shareddrive1:My Archive/video.ts"
2023/01/10 19:46:55 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/01/10 19:46:57 DEBUG : fs cache: adding new entry for parent of "shareddrive1:My Archive/video.ts", "shareddrive1:"My Archive"
2023/01/10 19:46:57 DEBUG : Creating backend with remote "internet_archive:test_upload_441"
2023/01/10 19:46:57 DEBUG : ia_c.archive.org: detected overridden config - adding "{21IlZ}" suffix to name
2023/01/10 19:46:59 DEBUG : fs cache: renaming cache item "internet_archive:test_upload_441" to be canonical "internet_archive{21IlZ}:test_upload_441"
2023/01/10 19:47:00 DEBUG : video.ts: Need to transfer - File not found at Destination
2023/01/10 19:47:59 INFO  : 
Transferred:   	  493.347 MiB / 493.347 MiB, 100%, 2.798 MiB/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       1m4.3s
Transferring:
 *                 video.ts:100% /493.347Mi, 2.764Mi/s, 0s

2023/01/10 19:48:59 INFO  : 
Transferred:   	  493.347 MiB / 493.347 MiB, 100%, 59.633 KiB/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       2m4.3s
Transferring:
 *                 video.ts:100% /493.347Mi, 58.904Ki/s, 0s

2023/01/10 19:49:03 DEBUG : video.ts: md5 = 91080ff6eafa0f98e714904e9ad0a5b9 OK
2023/01/10 19:49:03 INFO  : video.ts: Copied (new)
2023/01/10 19:49:03 INFO  : 
Transferred:   	  493.347 MiB / 493.347 MiB, 100%, 46.065 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:       2m8.1s

2023/01/10 19:49:03 DEBUG : 4 go routines active

ncw · January 10, 2023, 10:56pm

Your command looks ok.

Try -vv --dump headers and see if you can see the metadata being set.

oknchm · January 11, 2023, 5:02pm

This is the only part in the full log where the metadata that I try to add appear. But they're still not being set in the final upload. Also noticed that there are two Mediatype here?

2023/01/11 16:35:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:56 DEBUG : HTTP REQUEST (req 0xc000680e00)
2023/01/11 16:35:56 DEBUG : PUT /test_upload_441_v2%2Fvideo.ts HTTP/1.1
Host: s3.us.archive.org
User-Agent: rclone/v1.61.1
Content-Length: 517311892
Authorization: XXXX
X-Amz-Auto-Make-Bucket: 1
X-Amz-Filemeta-Collection: test_collection
X-Amz-Filemeta-Creator: test_name
X-Amz-Filemeta-Mediatype: movies
X-Amz-Filemeta-Rclone-Mtime: 2022-12-29T15:41:06Z
X-Amz-Filemeta-Rclone-Update-Track: humedut0parixad5runohuj9fuvobud4
X-Archive-Auto-Make-Bucket: 1
X-Archive-Cascade-Delete: 1
X-Archive-Keep-Old-Version: 0
X-Archive-Meta-Mediatype: data
X-Archive-Queue-Derive: 0
X-Archive-Size-Hint: 517311892
Accept-Encoding: gzip

Here is the full log with --dump headers:

2023/01/11 16:35:49 DEBUG : MetadataUpload map[collection:test_collection creator:test_name mediatype:movies]
2023/01/11 16:35:49 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "-vv" "--dump" "headers" "shareddrive1:My Archive/video.ts" "internet_archive:test_upload_441_v2" "--metadata" "--metadata-set" "creator=test_name" "--metadata-set" "mediatype=movies" "--metadata-set" "collection=test_collection" "--internetarchive-wait-archive" "10m0s"]
2023/01/11 16:35:49 DEBUG : Creating backend with remote "shareddrive1:My Archive/video.ts"
2023/01/11 16:35:49 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/01/11 16:35:49 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/01/11 16:35:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:49 DEBUG : HTTP REQUEST (req 0xc000051600)
2023/01/11 16:35:49 DEBUG : POST /token HTTP/1.1
Host: oauth2.googleapis.com
User-Agent: rclone/v1.61.1
Content-Length: 803
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip

2023/01/11 16:35:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:49 DEBUG : HTTP RESPONSE (req 0xc000051600)
2023/01/11 16:35:49 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Wed, 11 Jan 2023 16:35:49 GMT
Server: scaffolding on HTTPServer2
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

2023/01/11 16:35:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:49 DEBUG : HTTP REQUEST (req 0xc00086a100)
2023/01/11 16:35:49 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AKDJa_rgMg5rUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%270AKDJa_rgMg5rUk9PVA%27+in+parents%29+and+%28name%3D%27My+Archive%27%29+and+%28mimeType%3D%27application%2Fvnd.google-apps.folder%27+or+mimeType%3D%27application%2Fvnd.google-apps.shortcut%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.19.4 gdcl/0.103.0
Accept-Encoding: gzip

2023/01/11 16:35:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:49 DEBUG : HTTP RESPONSE (req 0xc00086a100)
2023/01/11 16:35:49 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Wed, 11 Jan 2023 16:35:49 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: ESF
Vary: Origin, X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

2023/01/11 16:35:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:49 DEBUG : HTTP REQUEST (req 0xc00086ad00)
2023/01/11 16:35:49 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AKDJa_rgMg5rUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271cXoTdMJppD6IHp7IKzCSO1guN9Swh1sX%27+in+parents%29+and+%28name%3D%video.ts%27%29+and+%28mimeType%3D%27application%2Fvnd.google-apps.folder%27+or+mimeType%3D%27application%2Fvnd.google-apps.shortcut%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.19.4 gdcl/0.103.0
Accept-Encoding: gzip

2023/01/11 16:35:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:50 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:50 DEBUG : HTTP RESPONSE (req 0xc00086ad00)
2023/01/11 16:35:50 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Wed, 11 Jan 2023 16:35:50 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: ESF
Vary: Origin, X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

2023/01/11 16:35:50 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:50 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:50 DEBUG : HTTP REQUEST (req 0xc000050a00)
2023/01/11 16:35:50 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AKDJa_rgMg5rUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%270AKDJa_rgMg5rUk9PVA%27+in+parents%29+and+%28name%3D%27My+Archive%27%29+and+%28mimeType%3D%27application%2Fvnd.google-apps.folder%27+or+mimeType%3D%27application%2Fvnd.google-apps.shortcut%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.19.4 gdcl/0.103.0
Accept-Encoding: gzip

2023/01/11 16:35:50 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:51 DEBUG : HTTP RESPONSE (req 0xc000050a00)
2023/01/11 16:35:51 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Wed, 11 Jan 2023 16:35:51 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: ESF
Vary: Origin, X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

2023/01/11 16:35:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:51 DEBUG : HTTP REQUEST (req 0xc00086b200)
2023/01/11 16:35:51 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AKDJa_rgMg5rUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271cXoTdMJppD6IHp7IKzCSO1guN9Swh1sX%27+in+parents%29+and+%28name%3D%27video.ts%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.19.4 gdcl/0.103.0
Accept-Encoding: gzip

2023/01/11 16:35:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:51 DEBUG : HTTP RESPONSE (req 0xc00086b200)
2023/01/11 16:35:51 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Wed, 11 Jan 2023 16:35:51 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: ESF
Vary: Origin, X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

2023/01/11 16:35:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:51 DEBUG : fs cache: adding new entry for parent of "shareddrive1:My Archive/video.ts", "shareddrive1:My Archive"
2023/01/11 16:35:51 DEBUG : Creating backend with remote "internet_archive:test_upload_441_v2"
2023/01/11 16:35:51 DEBUG : internet_archive: detected overridden config - adding "{21IlZ}" suffix to name
2023/01/11 16:35:51 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/01/11 16:35:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:51 DEBUG : HTTP REQUEST (req 0xc000051800)
2023/01/11 16:35:51 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:35:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:54 DEBUG : HTTP RESPONSE (req 0xc000051800)
2023/01/11 16:35:54 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:35:54 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:35:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:54 DEBUG : fs cache: renaming cache item "internet_archive:test_upload_441_v2" to be canonical "internet_archive{21IlZ}:test_upload_441_v2"
2023/01/11 16:35:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:54 DEBUG : HTTP REQUEST (req 0xc00086b900)
2023/01/11 16:35:54 DEBUG : GET /drive/v3/files?alt=json&corpora=drive&driveId=0AKDJa_rgMg5rUk9PVA&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271cXoTdMJppD6IHp7IKzCSO1guN9Swh1sX%27+in+parents%29+and+%28name%3D%27video.ts%27%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Goog-Api-Client: gl-go/1.19.4 gdcl/0.103.0
Accept-Encoding: gzip

2023/01/11 16:35:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:54 DEBUG : HTTP RESPONSE (req 0xc00086b900)
2023/01/11 16:35:54 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Wed, 11 Jan 2023 16:35:54 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: ESF
Vary: Origin, X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

2023/01/11 16:35:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:54 DEBUG : HTTP REQUEST (req 0xc000680900)
2023/01/11 16:35:54 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:35:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:54 DEBUG : HTTP RESPONSE (req 0xc000680900)
2023/01/11 16:35:54 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:35:54 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:35:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:54 DEBUG : video.ts: Need to transfer - File not found at Destination
2023/01/11 16:35:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:54 DEBUG : HTTP REQUEST (req 0xc00086bd00)
2023/01/11 16:35:54 DEBUG : GET /drive/v3/files/1VSa6Fq_qJPEA-CoZMMvnJhuUVX_WQY0Z?alt=media HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:35:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:55 DEBUG : HTTP RESPONSE (req 0xc00086bd00)
2023/01/11 16:35:55 DEBUG : HTTP/1.1 200 OK
Content-Length: 517311892
Cache-Control: private, max-age=0, must-revalidate
Content-Disposition: attachment
Content-Type: video/vnd.dlna.mpeg-tts
Date: Wed, 11 Jan 2023 16:35:54 GMT
Expires: Wed, 11 Jan 2023 16:35:54 GMT
Server: UploadServer
Vary: Origin
Vary: X-Origin
X-Goog-Hash: crc32c=DnV7Bw==
X-Guploader-Uploadid: ADPycdsz3OnG8HlTCMpBu_LCaHLHzeXbIBv8Jgnt7YOEnOWtfEAXnZKiomaji0KtOV5Q4OKBS81WUCWjkXp5YpHpf5b1Tw

2023/01/11 16:35:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:35:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:35:56 DEBUG : HTTP REQUEST (req 0xc000680e00)
2023/01/11 16:35:56 DEBUG : PUT /test_upload_441_v2%2Fvideo.ts HTTP/1.1
Host: s3.us.archive.org
User-Agent: rclone/v1.61.1
Content-Length: 517311892
Authorization: XXXX
X-Amz-Auto-Make-Bucket: 1
X-Amz-Filemeta-Collection: test_collection
X-Amz-Filemeta-Creator: test_name
X-Amz-Filemeta-Mediatype: movies
X-Amz-Filemeta-Rclone-Mtime: 2022-12-29T15:41:06Z
X-Amz-Filemeta-Rclone-Update-Track: humedut0parixad5runohuj9fuvobud4
X-Archive-Auto-Make-Bucket: 1
X-Archive-Cascade-Delete: 1
X-Archive-Keep-Old-Version: 0
X-Archive-Meta-Mediatype: data
X-Archive-Queue-Derive: 0
X-Archive-Size-Hint: 517311892
Accept-Encoding: gzip

2023/01/11 16:35:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:36:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:31 DEBUG : HTTP RESPONSE (req 0xc000680e00)
2023/01/11 16:36:31 DEBUG : HTTP/1.1 200 Ok
Connection: close
Content-Length: 0
Accept-Ranges: bytes
Access-Control-Allow-Headers: authorization,x-amz-acl,x-amz-auto-make-bucket,cache-control,x-requested-with,x-file-name,x-file-size,x-archive-ignore-preexisting-bucket,x-archive-interactive-priority,x-archive-meta-title,x-archive-meta-description,x-archive-meta-language,x-archive-meta-mediatype,x-archive-meta01-subject,x-archive-meta02-subject,x-archive-meta03-subject,x-archive-meta04-subject,x-archive-meta05-subject,x-archive-meta01-collection,x-archive-meta02-collection
Access-Control-Allow-Methods: GET,POST,PUT,DELETE
Access-Control-Allow-Origin: *
Date: Wed, 11 Jan 2023 16:35:56 GMT
Etag: "91080ff6eafa0f98e714904e9ad0a5b9"
Server: Apache/2.4.41 (Ubuntu)

2023/01/11 16:36:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:36:31 DEBUG : HTTP REQUEST (req 0xc00011e100)
2023/01/11 16:36:31 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:36:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:36:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:33 DEBUG : HTTP RESPONSE (req 0xc00011e100)
2023/01/11 16:36:33 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:36:33 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:36:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:36:43 DEBUG : HTTP REQUEST (req 0xc00011f400)
2023/01/11 16:36:43 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:36:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:36:44 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:44 DEBUG : HTTP RESPONSE (req 0xc00011f400)
2023/01/11 16:36:44 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:36:44 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:36:44 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:36:54 DEBUG : HTTP REQUEST (req 0xc000050a00)
2023/01/11 16:36:54 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:36:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:36:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:54 DEBUG : HTTP RESPONSE (req 0xc000050a00)
2023/01/11 16:36:54 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:36:54 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:36:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:36:54 INFO  : 
Transferred:   	  493.347 MiB / 493.347 MiB, 100%, 1.932 MiB/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       1m4.6s
Transferring:
 *                 video.ts:100% /493.347Mi, 1.929Mi/s, 0s

2023/01/11 16:37:04 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:04 DEBUG : HTTP REQUEST (req 0xc000050e00)
2023/01/11 16:37:04 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:37:04 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:04 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:04 DEBUG : HTTP RESPONSE (req 0xc000050e00)
2023/01/11 16:37:04 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:37:04 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:37:04 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:14 DEBUG : HTTP REQUEST (req 0xc00086a600)
2023/01/11 16:37:14 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:37:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:14 DEBUG : HTTP RESPONSE (req 0xc00086a600)
2023/01/11 16:37:14 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:37:14 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:37:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:24 DEBUG : HTTP REQUEST (req 0xc000051200)
2023/01/11 16:37:24 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:37:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:24 DEBUG : HTTP RESPONSE (req 0xc000051200)
2023/01/11 16:37:24 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:37:24 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:37:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:34 DEBUG : HTTP REQUEST (req 0xc000051500)
2023/01/11 16:37:34 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:37:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:34 DEBUG : HTTP RESPONSE (req 0xc000051500)
2023/01/11 16:37:34 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:37:34 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:37:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:44 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:44 DEBUG : HTTP REQUEST (req 0xc00086a800)
2023/01/11 16:37:44 DEBUG : GET /metadata/test_upload_441_v2 HTTP/1.1
Host: archive.org
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/01/11 16:37:44 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/01/11 16:37:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:45 DEBUG : HTTP RESPONSE (req 0xc00086a800)
2023/01/11 16:37:45 DEBUG : HTTP/2.0 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Jan 2023 16:37:45 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=15724800
Vary: Accept-Encoding

2023/01/11 16:37:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/01/11 16:37:45 DEBUG : video.ts: md5 = 91080ff6eafa0f98e714904e9ad0a5b9 OK
2023/01/11 16:37:45 INFO  : video.ts: Copied (new)
2023/01/11 16:37:45 INFO  : 
Transferred:   	  493.347 MiB / 493.347 MiB, 100%, 73.612 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      1m55.5s

2023/01/11 16:37:45 DEBUG : 4 go routines active

ncw · January 11, 2023, 7:45pm

Hmm that looks ok to me. I wonder if @Lesmiscore has any ideas?

Lesmiscore · January 12, 2023, 1:25pm

Can you please check how it looks in /metadata endpoint of IA? (check https://archive.org/metadata/itemnamehere )
Since https://archive.org/details/test_upload_441_v2 is deleted, I can't confirm what metadata is actually written

system · February 11, 2023, 1:25pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.