Add support for Internet Archive

Hello
Regarding uploading to archive.org
I tried this and it totally worked
https://ia800702.us.archive.org/27/items/ia-pex/ia
Now this library can be used to upload files to archive.org

ia help

A command line interface to Archive.org.

usage:
ia [--help | --version]
ia [--config-file FILE] [--log | --debug]
[--insecure] [--host HOST] []...

options:
-h, --help
-v, --version
-c, --config-file FILE Use FILE as config file.
-l, --log Turn on logging [default: False].
-d, --debug Turn on verbose logging [default: False].
-i, --insecure Use HTTP for all requests instead of HTTPS [default: false]
-H, --host HOST Host to use for requests (doesn't work for requests made to
s3.us.archive.org) [default: archive.org]

commands:
help Retrieve help for subcommands.
configure Configure ia.
metadata Retrieve and modify metadata for items on Archive.org.
upload Upload items to Archive.org.
download Download files from Archive.org.
delete Delete files from Archive.org.
search Search Archive.org.
tasks Retrieve information about your Archive.org catalog tasks.
list List files in a given item.
copy Copy files in archive.org items.
move Move/rename files in archive.org items.
reviews Submit/modify reviews for archive.org items.

Documentation for 'ia' is available at:

https://archive.org/services/docs/api/internetarchive/cli.html

See 'ia help ' for help on a specific command.

1 Like

I'd like to do an internet archive backend - I admire their mission and a lot of datahoarders have got stuff to upload there.

I couldn't find the source for the ia tool - can you see it?

this tool is developed by Internet Archive
docs for it are here
https://archive.org/services/docs/api/internetarchive/cli.html
source code

  1. create a free archive.org account
  2. get the s3 api keys here
  3. create a remote
[archive]
type = s3
provider = Other
access_key_id = 
secret_access_key = 
endpoint = s3.us.archive.org

run commands

list files

rclone lsf archive:HowDidWeFindOutAboutElectricity-English-Asimov --include=*.pdf --format=psh
electricitypix.pdf;441868;c953da763c29620fff8d8ea5f767e6c4

copy files

rclone copyto archive:HowDidWeFindOutAboutElectricity-English-Asimov ./downloads -vv --include=*.pdf --ignore-size 
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : rclone: Version "v1.55.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copyto" "archive:HowDidWeFindOutAboutElectricity-English-Asimov" "./downloads" "-vv" "--include=*.pdf" "--ignore-size"]
DEBUG : Creating backend with remote "archive:HowDidWeFindOutAboutElectricity-English-Asimov"
DEBUG : Creating backend with remote "./downloads"
DEBUG : fs cache: renaming cache item "./downloads" to be canonical "//?/C:/data/rclone/scripts/rr/other/archive/downloads"
DEBUG : HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent: Excluded
DEBUG : HowDidWeFindOutAboutElectricity-English-Asimov_files.xml: Excluded
DEBUG : HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml: Excluded
DEBUG : __ia_thumb.jpg: Excluded
DEBUG : electricitypix.gif: Excluded
DEBUG : electricitypix_abbyy.gz: Excluded
DEBUG : electricitypix_djvu.txt: Excluded
DEBUG : electricitypix_djvu.xml: Excluded
DEBUG : electricitypix_jp2.zip: Excluded
DEBUG : electricitypix_scandata.xml: Excluded
DEBUG : Local file system at //?/C:/data/rclone/scripts/rr/other/archive/downloads: Waiting for checks to finish
DEBUG : Local file system at //?/C:/data/rclone/scripts/rr/other/archive/downloads: Waiting for transfers to finish
DEBUG : electricitypix.pdf: MD5 = c953da763c29620fff8d8ea5f767e6c4 OK
INFO  : electricitypix.pdf: Copied (new)
INFO  : 
Transferred:   	  431.512k / 431.512 kBytes, 100%, 192.392 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.8s

mount

rclone mount archive:HowDidWeFindOutAboutElectricity-English-Asimov b:\mount\rclone\archive -vv 
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : rclone: Version "v1.55.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "mount" "archive:HowDidWeFindOutAboutElectricity-English-Asimov" "b:\\mount\\rclone\\archive" "-vv"]
DEBUG : Creating backend with remote "archive:HowDidWeFindOutAboutElectricity-English-Asimov"
INFO  : S3 bucket HowDidWeFindOutAboutElectricity-English-Asimov: poll-interval is not supported by this remote
DEBUG : Network mode mounting is disabled
DEBUG : Mounting on "b:\\mount\\rclone\\archive" ("archive HowDidWeFindOutAboutElec")
  1. be :grinning:
1 Like

That is cool. I was under the impression that the ia version of S3 wasn't very compatible and didn't work with rclone.

Are there any gotchas?

yes, very cool, i remember reading this book as a child, one of the many Asimov books i read as a child and purchased with money from working as a paperboy.
and now i can i use rclone to download it.

without --ignore-size, downloads fails.
ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 1 vs 441868

with --ignore-size, the downloads succeeds and uses md5
DEBUG : electricitypix.pdf: MD5 = c953da763c29620fff8d8ea5f767e6c4 OK

never tried to use rclone with ia before.
this weekend, i will do some more testing and report my findings.

It would be good to put s section in the docs about your adventures :slightly_smiling_face:

I created remote and when uploading with it problem appears

2021/04/23 15:01:35 ERROR : AcapelaINFOVOX3Fabiana.zip: Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>860d8cf4-adc4-40c1-bcb5-97dd8436faf9</RequestId></Error>
2021/04/23 15:01:35 ERROR : Attempt 1/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>860d8cf4-adc4-40c1-bcb5-97dd8436faf9</RequestId></Error>
2021/04/23 15:01:37 ERROR : AcapelaINFOVOX3Fabiana.zip: Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>6ae54974-8924-4ce1-8076-1b51b4c9096e</RequestId></Error>
2021/04/23 15:01:37 ERROR : Attempt 2/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>6ae54974-8924-4ce1-8076-1b51b4c9096e</RequestId></Error>
2021/04/23 15:01:39 ERROR : AcapelaINFOVOX3Fabiana.zip: Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>db8c0dd0-02a0-4242-a8bc-d1990f71f844</RequestId></Error>
2021/04/23 15:01:39 ERROR : Attempt 3/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>db8c0dd0-02a0-4242-a8bc-d1990f71f844</RequestId></Error>
2021/04/23 15:01:39 Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>db8c0dd0-02a0-4242-a8bc-d1990f71f844</RequestId></Error> 

as for those errors, first get this simple command working
change archive from whatever your remote is named

rclone ls archive:HowDidWeFindOutAboutElectricity-English-Asimov

if you still have a problem, then

  • post the top 20 lines of the debug log, not just a snippet from the middle.
  • post the config file

This works fine

     4068 HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent
     3562 HowDidWeFindOutAboutElectricity-English-Asimov_files.xml
     1071 HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml
    13486 __ia_thumb.jpg
    91685 electricitypix.gif
   441868 electricitypix.pdf
   563009 electricitypix_abbyy.gz
    42726 electricitypix_djvu.txt
   414087 electricitypix_djvu.xml
 27936575 electricitypix_jp2.zip
     4434 electricitypix_scandata.xml

Also this works fine
rclone lsjson archive:HowDidWeFindOutAboutElectricity-English-Asimov

[
{"Path":"HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent","Name":"HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent","Size":4068,"MimeType":"application/x-bittorrent","ModTime":"2020-02-26T21:58:53.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"HowDidWeFindOutAboutElectricity-English-Asimov_files.xml","Name":"HowDidWeFindOutAboutElectricity-English-Asimov_files.xml","Size":3562,"MimeType":"application/xml","ModTime":"2020-02-26T21:58:54.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml","Name":"HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml","Size":1071,"MimeType":"application/xml","ModTime":"2020-02-26T21:58:49.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"__ia_thumb.jpg","Name":"__ia_thumb.jpg","Size":13486,"MimeType":"image/jpeg","ModTime":"2018-07-07T04:39:49.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix.gif","Name":"electricitypix.gif","Size":91685,"MimeType":"image/gif","ModTime":"2016-05-29T06:54:50.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix.pdf","Name":"electricitypix.pdf","Size":441868,"MimeType":"application/pdf","ModTime":"2016-05-29T06:52:53.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_abbyy.gz","Name":"electricitypix_abbyy.gz","Size":563009,"MimeType":"application/octet-stream","ModTime":"2016-05-29T06:58:21.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_djvu.txt","Name":"electricitypix_djvu.txt","Size":42726,"MimeType":"text/plain","ModTime":"2016-05-29T06:58:44.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_djvu.xml","Name":"electricitypix_djvu.xml","Size":414087,"MimeType":"application/xml","ModTime":"2016-05-29T06:58:36.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_jp2.zip","Name":"electricitypix_jp2.zip","Size":27936575,"MimeType":"application/zip","ModTime":"2016-05-29T06:54:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_scandata.xml","Name":"electricitypix_scandata.xml","Size":4434,"MimeType":"application/xml","ModTime":"2016-05-29T06:58:43.000000000Z","IsDir":false,"Tier":"STANDARD"}
]

But with this code I am facing the problem
rclone lsjson archive:mxblind_Acapela_Voices

rclone lsjson archive:mxblind_Acapela_Voices
[
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Arabic.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Leila22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Mehdi22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Nizar22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Salma22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Canadian French/CanadianFrench.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Canadian French/Louise22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/British.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Graham22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Lucy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Peter22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Rachel22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Heather22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Kenny22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Nelly22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Ryan22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Tracy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/USEnglish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Alice22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Antoine22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Bruno22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Claire22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/French.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Julie22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Margaux22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Andreas22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/German.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Julia22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Klaus22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Sarah22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Chiara22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Fabiana22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Italian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Vittorio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Russian/Alyona22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Russian/Russian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Antonio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Ines22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Maria22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Spanish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Turkish/Ipek22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Turkish/Turkish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
{"Path":"mxblind_Acapela_Voices_archive.torrent","Name":"mxblind_Acapela_Voices_archive.torrent","Size":37562,"MimeType":"application/x-bittorrent","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_files.xml","Name":"mxblind_Acapela_Voices_files.xml","Size":13976,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.sqlite","Name":"mxblind_Acapela_Voices_meta.sqlite","Size":57344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T15:00:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.xml","Name":"mxblind_Acapela_Voices_meta.xml","Size":785,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:26.000000000Z","IsDir":false,"Tier":"STANDARD"}
]

Remote contents

[archive]
type = s3
provider = Other
access_key_id = xxxxxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxx
endpoint = s3.us.archive.org

run the command with -vv and post the entire log including the top lines.

rclone lsjson -vv archive:mxblind_Acapela_Voices

root@srv:~# rclone lsjson -vv archive:mxblind_Acapela_Voices
2021/04/23 18:35:14 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "lsjson" "-vv" "archive:mxblind_Acapela_Voices"]
2021/04/23 18:35:14 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/04/23 18:35:14 DEBUG : Creating backend with remote "archive:mxblind_Acapela_Voices"
[
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Arabic.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Leila22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Mehdi22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Nizar22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Salma22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Canadian French/CanadianFrench.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Canadian French/Louise22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/British.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Graham22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Lucy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Peter22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Rachel22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Heather22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Kenny22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Nelly22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Ryan22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Tracy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/USEnglish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Alice22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Antoine22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Bruno22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Claire22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/French.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Julie22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Margaux22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Andreas22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/German.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Julia22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Klaus22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Sarah22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Chiara22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Fabiana22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Italian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Vittorio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Russian/Alyona22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Russian/Russian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Antonio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Ines22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Maria22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Spanish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Turkish/Ipek22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Turkish/Turkish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
{"Path":"mxblind_Acapela_Voices_archive.torrent","Name":"mxblind_Acapela_Voices_archive.torrent","Size":37562,"MimeType":"application/x-bittorrent","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_files.xml","Name":"mxblind_Acapela_Voices_files.xml","Size":13976,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.sqlite","Name":"mxblind_Acapela_Voices_meta.sqlite","Size":57344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T15:00:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.xml","Name":"mxblind_Acapela_Voices_meta.xml","Size":785,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:26.000000000Z","IsDir":false,"Tier":"STANDARD"}
]
2021/04/23 18:35:19 DEBUG : 4 go routines active

rclone ls works
rclone lsjson does not work.

i did some testing, do not know why, perhaps @ncw does?

I have a vague memory of this. I don't think path filtering works for the ia s3 so you can't list individual directories.

If you add -R to lsjson to recurse fully then I think it will work - give that a try?

rclone -R -vv lsjson archive:mxblind_Acapela_Voices

2021/04/25 02:41:42 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "-R" "-vv" "lsjson" "archive:mxblind_Acapela_Voices"]
2021/04/25 02:41:42 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/04/25 02:41:42 DEBUG : Creating backend with remote "archive:mxblind_Acapela_Voices"
[
{"Path":"home/AcapelaVoices/Arabic/Arabic.zip","Name":"Arabic.zip","Size":4446225,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Leila22k_HQ.zip","Name":"Leila22k_HQ.zip","Size":108866357,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:42.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Mehdi22k_HQ.zip","Name":"Mehdi22k_HQ.zip","Size":103206324,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:35.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Nizar22k_HQ.zip","Name":"Nizar22k_HQ.zip","Size":117269781,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:51.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Salma22k_HQ.zip","Name":"Salma22k_HQ.zip","Size":132884928,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:57:01.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Canadian French/CanadianFrench.zip","Name":"CanadianFrench.zip","Size":1618912,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:52:47.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Canadian French/Louise22k_HQ.zip","Name":"Louise22k_HQ.zip","Size":110697675,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:21.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/British.zip","Name":"British.zip","Size":1379207,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:40.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Graham22k_HQ.zip","Name":"Graham22k_HQ.zip","Size":118815938,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:37.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Lucy22k_HQ.zip","Name":"Lucy22k_HQ.zip","Size":97077357,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:13.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Peter22k_HQ.zip","Name":"Peter22k_HQ.zip","Size":262403148,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:28.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Rachel22k_HQ.zip","Name":"Rachel22k_HQ.zip","Size":157657961,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:52.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Heather22k_HQ.zip","Name":"Heather22k_HQ.zip","Size":97672296,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Kenny22k_HQ.zip","Name":"Kenny22k_HQ.zip","Size":114647958,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:33.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Nelly22k_HQ.zip","Name":"Nelly22k_HQ.zip","Size":109838732,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:13.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Ryan22k_HQ.zip","Name":"Ryan22k_HQ.zip","Size":112047405,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:00.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Tracy22k_HQ.zip","Name":"Tracy22k_HQ.zip","Size":138838608,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/USEnglish.zip","Name":"USEnglish.zip","Size":1474772,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:04.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Alice22k_HQ.zip","Name":"Alice22k_HQ.zip","Size":94410031,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:48.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Antoine22k_HQ.zip","Name":"Antoine22k_HQ.zip","Size":72854988,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:08.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Bruno22k_HQ.zip","Name":"Bruno22k_HQ.zip","Size":93748499,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:15.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Claire22k_HQ.zip","Name":"Claire22k_HQ.zip","Size":89898286,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:55.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/French.zip","Name":"French.zip","Size":807879,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Julie22k_HQ.zip","Name":"Julie22k_HQ.zip","Size":82833376,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:02.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Margaux22k_HQ.zip","Name":"Margaux22k_HQ.zip","Size":90844355,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:22.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Andreas22k_HQ.zip","Name":"Andreas22k_HQ.zip","Size":218815215,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:59.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/German.zip","Name":"German.zip","Size":1754931,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:32.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Julia22k_HQ.zip","Name":"Julia22k_HQ.zip","Size":185610736,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:59:12.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Klaus22k_HQ.zip","Name":"Klaus22k_HQ.zip","Size":231435191,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:46.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Sarah22k_HQ.zip","Name":"Sarah22k_HQ.zip","Size":169396321,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:28.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Chiara22k_HQ.zip","Name":"Chiara22k_HQ.zip","Size":172179617,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:57:13.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Fabiana22k_HQ.zip","Name":"Fabiana22k_HQ.zip","Size":160171344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:57:25.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Italian.zip","Name":"Italian.zip","Size":1444226,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:16.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Vittorio22k_HQ.zip","Name":"Vittorio22k_HQ.zip","Size":926462633,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:14.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Russian/Alyona22k_HQ.zip","Name":"Alyona22k_HQ.zip","Size":145476836,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:34.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Russian/Russian.zip","Name":"Russian.zip","Size":3334423,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Antonio22k_HQ.zip","Name":"Antonio22k_HQ.zip","Size":153008667,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:55.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Ines22k_HQ.zip","Name":"Ines22k_HQ.zip","Size":145550756,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:45.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Maria22k_HQ.zip","Name":"Maria22k_HQ.zip","Size":108395880,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:03.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Spanish.zip","Name":"Spanish.zip","Size":1368836,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:06.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Turkish/Ipek22k_HQ.zip","Name":"Ipek22k_HQ.zip","Size":150044728,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:59:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Turkish/Turkish.zip","Name":"Turkish.zip","Size":1294367,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:59:14.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_archive.torrent","Name":"mxblind_Acapela_Voices_archive.torrent","Size":37563,"MimeType":"application/x-bittorrent","ModTime":"2021-04-23T23:16:30.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_files.xml","Name":"mxblind_Acapela_Voices_files.xml","Size":13977,"MimeType":"application/xml","ModTime":"2021-04-23T23:16:30.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.sqlite","Name":"mxblind_Acapela_Voices_meta.sqlite","Size":57344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T15:00:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.xml","Name":"mxblind_Acapela_Voices_meta.xml","Size":8496,"MimeType":"application/xml","ModTime":"2021-04-23T23:16:29.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home","Name":"home","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices","Name":"AcapelaVoices","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Arabic","Name":"Arabic","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Canadian French","Name":"Canadian French","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/English","Name":"English","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/English/British","Name":"British","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/English/USEnglish","Name":"USEnglish","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/French","Name":"French","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/German","Name":"German","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Italian","Name":"Italian","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Russian","Name":"Russian","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Spanish","Name":"Spanish","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Turkish","Name":"Turkish","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true}
]
2021/04/25 02:42:22 DEBUG : 4 go routines active

Is it possible to get the download link of the file within the json info?

The download link should just be put the Path on the bucket URL.

Try downloading a file with -vv --dump header and you'll see what I mean.