Add support for Internet Archive

MesterPerfect · April 22, 2021, 5:27pm

Hello
Regarding uploading to archive.org
I tried this and it totally worked
https://ia800702.us.archive.org/27/items/ia-pex/ia
Now this library can be used to upload files to archive.org

ia help

A command line interface to Archive.org.

usage:
ia [--help | --version]
ia [--config-file FILE] [--log | --debug]
[--insecure] [--host HOST] []...

options:
-h, --help
-v, --version
-c, --config-file FILE Use FILE as config file.
-l, --log Turn on logging [default: False].
-d, --debug Turn on verbose logging [default: False].
-i, --insecure Use HTTP for all requests instead of HTTPS [default: false]
-H, --host HOST Host to use for requests (doesn't work for requests made to
s3.us.archive.org) [default: archive.org]

commands:
help Retrieve help for subcommands.
configure Configure ia.
metadata Retrieve and modify metadata for items on Archive.org.
upload Upload items to Archive.org.
download Download files from Archive.org.
delete Delete files from Archive.org.
search Search Archive.org.
tasks Retrieve information about your Archive.org catalog tasks.
list List files in a given item.
copy Copy files in archive.org items.
move Move/rename files in archive.org items.
reviews Submit/modify reviews for archive.org items.

Documentation for 'ia' is available at:

https://archive.org/services/docs/api/internetarchive/cli.html

See 'ia help ' for help on a specific command.

ncw · April 22, 2021, 7:48pm

I'd like to do an internet archive backend - I admire their mission and a lot of datahoarders have got stuff to upload there.

I couldn't find the source for the ia tool - can you see it?

MesterPerfect · April 22, 2021, 8:52pm

this tool is developed by Internet Archive
docs for it are here
https://archive.org/services/docs/api/internetarchive/cli.html
source code

asdffdsa · April 22, 2021, 9:28pm

create a free archive.org account
get the s3 api keys here
create a remote

[archive]
type = s3
provider = Other
access_key_id = 
secret_access_key = 
endpoint = s3.us.archive.org

run commands

list files

rclone lsf archive:HowDidWeFindOutAboutElectricity-English-Asimov --include=*.pdf --format=psh
electricitypix.pdf;441868;c953da763c29620fff8d8ea5f767e6c4

copy files

rclone copyto archive:HowDidWeFindOutAboutElectricity-English-Asimov ./downloads -vv --include=*.pdf --ignore-size 
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : rclone: Version "v1.55.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copyto" "archive:HowDidWeFindOutAboutElectricity-English-Asimov" "./downloads" "-vv" "--include=*.pdf" "--ignore-size"]
DEBUG : Creating backend with remote "archive:HowDidWeFindOutAboutElectricity-English-Asimov"
DEBUG : Creating backend with remote "./downloads"
DEBUG : fs cache: renaming cache item "./downloads" to be canonical "//?/C:/data/rclone/scripts/rr/other/archive/downloads"
DEBUG : HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent: Excluded
DEBUG : HowDidWeFindOutAboutElectricity-English-Asimov_files.xml: Excluded
DEBUG : HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml: Excluded
DEBUG : __ia_thumb.jpg: Excluded
DEBUG : electricitypix.gif: Excluded
DEBUG : electricitypix_abbyy.gz: Excluded
DEBUG : electricitypix_djvu.txt: Excluded
DEBUG : electricitypix_djvu.xml: Excluded
DEBUG : electricitypix_jp2.zip: Excluded
DEBUG : electricitypix_scandata.xml: Excluded
DEBUG : Local file system at //?/C:/data/rclone/scripts/rr/other/archive/downloads: Waiting for checks to finish
DEBUG : Local file system at //?/C:/data/rclone/scripts/rr/other/archive/downloads: Waiting for transfers to finish
DEBUG : electricitypix.pdf: MD5 = c953da763c29620fff8d8ea5f767e6c4 OK
INFO  : electricitypix.pdf: Copied (new)
INFO  : 
Transferred:   	  431.512k / 431.512 kBytes, 100%, 192.392 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.8s

mount

rclone mount archive:HowDidWeFindOutAboutElectricity-English-Asimov b:\mount\rclone\archive -vv 
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : rclone: Version "v1.55.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "mount" "archive:HowDidWeFindOutAboutElectricity-English-Asimov" "b:\\mount\\rclone\\archive" "-vv"]
DEBUG : Creating backend with remote "archive:HowDidWeFindOutAboutElectricity-English-Asimov"
INFO  : S3 bucket HowDidWeFindOutAboutElectricity-English-Asimov: poll-interval is not supported by this remote
DEBUG : Network mode mounting is disabled
DEBUG : Mounting on "b:\\mount\\rclone\\archive" ("archive HowDidWeFindOutAboutElec")

be

ncw · April 22, 2021, 9:55pm

That is cool. I was under the impression that the ia version of S3 wasn't very compatible and didn't work with rclone.

Are there any gotchas?

asdffdsa · April 22, 2021, 10:03pm

yes, very cool, i remember reading this book as a child, one of the many Asimov books i read as a child and purchased with money from working as a paperboy.
and now i can i use rclone to download it.

without --ignore-size, downloads fails.
ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 1 vs 441868

with --ignore-size, the downloads succeeds and uses md5
DEBUG : electricitypix.pdf: MD5 = c953da763c29620fff8d8ea5f767e6c4 OK

never tried to use rclone with ia before.
this weekend, i will do some more testing and report my findings.

ncw · April 22, 2021, 10:16pm

It would be good to put s section in the docs about your adventures

MesterPerfect · April 23, 2021, 1:11pm

I created remote and when uploading with it problem appears

2021/04/23 15:01:35 ERROR : AcapelaINFOVOX3Fabiana.zip: Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>860d8cf4-adc4-40c1-bcb5-97dd8436faf9</RequestId></Error>
2021/04/23 15:01:35 ERROR : Attempt 1/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>860d8cf4-adc4-40c1-bcb5-97dd8436faf9</RequestId></Error>
2021/04/23 15:01:37 ERROR : AcapelaINFOVOX3Fabiana.zip: Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>6ae54974-8924-4ce1-8076-1b51b4c9096e</RequestId></Error>
2021/04/23 15:01:37 ERROR : Attempt 2/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>6ae54974-8924-4ce1-8076-1b51b4c9096e</RequestId></Error>
2021/04/23 15:01:39 ERROR : AcapelaINFOVOX3Fabiana.zip: Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>db8c0dd0-02a0-4242-a8bc-d1990f71f844</RequestId></Error>
2021/04/23 15:01:39 ERROR : Attempt 3/3 failed with 1 errors and: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>db8c0dd0-02a0-4242-a8bc-d1990f71f844</RequestId></Error>
2021/04/23 15:01:39 Failed to copy: s3 upload: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><Resource /><RequestId>db8c0dd0-02a0-4242-a8bc-d1990f71f844</RequestId></Error>

asdffdsa · April 23, 2021, 1:23pm

as for those errors, first get this simple command working
change archive from whatever your remote is named

rclone ls archive:HowDidWeFindOutAboutElectricity-English-Asimov

if you still have a problem, then

post the top 20 lines of the debug log, not just a snippet from the middle.
post the config file

MesterPerfect · April 23, 2021, 4:15pm

This works fine

     4068 HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent
     3562 HowDidWeFindOutAboutElectricity-English-Asimov_files.xml
     1071 HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml
    13486 __ia_thumb.jpg
    91685 electricitypix.gif
   441868 electricitypix.pdf
   563009 electricitypix_abbyy.gz
    42726 electricitypix_djvu.txt
   414087 electricitypix_djvu.xml
 27936575 electricitypix_jp2.zip
     4434 electricitypix_scandata.xml

Also this works fine
rclone lsjson archive:HowDidWeFindOutAboutElectricity-English-Asimov

[
{"Path":"HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent","Name":"HowDidWeFindOutAboutElectricity-English-Asimov_archive.torrent","Size":4068,"MimeType":"application/x-bittorrent","ModTime":"2020-02-26T21:58:53.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"HowDidWeFindOutAboutElectricity-English-Asimov_files.xml","Name":"HowDidWeFindOutAboutElectricity-English-Asimov_files.xml","Size":3562,"MimeType":"application/xml","ModTime":"2020-02-26T21:58:54.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml","Name":"HowDidWeFindOutAboutElectricity-English-Asimov_meta.xml","Size":1071,"MimeType":"application/xml","ModTime":"2020-02-26T21:58:49.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"__ia_thumb.jpg","Name":"__ia_thumb.jpg","Size":13486,"MimeType":"image/jpeg","ModTime":"2018-07-07T04:39:49.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix.gif","Name":"electricitypix.gif","Size":91685,"MimeType":"image/gif","ModTime":"2016-05-29T06:54:50.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix.pdf","Name":"electricitypix.pdf","Size":441868,"MimeType":"application/pdf","ModTime":"2016-05-29T06:52:53.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_abbyy.gz","Name":"electricitypix_abbyy.gz","Size":563009,"MimeType":"application/octet-stream","ModTime":"2016-05-29T06:58:21.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_djvu.txt","Name":"electricitypix_djvu.txt","Size":42726,"MimeType":"text/plain","ModTime":"2016-05-29T06:58:44.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_djvu.xml","Name":"electricitypix_djvu.xml","Size":414087,"MimeType":"application/xml","ModTime":"2016-05-29T06:58:36.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_jp2.zip","Name":"electricitypix_jp2.zip","Size":27936575,"MimeType":"application/zip","ModTime":"2016-05-29T06:54:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"electricitypix_scandata.xml","Name":"electricitypix_scandata.xml","Size":4434,"MimeType":"application/xml","ModTime":"2016-05-29T06:58:43.000000000Z","IsDir":false,"Tier":"STANDARD"}
]

But with this code I am facing the problem
rclone lsjson archive:mxblind_Acapela_Voices

rclone lsjson archive:mxblind_Acapela_Voices
[
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Arabic.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Leila22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Mehdi22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Nizar22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Arabic/Salma22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Canadian French/CanadianFrench.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Canadian French/Louise22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/British.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Graham22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Lucy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Peter22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/British/Rachel22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Heather22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Kenny22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Nelly22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Ryan22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/Tracy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/English/USEnglish/USEnglish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Alice22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Antoine22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Bruno22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Claire22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/French.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Julie22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/French/Margaux22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Andreas22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/German.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Julia22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Klaus22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/German/Sarah22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Chiara22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Fabiana22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Italian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Italian/Vittorio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Russian/Alyona22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Russian/Russian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Antonio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Ines22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Maria22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Spanish/Spanish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Turkish/Ipek22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:11:13 ERROR : home/AcapelaVoices/Turkish/Turkish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
{"Path":"mxblind_Acapela_Voices_archive.torrent","Name":"mxblind_Acapela_Voices_archive.torrent","Size":37562,"MimeType":"application/x-bittorrent","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_files.xml","Name":"mxblind_Acapela_Voices_files.xml","Size":13976,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.sqlite","Name":"mxblind_Acapela_Voices_meta.sqlite","Size":57344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T15:00:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.xml","Name":"mxblind_Acapela_Voices_meta.xml","Size":785,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:26.000000000Z","IsDir":false,"Tier":"STANDARD"}
]

Remote contents

[archive]
type = s3
provider = Other
access_key_id = xxxxxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxx
endpoint = s3.us.archive.org

asdffdsa · April 23, 2021, 4:19pm

run the command with -vv and post the entire log including the top lines.

MesterPerfect · April 23, 2021, 4:36pm

rclone lsjson -vv archive:mxblind_Acapela_Voices

root@srv:~# rclone lsjson -vv archive:mxblind_Acapela_Voices
2021/04/23 18:35:14 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "lsjson" "-vv" "archive:mxblind_Acapela_Voices"]
2021/04/23 18:35:14 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/04/23 18:35:14 DEBUG : Creating backend with remote "archive:mxblind_Acapela_Voices"
[
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Arabic.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Leila22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Mehdi22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Nizar22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Arabic/Salma22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Canadian French/CanadianFrench.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Canadian French/Louise22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/British.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Graham22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Lucy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Peter22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/British/Rachel22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Heather22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Kenny22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Nelly22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Ryan22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/Tracy22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/English/USEnglish/USEnglish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Alice22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Antoine22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Bruno22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Claire22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/French.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Julie22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/French/Margaux22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Andreas22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/German.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Julia22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Klaus22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/German/Sarah22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Chiara22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Fabiana22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Italian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Italian/Vittorio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Russian/Alyona22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Russian/Russian.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Antonio22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Ines22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Maria22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Spanish/Spanish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Turkish/Ipek22k_HQ.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
2021/04/23 18:35:15 ERROR : home/AcapelaVoices/Turkish/Turkish.zip: Entry doesn't belong in directory "" (contains subdir) - ignoring
{"Path":"mxblind_Acapela_Voices_archive.torrent","Name":"mxblind_Acapela_Voices_archive.torrent","Size":37562,"MimeType":"application/x-bittorrent","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_files.xml","Name":"mxblind_Acapela_Voices_files.xml","Size":13976,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.sqlite","Name":"mxblind_Acapela_Voices_meta.sqlite","Size":57344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T15:00:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.xml","Name":"mxblind_Acapela_Voices_meta.xml","Size":785,"MimeType":"application/xml","ModTime":"2021-04-23T15:11:26.000000000Z","IsDir":false,"Tier":"STANDARD"}
]
2021/04/23 18:35:19 DEBUG : 4 go routines active

asdffdsa · April 23, 2021, 4:49pm

rclone ls works
rclone lsjson does not work.

i did some testing, do not know why, perhaps @ncw does?

ncw · April 24, 2021, 7:50pm

I have a vague memory of this. I don't think path filtering works for the ia s3 so you can't list individual directories.

If you add -R to lsjson to recurse fully then I think it will work - give that a try?

MesterPerfect · April 25, 2021, 12:46am

rclone -R -vv lsjson archive:mxblind_Acapela_Voices

2021/04/25 02:41:42 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "-R" "-vv" "lsjson" "archive:mxblind_Acapela_Voices"]
2021/04/25 02:41:42 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/04/25 02:41:42 DEBUG : Creating backend with remote "archive:mxblind_Acapela_Voices"
[
{"Path":"home/AcapelaVoices/Arabic/Arabic.zip","Name":"Arabic.zip","Size":4446225,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:27.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Leila22k_HQ.zip","Name":"Leila22k_HQ.zip","Size":108866357,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:42.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Mehdi22k_HQ.zip","Name":"Mehdi22k_HQ.zip","Size":103206324,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:35.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Nizar22k_HQ.zip","Name":"Nizar22k_HQ.zip","Size":117269781,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:51.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Arabic/Salma22k_HQ.zip","Name":"Salma22k_HQ.zip","Size":132884928,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:57:01.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Canadian French/CanadianFrench.zip","Name":"CanadianFrench.zip","Size":1618912,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:52:47.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Canadian French/Louise22k_HQ.zip","Name":"Louise22k_HQ.zip","Size":110697675,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:21.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/British.zip","Name":"British.zip","Size":1379207,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:40.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Graham22k_HQ.zip","Name":"Graham22k_HQ.zip","Size":118815938,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:37.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Lucy22k_HQ.zip","Name":"Lucy22k_HQ.zip","Size":97077357,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:13.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Peter22k_HQ.zip","Name":"Peter22k_HQ.zip","Size":262403148,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:28.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/British/Rachel22k_HQ.zip","Name":"Rachel22k_HQ.zip","Size":157657961,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:52.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Heather22k_HQ.zip","Name":"Heather22k_HQ.zip","Size":97672296,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Kenny22k_HQ.zip","Name":"Kenny22k_HQ.zip","Size":114647958,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:33.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Nelly22k_HQ.zip","Name":"Nelly22k_HQ.zip","Size":109838732,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:13.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Ryan22k_HQ.zip","Name":"Ryan22k_HQ.zip","Size":112047405,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:00.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/Tracy22k_HQ.zip","Name":"Tracy22k_HQ.zip","Size":138838608,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/English/USEnglish/USEnglish.zip","Name":"USEnglish.zip","Size":1474772,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:04.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Alice22k_HQ.zip","Name":"Alice22k_HQ.zip","Size":94410031,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:48.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Antoine22k_HQ.zip","Name":"Antoine22k_HQ.zip","Size":72854988,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:08.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Bruno22k_HQ.zip","Name":"Bruno22k_HQ.zip","Size":93748499,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:15.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Claire22k_HQ.zip","Name":"Claire22k_HQ.zip","Size":89898286,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:55:55.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/French.zip","Name":"French.zip","Size":807879,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Julie22k_HQ.zip","Name":"Julie22k_HQ.zip","Size":82833376,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:02.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/French/Margaux22k_HQ.zip","Name":"Margaux22k_HQ.zip","Size":90844355,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:56:22.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Andreas22k_HQ.zip","Name":"Andreas22k_HQ.zip","Size":218815215,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:59.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/German.zip","Name":"German.zip","Size":1754931,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:32.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Julia22k_HQ.zip","Name":"Julia22k_HQ.zip","Size":185610736,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:59:12.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Klaus22k_HQ.zip","Name":"Klaus22k_HQ.zip","Size":231435191,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:46.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/German/Sarah22k_HQ.zip","Name":"Sarah22k_HQ.zip","Size":169396321,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:28.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Chiara22k_HQ.zip","Name":"Chiara22k_HQ.zip","Size":172179617,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:57:13.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Fabiana22k_HQ.zip","Name":"Fabiana22k_HQ.zip","Size":160171344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:57:25.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Italian.zip","Name":"Italian.zip","Size":1444226,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:16.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Italian/Vittorio22k_HQ.zip","Name":"Vittorio22k_HQ.zip","Size":926462633,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:58:14.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Russian/Alyona22k_HQ.zip","Name":"Alyona22k_HQ.zip","Size":145476836,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:34.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Russian/Russian.zip","Name":"Russian.zip","Size":3334423,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Antonio22k_HQ.zip","Name":"Antonio22k_HQ.zip","Size":153008667,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:55.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Ines22k_HQ.zip","Name":"Ines22k_HQ.zip","Size":145550756,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:53:45.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Maria22k_HQ.zip","Name":"Maria22k_HQ.zip","Size":108395880,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:03.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Spanish/Spanish.zip","Name":"Spanish.zip","Size":1368836,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:54:06.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Turkish/Ipek22k_HQ.zip","Name":"Ipek22k_HQ.zip","Size":150044728,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:59:24.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home/AcapelaVoices/Turkish/Turkish.zip","Name":"Turkish.zip","Size":1294367,"MimeType":"application/octet-stream","ModTime":"2021-04-23T14:59:14.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_archive.torrent","Name":"mxblind_Acapela_Voices_archive.torrent","Size":37563,"MimeType":"application/x-bittorrent","ModTime":"2021-04-23T23:16:30.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_files.xml","Name":"mxblind_Acapela_Voices_files.xml","Size":13977,"MimeType":"application/xml","ModTime":"2021-04-23T23:16:30.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.sqlite","Name":"mxblind_Acapela_Voices_meta.sqlite","Size":57344,"MimeType":"application/octet-stream","ModTime":"2021-04-23T15:00:41.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"mxblind_Acapela_Voices_meta.xml","Name":"mxblind_Acapela_Voices_meta.xml","Size":8496,"MimeType":"application/xml","ModTime":"2021-04-23T23:16:29.000000000Z","IsDir":false,"Tier":"STANDARD"},
{"Path":"home","Name":"home","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices","Name":"AcapelaVoices","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Arabic","Name":"Arabic","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Canadian French","Name":"Canadian French","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/English","Name":"English","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/English/British","Name":"British","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/English/USEnglish","Name":"USEnglish","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/French","Name":"French","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/German","Name":"German","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Italian","Name":"Italian","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Russian","Name":"Russian","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Spanish","Name":"Spanish","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true},
{"Path":"home/AcapelaVoices/Turkish","Name":"Turkish","Size":-1,"MimeType":"inode/directory","ModTime":"2021-04-25T02:42:22.715320045+02:00","IsDir":true}
]
2021/04/25 02:42:22 DEBUG : 4 go routines active

MesterPerfect · April 25, 2021, 12:48am

Is it possible to get the download link of the file within the json info?

ncw · April 25, 2021, 10:15am

The download link should just be put the Path on the bucket URL.

Try downloading a file with -vv --dump header and you'll see what I mean.

system · June 25, 2021, 6:16am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.