I'm now writing Internet Archive support from the scratch, as adding it as S3 remote will cause rclone to be confused (and impression of degraded behavior).
Now, most of the functionalities including hash check work, except rclone deletefile.
Once I delete files using rclone delete or mount IA item (consider it "bucket" on S3) using rclone mount, files can be deleted, without any problem.
Here's the log when I tried removing test file uploaded at IA:
% ./rclone -vvP deletefile ia-test:lesmi-rclone-test/rclone
<7>DEBUG : rclone: Version "v1.59.0-beta.12.9c9c8d4.internetarchive" starting with parameters ["./rclone" "-vvP" "deletefile" "ia-test:lesmi-rclone-test/rclone"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "ia-test:lesmi-rclone-test/rclone"
<7>DEBUG : Using config file from "/home/lesmi/.config/rclone/rclone.conf"
2022-04-14 23:09:17 ERROR : Attempt 1/3 failed with 1 errors and: ia-test:lesmi-rclone-test/rclone is a directory or doesn't exist
2022-04-14 23:09:17 ERROR : Attempt 2/3 failed with 1 errors and: ia-test:lesmi-rclone-test/rclone is a directory or doesn't exist
2022-04-14 23:09:17 ERROR : Attempt 3/3 failed with 1 errors and: ia-test:lesmi-rclone-test/rclone is a directory or doesn't exist
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.0s
<7>DEBUG : 2 go routines active
Failed to deletefile: ia-test:lesmi-rclone-test/rclone is a directory or doesn't exist
Nice! I noticed there were lots of interesting hash formats it supports.
I think your problem is probably to do with mis-handling of the fs.ErrorIsFile error returned from NewFs. It is the backends job to identify files and return that error along with the pointer to the directory above.
Create a config entry called TestRemote for the unit tests to use
Create a backend/remote/remote_test.go - copy and adjust your example remote
Make sure all tests pass with go test -v
If you keep chipping away at errors in go test -v. Fix the first error, then the next and before you know it you'll have a completely working backend! It sounds like you've got to the stage that the tests should mostly work now.
The tests have all the detail needed to make sure that you'll make a first class rclone backend and that all the rclone commands will work!
That is the way I make new backends anyway - let the tests guide you as to what needs fixing.
After you've got that working, your backend is nearly ready - the test_all will winkle out any last incompatibilities.
I gave the code a quick once over.
user agent shouldn't be necessary. lib/rest sets it for you to something sensible and there is a global flag --user-agent to override.
hashes you've specified MD5,SHA1,SHA256 here but the IAFile Object has CRC32, MD5, SHA1
PutStream is very useful if you can implement it as it enables uploads from rclone mount with --vfs-cache-mode off without having to buffer to disk
Looking great! Keeping working through the tests and let me know if there is a problem you can't fix.
I think your problem is probably to do with mis-handling of the fs.ErrorIsFile error returned from NewFs. It is the backends job to identify files and return that error along with the pointer to the directory above.
I see the problem at NewFs code. That makes sense.
user agent shouldn't be necessary. lib/rest sets it for you to something sensible and there is a global flag --user-agent to override.
Removed User-Agent as IAS3 only mandates it, no requirement for its format.
hashes you've specified MD5,SHA1,SHA256 here but the IAFile Object has CRC32, MD5, SHA1
Oops, I didn't notice I misenumerated hashes. Keys from IAFile object is correct.
PutStream is very useful if you can implement it as it enables uploads from rclone mount with --vfs-cache-mode off without having to buffer to disk
Sadly it can't be done. (I pushed the code but will remove it in few hours)
IAS3 doesn't accept any requests without Content-Length header.
Their official IAS3 client dumps stdin to a temporary file to "stream" file to the server, for your reference.
When you are ready for your PR review I'll want to know the results of all the integration tests.
There are few more problems on server-side:
Reflecting files to download endpoint always delays, because of how it processes files. It will confuse test cases even though files are properly uploaded.
Modified time is taken by server's choice, so there will always be discrepancy between local and IA backend. We can add custom metadata fields upon upload, but it will bloat up top page of the Item. ("top page" here means pages like this: lesmi-rclone-test : Free Download, Borrow, and Streaming : Internet Archive where lesmi-rclone-test is the name of this Item)
The rclone test suite can deal with "eventual consistency" delays of a few 10s of seconds.
Experiment with the -listretries parameter to go test -v
Maybe it would be best to start without adding metadata? In that case you set the Precision returned by the backend to fs.ModTimeNotSupported then the test cases won't moan about the differing times.
So you can't add metadata to a file as it is being uploaded?
Apparently I've done that before! What is the etiquette for creating and uploading things like the rclone integration test directory?
Usually less than a second for smaller files used in tests, but it can take up to more than a day for very large files. (>800GB or so)
Yes, it can wait in this case. But fails with Modification time difference too big
Yes. IAS3 requires to amend it later.
You shouldn't let it make a lot of Items with tests, as cleanup is currently not implemented. (note to myself: make_dark)
Too high rate of upload requests could cause you banned.
P.S. attached logs from go test -v tests.log (48.7 KB)
It means that the modification times from the backend shouldn't be used for syncing as they can't be set.
Lets get this working first and then we can think about whether we should be adding metadata to the objects or not to support the modified time. I think probably not as the internet archive isn't a general purpose backup but I might be wrong!
I thought you're right for here, and I was digging around there. But one thing I noticed was the files are remaining, even though the test cases delete them. Since files are correctly created and displayed on website, I doubt there's a huge problem on encoding.
