Just FYI, I released the tool (forum post, github) and took your suggestion. I created a proof-of-concept for two different approaches (easy: using known hashes, hard: tracking diffs).
IT WORKS... mostly. I did end up adding some additional artifacts to the backup (saving the computed diffs) which made it easier. The problem is, it's fragile. If a run is interrupted anywhere between the desired point and the end, then the information/artifacts are not representing truth. To address this, I (a) wrote them first so they will be there and (b) added prefix that is later removed so it is clear.
The end result is that it is doable in ideal conditions and, with some work, can be backed-out in less-than-ideal ones.
However, my conclusion was that, while doable, if point-in-time restore is your primary use case, this is not the right tool! (use restic or Kopia or the like).
It did also confirm that this can be thought of as "reverse incremental". You do not need anything prior to the desired restore point!
Thanks for the feedback!