I had proposed a chunk-based deuplicating backend which, if ever done (doubtful) could easily also do this. The idea is that it uses content-defined chunking (a la restic) and then has a blob directory and a file directory. Each file the metadata and a list of blocks. I like the idea a lot but it is also a ton of work!
Something like what you said has also been discussed recently (I was dissenting because I think --backup-dir gets that job done too).