Idea: Bypass a remotes character limit with .metadata file?

[nle]

I previously posted a question about the possibility of bypassing Jotta Clouds 255 character limit, and I'm wondering if a solution could be to store a metadata file that includes the whole path?

Shouldn't it be possible for rclone to write to and then read a metadata-file and present the full name (even if the actual name is ie. a hash of the file)

Let's say we have a long path and filename (my issue is usually that the path is too long):
a-very-long-path/that-contains-alot-of-sub-folders/really-alot-of-folders-with-looooooooooooooooooooooooooooooooooooooooooong-names/and-lets-throw-in-a-looooooooooooooooooooooooooooooooooooooooooong-filename-too/reeaaaaaaaaaly-looooooooooooooooong-filename.filetype [265 characters]

Tree of the same path:

a-very-long-path
└── that-contains-alot-of-sub-folders
    └── really-alot-of-folders-with-looooooooooooooooooooooooooooooooooooooooooong-names
        └── and-lets-throw-in-a-looooooooooooooooooooooooooooooooooooooooooong-filename-too
            └── reeaaaaaaaaaly-looooooooooooooooong-filename.filetype

What if rclone made a new metadata file based on the md5sum of the path/and/filename.ext stored the path and filename in that file to bypass the character limit

ie.

echo -n "a-very-long-path/that-contains-alot-of-sub-folders/really-alot-of-folders-with-looooooooooooooooooooooooooooooooooooooooooong-names/and-lets-throw-in-a-looooooooooooooooooooooooooooooooooooooooooong-filename-too/reeaaaaaaaaaly-looooooooooooooooong-filename.filetype" | md5sum
5ac280d8685b21e051274b9251302d5e`

rclone then created a file named 5ac280d8685b21e051274b9251302d5e.rcmeta that kept the path and file name information.

That means that when you list the remote natively (i.e via webinterface, etc) you would see:

5ac280d8685b21e051274b9251302d5e
5ac280d8685b21e051274b9251302d5e.rcmeta

But when you list the remote via rclone, rclone would read the .rcmeta file and present the listing as you would expect:

rclone ls remotet:
a-very-long-path/that-contains-alot-of-sub-folders/really-alot-of-folders-with-looooooooooooooooooooooooooooooooooooooooooong-names/and-lets-throw-in-a-looooooooooooooooooooooooooooooooooooooooooong-filename-too/reeaaaaaaaaaly-looooooooooooooooong-filename.filetype`

I hope this makes sense, but I have no idea how complex this would be to implement, but I guess it would be usefull on multiple remotes. Also I know there has been talks about storing hashes in metadata files, and I guess ie. .rcmeta could contain whatever metadata you want.

Thanks for reading my wall of text.

[ncw (Nick Craig-Wood)]

This idea has come up before - you can find it several times in the rclone issues.

It is harder than you might think though Rclone doesn't store any metadata on disk, so it would have to fetch the .rcmeta files when you did a listing which would really slow things down a lot.

Having one file per directory is probably a workable scheme but it would slow directory listings down a lot.

For max performance you'd try to only use it if there was a file that was too long so most of the time you wouldn't use it.

This could also be implemented by an rclone overlay (like crypt or chunker) so it would work with all backends.