Use shell access and `find` (or even rclone) to accomplish listR on SFTP

Feature
1

SFTP listing is very slow because of the handshake and lack of recursive listing. I know the remote already calls shell functions like md5sum and the like. It would be awesome if it could use a similar call to find so that it can list files and sizes. Or even something like testing if rclone is in the remote $PATH and using that to walk and provide the information.

This would greatly speed up rclone over ssh. And bring the listing more in line with the speed of rsync. (but add all of the great features of rclone such as encryption)

2

if over ssh, perhaps rclone on the client can communicate with rclone rcd daemon running on the server.

3

That could work if you have it running.

I wonder what an rclone protocol could look like. It would be great to have something like:

$ rclone serve rclone remote:

But you may ask:

  • Why not serve sftp: (A) Non-http protocol on its own port. Not great behind some firewalls and (b) slow (hence the original issue)
  • Why not serve webdav: This is promising and faster than sftp but webdav doesn't support hashes or mtimes. It would be great it they update it to do so (there is an issue about this on github...)

But still, my original question stands: Could the SFTP remote be sped up to support recursive listing? At least optionally?

4

took a look at the source code, and looks like rclone can run commands over remote
https://github.com/rclone/rclone/blob/0681a5c86a376c85139770e31e259ba483af76c5/backend/sftp/sftp.go#L1118

rclone serve sftp as it runs on the remote server, could be modified to use find or any number of other such commands.

from https://rclone.org/overview/#notes

³ WebDAV supports hashes when used with Owncloud and Nextcloud only.
⁴ WebDAV supports modtimes when used with Owncloud and Nextcloud only.
5

I am not concerned about the server commands. Just the client to any server (including rclone). It would be great for faster walking.

Yep. But there is #3147 and my forum post about it

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.