SFTP command limiter

[mavatko]

Hello,

Would be nice to have some kind of command limiter for SFTP provider as well. I'm using it to sync out data from Egnyte storage, however they are imposing limits limits there. Webdav is not an option as they don't support connecting there with a service user.

In my case 20000 commands per user per 15-minutes.

I'm getting following messages in the logs during checks:
sftp: "Commands quota exceeded" (SSH_FX_FAILURE)

--retries-sleep helps a bit, but the problem is, rclone seems to be waiting for a failed file, but still tries to continue with it's checks for other files in its queue, effectively ddosing the backend.

For now these are my settings, will have to increase the sleep timer a bit since I'm syncing 120k files

--checkers=1
--transfers=6
--multi-thread-streams 1
--retries-sleep 20m
--fast-list

version 1.64.2

[ncw (Nick Craig-Wood)]

I think that --tpslimit works with the sftp backend also. Give it a go.

[mavatko]

that's what I was looking for thanks!, I was looking in sync and sftp man page, but not in general.

While it is doing the trick, would be nice to have possibility to configure it per provider. I suppose this is limiting both calls, at source and destination while doing compare. I don't think those are exactly 1:1 when using fast-list thou, so may be difficult to calculate optimal formula. Anyway thats just suggestion to squeeze the most out of it. Thanks a lot for this tool

[ncw (Nick Craig-Wood)]

Yes tpslimit does limit both sides. It doesn't seem to cause too much trouble in practice though.

If you create your backend via the API you can limit then individually, but rclone doesn't yet have a way of specifying that!