Restrict Transfers per Remote

Hello, firstly thanks to the developers for all your work on a great product.

I am wondering if I can restrict the number of transfers per remote. I know I can restrict this using the transfers flag for a specific command however I am utilising the remote control functionality and executing commands using the _async flag. This means that if I execute two copy commands in succession the transfers will be restricted per command but both commands will be run in parallel which means the total number of concurrent transfers is cumulative. I guess what I'm asking for is some sort of cumulative queuing functionality per remote rather than just per command.

The remote I am working with at present - SFTP - also has a concurrent reads and writes option but this is per file as I understand it.

Hope this is clear.

Thanks.

I think if you make a new stats group for each command they will have independent delete stats.

Sorry Nick I don't understand how your answer relates to my question - can you elaborate?

Sorry I mis-understood your question!

You can set config parameters on a per rc command level.

So something like "_config":{"Transfers": "5"} will enable you to set a different number of transfers for each copy job.

That isn't implemented at the moment. It would be possible to do that with a bit of work though and make a new --global-transfers flag or something like that. Maybe something accessible only over the API as normal CLI usage won't be affected.

I think what you are asking for is something like this (which is for the FTP backend not the SFTP backend)

  --ftp-concurrency int              Maximum number of FTP simultaneous connections, 0 for unlimited

It turns out that is a really easy way of causing deadlocks so I'm not a big fan!

Thanks Nick.

  1. Understand the Transfers config option - that is per command
  2. Yes the --global-transfers flag is what I'm talking about - ideally this would be a per remote setting i suppose to ensure that it covers any number of commands/jobs for that remote
  3. No I wasn't suggesting a global concurrency setting - I just mentioned that setting in case it caused some confusion to what I'm asking

How do I add the global-transfers flag to the backlog? Do you think it should be for all backends or just sftp? I don't have any experience with any other backends so far but I would think it might be a useful flag.

Open a new issue on Github is the way!

To be useful it needs to be for all backends.

Do you want to try to implement this?