Hello, firstly thanks to the developers for all your work on a great product.
I am wondering if I can restrict the number of transfers per remote. I know I can restrict this using the transfers flag for a specific command however I am utilising the remote control functionality and executing commands using the _async flag. This means that if I execute two copy commands in succession the transfers will be restricted per command but both commands will be run in parallel which means the total number of concurrent transfers is cumulative. I guess what I'm asking for is some sort of cumulative queuing functionality per remote rather than just per command.
The remote I am working with at present - SFTP - also has a concurrent reads and writes option but this is per file as I understand it.
You can set config parameters on a per rc command level.
So something like "_config":{"Transfers": "5"} will enable you to set a different number of transfers for each copy job.
That isn't implemented at the moment. It would be possible to do that with a bit of work though and make a new --global-transfers flag or something like that. Maybe something accessible only over the API as normal CLI usage won't be affected.
I think what you are asking for is something like this (which is for the FTP backend not the SFTP backend)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
It turns out that is a really easy way of causing deadlocks so I'm not a big fan!
Understand the Transfers config option - that is per command
Yes the --global-transfers flag is what I'm talking about - ideally this would be a per remote setting i suppose to ensure that it covers any number of commands/jobs for that remote
No I wasn't suggesting a global concurrency setting - I just mentioned that setting in case it caused some confusion to what I'm asking
How do I add the global-transfers flag to the backlog? Do you think it should be for all backends or just sftp? I don't have any experience with any other backends so far but I would think it might be a useful flag.