VFS Upload Priority/Burst

As far as I'm aware, there is no documentation on this idea, so my apologies if this should have been a help/support thread.

Feature

To set upload priorities for VFS cache, such as smaller files first. Alternatively, allow an option to ignore the --transfers value for files smaller than X, where X is a file size (1M, 100K, etc)

Example

Currently, I use vfs writes mode. If I have more files to upload than the value set by --transfers, then the upload is queued. So far, this is 100% expected behavior. However, if the current transfers are large enough, it could take a long time to get through the queue. Again, this 100% expected. The only issue then is that smaller files, although waiting just like all other items in the queue, seem to wait disproportionately to their file size. To summarize, smaller files can quickly be uploaded, refining the queue to just be large files waiting to be uploaded. Allowing smaller files a higher priority shouldn't have much of an impact on bandwidth (obviously), therefore the current transfer wont be affected much.

The Implementation

As noted above, a feature to do either 1 of 2 things could accomplish this:

  1. Allow smaller files to move up in the queue, taking priority over age in the queue position.

    • Use a file size and suffix to note the size of the file which will receive higher priority.
    • an example flag could be --vfs-transfers-priority-below SizeSuffix for prioritizing files smaller than the stated size and --vfs-transfers-priority-above SizeSuffix for prioritizing files larger than the stated size
  2. Allow a "burst" transfer for files

    • Allow an option that would ignore the hard limit set by --transfers and burst temporarily with files smaller than a specifed size
    • Use a file size and suffix to note the size of the file which will be considered for the burst transfer
    • example flags could be --vfs-transfers-burst-size SizeSuffix to set the file size for the burst and --vfs-transfers-burst INT to set how many transfers the burst is allowed to do. The burst should be higher than the --transfers but should be set appropriately, as to not impact performance of uploading too many files at once. If the burst limit is set with no size flag provided, then the burst size would default to 0, effectively making the burst limit benign. Note, this option only considers smaller files, as allowing a "burst" for larger files doesn't accomplish anything that the normal --transfers INT would do.

I have also made my case for this here.
Have you opened a GIT issue yet? i havent gotten around to it. sorry.

Rclone has the --order-by flag for when using rclone copy/sync.

Is that the kind of control you were thinking of?

So rclone could build a priority queue for the files which need uploading and use the --order-by flag to prioritise them.

I think --order-by is a great step. Also, maybe adding the queue details into the log would be useful (i.e. when vfs logs every minute an when an item is getting queued, it would be nice to know what is next up in the queue or to have a separate queue log).

There's an easy way to accomplish this without any developmental changes.

Use a union mount to combine your local disk with your cloud remote, then have a cron job or two tmux instances uploading your local folder to your cloud remote. Use filters so that instance 1 uploads only your large files above X size, with --transfers 2 or something, and instance 2 could upload anything below X size with --transfers 6.

If you're a windows person, cron is a scheduling tool, and tmux is a terminal multiplexer which allows you to run shells in the background.

Yeah I just got away from using union file systems. I’m not using them again. The benefits of VFS far outweigh the benefits of ufs for me. Even if there’s no development on this feature request, the new setup is miles better than the old, which relied on UFS

Can you not use VFS and union?

If you'd like to see that can you make a feature request on Github.

Not difficult but might be a bit verbose...

New feature request issued:

1 Like

Opened issue:

1 Like