--buffer-size not working with --bwlimit

Help and Support
1

What is the problem you are having with rclone?

I changed something in my mount and my rclone copy no longer buffers when I am also using bwlimit. I have tried with and without using vfs cache without any difference.

I like the buffer because It makes my hdd goes in longer reads rather than constantly churning/seeking. Without bwlimit the buffer works just fine because I can see the size copied goes up in buffer chunks and then stops vs goes up in the bwlimit increments.

What is your rclone version (output from rclone version)

v1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --size-only --progress --fast-list --bwlimit=8.5M --log-file "Location" "Source" "Destination"

The rclone config contents with secrets removed.

mount --config "C:\..." --allow-other --dir-cache-time 1000h --vfs-read-chunk-size 32M  --drive-chunk-size 32M --vfs-cache-mode off  --buffer-size 64M --vfs-write-wait 10s --vfs-read-wait 60ms --transfers 4 gcrypt2: A:

A log from the command with the -vv flag

2020/07/04 20:09:36 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "copy" "--size-only" "--progress" "--stats-one-line" "--fast-list" "--bwlimit=8.5M" "--log-file" "Location" "Source" "Destination" "-vv"]
2020/07/04 20:09:36 DEBUG : Using config file from "C:\\..."
2020/07/04 20:09:36 INFO  : Starting bandwidth limiter at 8.500MBytes/s
2020/07/04 20:09:36 DEBUG : fs cache: renaming cache item "Source" to be canonical "..."
2020/07/04 20:09:36 DEBUG : fs cache: renaming cache item "Destination" to be canonical "..."
2020/07/04 20:09:40 DEBUG : "File" Sizes identical
2

Are you copying from the mount with rclone copy?

You have a bwlimit on the mount too so if you are the one on the mount will likely be doing all the limiting with the one on the copy staying idle...

1 Like
3

Yes, I am using rclone copy. When I use just a windows copy/paste from my HDD to rclone it seems to buffer just fine from watching the disk read and memory usage. Thanks for thinking about this and sorry if I am not completely used to all in info. Also side note, I am also stuck on 4 transfers even when I use --transfers 8 on both mount and/or rclone copy. From what I can tell all other flags are still adjustable, so not sure if I somehow things are related.

My mount:

mount --config "C:\..." --allow-other --dir-cache-time 1000h --vfs-read-chunk-size 32M  --drive-chunk-size 32M --vfs-cache-mode off  --buffer-size 64M --vfs-write-wait 10s --vfs-read-wait 60ms gcrypt2: A:

My copy command:

rclone copy --size-only --progress --fast-list --bwlimit=8.5M --log-file "Log Location" "Source" "Destination"
4

Is the Destination in A: or is it an rclone remote gcrypt2:?

How about the Source?

1 Like
6

The destination is an encrypted rclone remote named gcrypt2 mounted to drive A:
Hopefully this is the right answer to your question, I am definitely new to rclone
I still seem to have no issues with the buffer when I don't use bwlimit
I just tried my same configuration and commands on another computer and had the same results of no buffer with bwlimit and normal buffer without bwlimit. I did try different orders for the same flags but no change.
I wish I had my old log file so I could see what the change was to the mount

7

Can paste the command line? I'm still not sure whether the destination is gcrypt2: or A: - thanks!

1 Like
8

Sorry! I wasn't thinking that you might want to check that. And no issues with the actual copied files at all, they are completely accessible from the mounted drive.

rclone copy --size-only --progress --fast-list --bwlimit=8.5M --log-file "C:\Program Files\rclone\logs\log.txt" "G:\New Folder" "A:\New Folder"

Edit: Fixed log location

9

No worries!

Can you try the exact same command but use it to copy a directory to your local disk?

rclone copy --size-only --progress --fast-list --bwlimit=8.5M --log-file "C:\Program Files\rclone\logs\log.txt" "G:\New Folder" "G:\New Folder Copy"

Does that exhibit the "constantly churning/seeking" behaviour too?

Are you copying very big files?

Can you try your original command with --multi-thread-streams 0 and see if that helps with the churning.

10

Yeah it seems to still not utilize the buffer even with the new copy command to the same local HDD. --multi-thread-streams 0 seems to make the HDD not as frantically active but I really just want to be able to use the buffer instead of constantly seeking the drive and I just can't replicate what changed from when it used to use the buffer and I am still regretting not having my original mount and command.

I am copying large media files, but they were successfully able to use the buffer beforehand.

Do you have any suggestions of mounts and rclone copy commands that successfully use both the buffer and bwlimit that I could try to maybe narrow down the issue? With my own trial and error eliminating flags I could not use both buffer and bwlimit regardless of how simple the mount and command were.

11

So watching the memory usage change from initiating the copy it seems like the buffer is being used is still being used (~490-530MB), but before it would get to the plateau of usage at the rate of my HDD speed rather than now being something much closer to my bwlimit. My internet is faster than my bwlimit, but I understand there could be a variety of factors why the upload is unable to keep up with the bwlimit and the buffer is still being filled. The copied Bytes creeps up at the bwlimit rate, when before it was at my HDD speed before filling up the --buffer limit until it moved to the next segment to be copied and again increases at my HDD speed rate.

12

Did you change your --vfs-cache-mode on the mount? That would explain it.

Did you try copying directly to the remote, not via the mount?

13

No, I had --vfs-cache-mode off on the original and current. When the buffer was working I felt like there was little reason for me to use --vfs-cache-mode.

Copying directly to the remote seems to use the buffer just fine. The transfer peaks for the first few seconds looking like it is filling up the buffer at 100MB/s or whatever. Then dies to zero for a while. Then peaks again. Like the buffer is filling, somewhat emptying, then filling again.

14

When copying to the mount, exactly how the buffering works will depend somewhat on your OS.

Try just having the --bwlimit on the rclone mount command and not on the rclone copy command - I think that should do the right things in terms of buffering.

15

Still no success! I don't know what it up. So the buffer is still filling until full, but still at the --bwlimit speed. Haha, thanks for your continued efforts! I don't get it, I tried it with a computer that never had rclone before and still no difference. It is just weird to me that it used to work and whatever I changed before somehow broke it for me.

I tried adjusting --vfs-read-chunk-size 2G and --drive-chunk-size 128M. Neither changed the buffer filling. I did not expect the --vfs-read-chunk-size-limit to do anything though from my understanding of it being for downloads form my remote rather than a copy/upload command. But still trying.

16

Just another thing, I just tried using --vfs-cache-mode minimal and the rclone copy does not upload to remote nor use the buffer until the entire file copied to the cache. Copying to the cache occurs at the speed of --bwlimit. I don't think I experienced this before. I'm just watching the system processes and there is no network activity and rclone memory usage stays at about 18MB until the entire file is in the cache folder and then the buffer/memory is used.

Edit: Added that cache fills at --bwlimit speed.

17

Hmm, puzzling!

I'm planning on changing the bandwidth limiting to move it to the edges which will do exactly what you want and also be more accurate.

18

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.