Rclone mount bandwidth usage?

Help and Support
1

I apologize if this question is covered in the documentation, but I wasn't able to find anything that covered it.

If I mount a B2 bucket, and am writing a file in a "slowly growing" manner, what does the bandwidth usage look like? As an example, let's say that I am uploading a file via FTP to the mounted drive, and the FTP server writes out 1MB, then expands the file to 2MB, and on and on up to 100MB.
Does rclone need to only send 100MB? Or does it send 1 + 2 + 3 + ... + 100? Or something in between?

Similarly, on the reading end, if a user is doing random reads from a file stored on B2, does rclone / B2 support only fetching the small pieces of the file that the user is requiring?

Thanks in advance!

Cheers,
Jon

2

hello,

hard to give a good answer, as you did not answer any of the questions...
using -v of -vv, rclone's output will let know what it is doing.

the documentation for rclone mount can be found here

3

I understand that there is a requested format for questions in this forum, but my question doesn't really fit the type where output of '-v' is helpful. I read the documentation that you linked to, and unless I'm missing something, it doesn't cover my question.

4

i guess i am not understanding your use-use, are you using b2 or ftp?

5

I want to use b2, and am asking how rclone mount works when a program is writing a file out in stages. Perhaps I confused things by using FTP as that example program.
If some program writes out 1MB, then expands the file to 2MB, etc, what does the bandwidth between rclone and b2 look like? Does it send 1MB, then 2MB over the internet?

6

it depends on the rclone version, rclone mount command and flags.

once rclone determines a file is not in use for a period of time, it will upload the entire file to cloud.

let's say you have a text file in b2
the time period window is 30 seconds --vfs-write-back=30s

within that 30 second time window, you edit and save that text file several times.
rclone does not re-upload the entire file per each change/save.
for each file change, rclone resets the time window back to 30 seconds.
once that time period expires and rclone notices that the file has not changed, only then does rclone upload the file from the local vfs cache to b2.

this can be seen using a log.
here i open a text file and edit/save it 3 times within 30 seconds.
for each of the thee edit/save, rclone resets the time window to 30 seconds.
once 30 seconds go by and rclone knows the file has not changed, then rclone uploads it one time

2021/10/13 14:21:15 INFO  : file.txt: vfs cache: queuing for upload in 30s
2021/10/13 14:21:20 INFO  : file.txt: vfs cache: queuing for upload in 30s
2021/10/13 14:21:24 INFO  : file.txt: vfs cache: queuing for upload in 30s
2021/10/13 14:21:55 INFO  : file.txt: Copied (replaced existing)
2021/10/13 14:21:55 INFO  : file.txt: vfs cache: upload succeeded try #1
7

If you are not using --vfs-cache-mode then the file will be uploaded straight to b2, so 100M of usage. If you are using --vfs-cache-mode then it depends on quite a few things!

Firstly if the file is open then it won't be uploaded so the file needs to be closed in order to be uploaded.

Secondly the file won't be uploaded for --vfs-write-back interval once it is closed. So the worst case would be open file, write something, close, wait for more than --vfs-write-back - that will upload the whole file so far. However if the file is being uploaded and it is opened again, then the upload will be cancelled.

So its complicated!

However as @asdffdsa suggested you can see the INFO lines in the log to see exactly when a file was uploaded and use -vv to get more insight with the DEBUG logs.

8

That's super helpful. Thank you!

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.