Backblaze B2 mount with Rclone generating Class C transcactions

What is the problem you are having with rclone?

I have 37 million files in a bucket, mounted with RClone, so far it has cost me $584 to upload, I need to know what i am doing in the mountbucket that causes so many class C transactions

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-186-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.4
  • go/linking: static
  • go/tags: none

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
-->

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --vfs-cache-mode writes --transfers 200 --no-modtime --allow-other --vfs-read-chunk-size off b2-2:fsassets-vw /mnts/b2-ass3 --inplace --daemon

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[b2-2]
type = b2
account = xxx
key = xxx
endpoint = 
download_auth_duration = 36m

A log from the command that you were trying to run with the -vv flag

2024/07/01 00:50:24 DEBUG : rclone: Version "v1.65.0" starting with parameters ["/usr/bin/rclone" "mount" "--transfers" "30" "--no-modtime" "--allow-other" "--vfs-read-chunk-size" "off" "b2-2:fsassets-vw" "/mnts/b2-ass3" "--inplace" "--allow-non-empty" "-vv"]
2024/07/01 00:50:24 DEBUG : Creating backend with remote "b2-2:fsassets-vw"
2024/07/01 00:50:24 DEBUG : Using config file from "/home/cliffh/.config/rclone/rclone.conf"
2024/07/01 00:50:24 INFO  : B2 bucket fsassets-vw: poll-interval is not supported by this remote
2024/07/01 00:50:24 DEBUG : B2 bucket fsassets-vw: Mounting on "/mnts/b2-ass3"
2024/07/01 00:50:24 DEBUG : : Root: 
2024/07/01 00:50:24 DEBUG : : >Root: node=/, err=<nil>
2024/07/01 00:50:24 DEBUG : /: Lookup: name=".Trash"
2024/07/01 00:50:53 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2024/07/01 00:50:53 DEBUG : /: Lookup: name=".Trash-1000"
2024/07/01 00:50:53 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory

welcome to the forum,

no idea what programs are accessing the mount and how they access it.

tho, the answer would be in the debug log.
can use --dump=headers to see each api call.

one reason could be listing all those files over and over again
from the docs "The b2_list_file_names request will be sent once for every 1k files in the remote path"
might increase --dir-cache-time

and what is the reason for --vfs-read-chunk-size off ?

can check out my summary of the two vfs caches.
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34

i have over 30million files so list files names uses 30,000 class C transactions
was reccommmened as most if not all files are small and this setting gives better performance.
What am I doing that is generating the b2_list_file_names operation?
all I am doing is creating a mount point?
and was using rclone copy to upload, i have had to stop as the current bill from Backblaze is existentially threatening.

rclone will list the files.
take a few minutes, read the debug log.

can check the debug log.

the vfs dir cache expires after five minutes, i suggested to increase --dir-cache-time
and might try test --vfs-refresh

fwiw, might choose a provider that does not charge for api calls.

I had no idea i was invoking an api just by creating a mountpoint and adding files to it

tl;dr for many tasks, rclone copy|sync is a better choice, more tweaks to reduce api calls.

that is a huge amount of money, what is the total size of all the files?
and can you post the details from the b2 website, breakdown of the api calls and costs?

just starting the mount should not incur an api call.
rclone is pretty good about not accessing the remote until asked to by a program or the OS.

that would incure api calls.
--dump=headers will capture the api calls.

Suggestion: mounting the remote drive is a convenience and also very useful for interactive access. Why not copy all your files one time with the rclone copy command, then mount the drive after transfering the bulk of the data, and use the mounted drive thereafter for small changes?

the project has two phases

  1. upload the assets approx 1Tb .gz files
  2. update and access

phase 1 has cost $584 so far and is not yet complete
phase 2 must use a mount and accessing the mount will incurr class C transactions confirmed by backblaze

B2 Cloud Storage Fees (Estimate)
07/05/2024

$598.59
Estimate

For dates 05/05/2024 - 07/05/2024

  • Download Bandwidth($0.00)
  • Class B Transactions($0.98)
  • Class C Transactions($594.04)
    • api authorize account called: 457($0.00)
    • s3 list objects called: 3128913($12.52)
    • api create key called: 2($0.00)
    • api list unfinished large files called: 2($0.00)
    • api copy file called: 91474($0.37)
    • api more file versions: 1584($0.01)
    • api list buckets called: 324($0.00)
    • s3 get bucket encryption called: 3($0.00)
    • s3 list buckets called: 1($0.00)
    • api list file versions called: 361($0.00)
    • api list keys called: 16($0.00)
    • api list file names called: 145338572($581.35)
    • api create bucket called: 2($0.00)
    • s3 get location called: 2($0.00)
    • Free level use$0.21
  • Storage($3.57)
  • Total: $598.59

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.