Rclone copy seems to create quite a number of API calls in azure

What is the problem you are having with rclone?

I have a question regarding the amount of WRITE operations generated for azure blob as destination that should be expected.

I was copying the below amount of data:
Total objects: 210304
Total size: 3.954 TBytes (4346937048495 Bytes)

Using Azure blob (hot) as destination and immediately setting the archive-flag with rclone

after copy I did a destination check as well to confirm:
time rclone -vv size ""
2021/03/05 09:17:25 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "-vv" "size" ""]
2021/03/05 09:17:25 DEBUG : Using config file from ""
2021/03/05 09:17:25 DEBUG : Creating backend with remote ""
Total objects: 210281
Total size: 3.954 TBytes (4346933420158 Bytes)
2021/03/05 09:19:21 DEBUG : 4 go routines active

( a few files failed so number does not match 100%)

Above worked fine, but, and here is my concern / surprise, uploading these just over 200k objects as described above resulted in ~1.4 million "Tiered Block Blob - Hot LRS - Write Operations" on my azure charge, ie on average every object uploaded seems to have triggered almost 7 hot write operations in azure. any ideas why, and if this can be optimized in any way?

as a reference , the "Tiered Block Blob - Archive LRS - Write Operations" I was charged matches the ~200k quite nicely..

The azure figures comes from my actual monthly charges as published by Microsoft in my portal.

Comments or thoughts?

What is your rclone version (output from rclone version)

rclone v1.54.0

  • os/arch: linux/amd64
  • go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian - bullseye/sid

Which cloud storage system are you using? (eg Google Drive)

Azure Blob (Hot->Archive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone --progress --max-depth -1 --checkers 28 --dropbox-impersonate <MASKED> --config <MASKED> --transfers=28 --log-file=$LOGPATH/rclone-<MASKED>-$date.log --log-level INFO

The rclone config contents with secrets removed.

type = azureblob
account = <MASKED>
key = <MASKED>
access_tier = Archive
archive_tier_delete = true

A log from the command with the -vv flag

did not use -vv unfortunately

You'd want to check these out to increase chunk sizes I'd imagine:

1 Like

Ah, but of course... any file smaller than 4Mb will be uploaded in chunks, and each chunk will then consist of a separate write operation in azure...
Assuming an average size of 20M, each upload would be 5 chunks, and when if there is also one put or append before the set-tier operation. Yes that well explains the numbers I see.
I will tweak my --azureblob-chunk-size in my scripts before my next runs.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.