Access to Azure BLOB storage only works once every few minutes

What is the problem you are having with rclone?

We currently have this problem only (and it only started recently, probably after an update of rclone or other packages via apt) on Ubuntu Server 22.04.3 LTS. Doing the same with rclone on Windows, using the same Azure BLOB Storage, same SAS Token, same Internet connection / network etc. works fine.
The problem is, that rclone connects only once every few minutes properly to an Azure BLOB storage, then we see Authorization Failures until we wait a few minutes more for another attempt. At the same time from a Windows VM using rclone (1.64.2 or an older version does not matter) access works just fine.
Although the Logs posted here are from rclone 1.53.3, we also tried it with the latest 1.64.2 with the same results.
So if we run the ls command posted below once, it shows the contents of the BLOB storage. Run it a 2nd time and you see the error message posted below. Wait a few minutes and try it again = it works.
We were using this setup for a few years now with cron jobs for automatic backups, but realized today that it just stopped working recently. What gets changed frequently on the server is that it is kept to the latest versions with apt update | apt upgrade.

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV

  • os/arch: linux/amd64
  • go version: go1.18.1

Which cloud storage system are you using? (eg Google Drive)

Microsoft Azure BLOB Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls rclonerepo:container

The rclone config contents with secrets removed.

[rclonerepo]
type = azureblob
sas_url = https://rclonerepo.blob.core.windows.net/?sv=2020-02-10&ss=b&srt=sco&sp=rwdlactf&se=2031-06-29T14:06:27Z&st=2021-06-29T06:06:27Z&spr=https&sig=REDACTED
endpoint = rclonerepo.blob.core.windows.net

A log from the command with the -vv flag

2023/11/16 12:56:24 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "ls" "rclonerepo:container" "-vv"]
2023/11/16 12:56:24 DEBUG : Using config file from "/REDACTED/.config/rclone/rclone.conf"
2023/11/16 12:56:24 DEBUG : Creating backend with remote "rclonerepo:container"
2023/11/16 12:56:24 DEBUG : 4 go routines active
2023/11/16 12:56:24 Failed to ls: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=This request is not authorized to perform this operation.
RequestId:REDACTED
Time:2023-11-16T11:56:24.7015534Z, Details:
   Code: AuthorizationFailure
   GET https://rclonerepo.blob.core.windows.net/container?comp=list&delimiter=&include=metadata&maxresults=5000&restype=container&se=2031-06-29t14%3A06%3A27z&sig=REDACTED&sp=rwdlactf&spr=https&srt=sco&ss=b&st=2021-06-29t06%3A06%3A27z&sv=2020-02-10&timeout=31536001
   User-Agent: [rclone/v1.53.3-DEV]
   X-Ms-Client-Request-Id: [f1b1d692-00fc-4e1a-7772-5aa9ce2b990b]
   X-Ms-Version: [2019-02-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Content-Length: [246]
   Content-Type: [application/xml]
   Date: [Thu, 16 Nov 2023 11:56:24 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [REDACTED]
   X-Ms-Error-Code: [AuthorizationFailure]
   X-Ms-Request-Id: [REDACTED]

This is years old rclone version - there were 25+ releases since then.

Whatever problem you are facing I do not think many people are interested in investigating it. It is vintage computing category...

You can install the latest version easy using your preferred method:

But uninstall the apt version first as clearly your OS maintainers stopped updating it years ago.

Thanks for your reply. But as I wrote, I have the same behaviour with 1.64.2.

Writing down all of this after 5 hours of finding the root cause might have helped though. I am currently suspecting a DNS issue, but I have to investigate further and see if I am right. I will update this post with my findings, especially if it turns out that I can fix it outside of rclone.

Ohh sorry. When I see such old version used I do not read the rest:) as so many things have changed and were fixed in the meantime...

No worries! And it turns out that we are fighting some really strange and hard to track down DNS issue with conditional forwarders for azure private links. So rclone gets the correct IP the first time and in consecutive requests it gets the public IP of the BLOB storage (which is only accessible via a private link). After a few minutes, the private IP pops up once again and the cycle repeats. I can now reproduce this also on Windows VMs utilizing the same DNS servers. This has nothing to do with rclone at all, so I will try to close this one, so it gets archived.
Since using Azure BLOB storage with private links if you store sensitive data could be pretty common, maybe someone stumbles upon this when facing the same problem and saves a few hours troubleshooting :wink:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.