Rclone azure blob storage authorization failure

What is the problem you are having with rclone?

Authorization failure using rclone with azure blob storage. I experimented with both account and container level SAS urls, as well as primary access key of the storage account, in all cases I get an authorization failure. Interestingly there are no issues when the storage account is in the same subscription as where rclone is deployed. However I run into this problem when the storage account is in another subscription. Rclone is used an init container in Kubernetes.

Run the command 'rclone version' and share the full output of the command.

v1.56.2.

Which cloud storage system are you using? (eg Google Drive)

Azure

2022/05/19 16:05:39 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "copy" "-vv" "az:dbrickseldonwsstocont/mlflow-models/mnist4325" "/mnt/models"]
2022/05/19 16:05:39 DEBUG : Creating backend with remote "az:dbrickseldonwsstocont/mlflow-models/mnist4325"
2022/05/19 16:05:39 DEBUG : Setting type="azureblob" for "az" from environment variable RCLONE_CONFIG_AZ_TYPE
2022/05/19 16:05:39 DEBUG : Setting sas_url="https://dbrickseldonwsstoacc.blob.core.windows.net?se=2024-01-01&sp=rwdxlacupfti&sv=2021-04-10&ss=qbft&srt=soc&sig=REDACTED" for "az" from environment variable RCLONE_CONFIG_AZ_SAS_URL
2022/05/19 16:05:39 DEBUG : az: detected overridden config - adding "{SFrlp}" suffix to name
2022/05/19 16:05:39 NOTICE: Config file "/.rclone.conf" not found - using defaults
2022/05/19 16:05:39 DEBUG : Setting sas_url="https://dbrickseldonwsstoacc.blob.core.windows.net?se=2024-01-01&sp=rwdxlacupfti&sv=2021-04-10&ss=qbft&srt=soc&sig=REDACTED" for "az" from environment variable RCLONE_CONFIG_AZ_SAS_URL
2022/05/19 16:05:39 Failed to create file system for "az:dbrickseldonwsstocont/mlflow-models/mnist4325": -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=403 This request is not authorized to perform this operation., Details: (none)
   HEAD https://dbrickseldonwsstoacc.blob.core.windows.net/dbrickseldonwsstocont/mlflow-models/mnist4325?se=2024-01-01&sig=REDACTED&sp=rwdxlacupfti&srt=soc&ss=qbft&sv=2021-04-10&timeout=31536001
   User-Agent: [rclone/v1.56.2]
   X-Ms-Client-Request-Id: [32132c3b-989a-424d-4fd1-2bb077b3db2b]
   X-Ms-Version: [2019-12-12]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Date: [Thu, 19 May 2022 16:05:38 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [32132c3b-989a-424d-4fd1-2bb077b3db2b]
   X-Ms-Error-Code: [AuthorizationFailure]
   X-Ms-Request-Id: [22ac02cb-301e-001a-5e9a-6b23d2000000]


hello and welcome to the forum,

please update to latest stable, v1.58.1 and test again

Did not work on 1.58.1

2022/05/20 04:56:23 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "copy" "-vv" "az:dbrickseldonwsstocont/mlflow-models/mnist4325" "/mnt/models"]
2022/05/20 04:56:23 DEBUG : Creating backend with remote "az:dbrickseldonwsstocont/mlflow-models/mnist4325"
2022/05/20 04:56:23 DEBUG : Setting type="azureblob" for "az" from environment variable RCLONE_CONFIG_AZ_TYPE
2022/05/20 04:56:23 DEBUG : Setting sas_url="https://dbrickseldonwsstoacc.blob.core.windows.net?se=2024-01-01&sp=rwdxlacupfti&sv=2021-04-10&ss=btfq&srt=osc&sig=REDACTED" for "az" from environment variable RCLONE_CONFIG_AZ_SAS_URL
2022/05/20 04:56:23 DEBUG : az: detected overridden config - adding "{tBLJy}" suffix to name
2022/05/20 04:56:23 NOTICE: Config file "/.rclone.conf" not found - using defaults
2022/05/20 04:56:23 DEBUG : Setting sas_url="https://dbrickseldonwsstoacc.blob.core.windows.net?se=2024-01-01&sp=rwdxlacupfti&sv=2021-04-10&ss=btfq&srt=osc&sig=REDACTED" for "az" from environment variable RCLONE_CONFIG_AZ_SAS_URL
2022/05/20 04:56:23 Failed to create file system for "az:dbrickseldonwsstocont/mlflow-models/mnist4325": -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.14.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=403 This request is not authorized to perform this operation., Details: (none)
   HEAD https://dbrickseldonwsstoacc.blob.core.windows.net/dbrickseldonwsstocont/mlflow-models/mnist4325?se=2024-01-01&sig=REDACTED&sp=rwdxlacupfti&srt=osc&ss=btfq&sv=2021-04-10&timeout=31536001
   User-Agent: [rclone/v1.58.1]
   X-Ms-Client-Request-Id: [c659dc14-92f8-4840-71dc-1cfa94859d1b]
   X-Ms-Version: [2020-04-08]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Date: [Fri, 20 May 2022 04:56:22 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [c659dc14-92f8-4840-71dc-1cfa94859d1b]
   X-Ms-Error-Code: [AuthorizationFailure]
   X-Ms-Request-Id: [b5c319f4-d01e-003d-3205-6c3416000000]

What does this mean "in another subscription"?

This sounds like a problem setting up the Auth on Azure rather than an rclone problem but I'm not 100% sure.

If you use a container level SAS URL make sure you name the container in the rclone command line - it won't work without it.

e.g.

rclone ls azureblob:container

You can also list the single container from the root. This will only show the container specified by the SAS URL.

$ rclone lsd azureblob:
container/

Note that you can't see or access any other containers - this will fail

When we use rclone in the same azure subscription as where the storage account is, there are no issues, however if the storage account is in another subscription, then it runs into authorization failures. We use an account level sas url not container.

This must surely be some auth problem on your side musn't it?

I'm not an azure expert so I'm not sure what the problem is, but if it works with one subscription and not another then this must be something wrong with the auth I would have thought?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.