Rclone rcat azure blob container sas token => 403 error

Suspected Bug
1

What is the problem you are having with rclone?

Copy one file (streaming via rcat) to azure blob container, autentication via sas token (blob container level)
Same command works when using storageaccount-level sas token.
A normal "rclone copy" command also works with container-level sas toke, so the issue seems to be specific to rcat.

What is your rclone version (output from rclone version)

1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

debian buster (docker container)

Which cloud storage system are you using? (eg Google Drive)

Azure Blob

The command you were trying to run (eg rclone copy /tmp remote:tmp)

cat rclone-current-linux-amd64.zip | rclone rcat bck:<container>/rclone-current-linux-amd64.zip

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020/05/10 21:04:25 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "rcat" "bck:/mysqlha-backup/rclone-current-linux-amd64.zip" "-vv"]
2020/05/10 21:04:25 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2020/05/10 21:04:25 DEBUG : Azure container mysqlha-backup: Target remote doesn't support streaming uploads, creating temporary local FS to spool file
2020/05/10 21:04:25 DEBUG : rclone-current-linux-amd64.zip: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/05/10 21:04:25 ERROR : rclone-current-linux-amd64.zip: Failed to copy: -> github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/runner/work/rclone/src/github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=This request is not authorized to perform this operation.
RequestId:2377b81b-701e-0007-6d0e-272d34000000
Time:2020-05-10T21:04:25.9503108Z, Details:
   Code: AuthorizationFailure
   PUT https://REDACTED.blob.core.windows.net/mysqlha-backup?restype=container&se=2030-12-30t22%3A00%3A00z&sig=REDACTED&sp=racwdl&sr=c&st=2020-05-10t19%3A58%3A32z&sv=2019-10-10&timeout=31536001
   User-Agent: [rclone/v1.51.0]
   X-Ms-Client-Request-Id: [812e60a6-babd-4461-51e9-68d960ac88f9]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Content-Length: [246]
   Content-Type: [application/xml]
   Date: [Sun, 10 May 2020 21:04:25 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthorizationFailure]
   X-Ms-Request-Id: [2377b81b-701e-0007-6d0e-272d34000000]
   X-Ms-Version: [2018-11-09]
2

Anything jump out @ncw? I'm definitely not a S3 guy :slight_smile:

3

It appears to be trying to create the container which isn't allowed with a container based sas.

This is an easy fix though - we know we have a container already if we are using a SAS URL limited to a container so no need to create the container.

I've merged a fix to the latest beta - can you give that a go?

https://beta.rclone.org/v1.51.0-312-gd342f9f9-beta/ (uploaded in 15-30 mins)

4

It works now! Thank you so much!

Just another question: I see from the logs:
"Azure container REDACTED: Target remote doesn't support streaming uploads, creating temporary local FS to spool file".
Actually Azure blobs should support streaming uploads (some kind of streaming, at least), is it a feature that will be implemented in the future?

Roberto

5

Glad it is working!

Azureblob needs to know the size of the thing we are streaming in advance is what rclone is saying here.

I had a brief glance at the code and it doesn't look like it actually uses the size, so maybe this is possible after all...

Give this a go

https://github.com/rclone/rclone/pull/new/fix-azureblob-stream (uploaded in 15-30 mins)

6

Unfortunately this does not seem to work:
cat ./rclone | rclone rcat -vv dest:REDACTED/rclone
Segmentation fault (core dumped)

Roberto

7

Strange! It works for me...

Can you try this instead please?

https://beta.rclone.org/branch/v1.51.0-321-ge8155d63-fix-azureblob-stream-beta/ (uploaded in 15-30 mins)

If it crashes again it should show a backtrace - can you send that please?

8

This version actually works...I do not know if I used an incorrect version yesterday,sorry.

By the way, that's great, thanks!
Roberto

9

Thanks for testing!

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52

1 Like
10

That's great, many thanks.
Roberto

1 Like
11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.