Rclone user metadata and azureblob

Help and Support
1

What is the problem you are having with rclone?

When i upload using rclone to azure blob storage, user metadata doesn't seem to get uploaded. I tried copying between two local folders using rclone, and then, when using --metadata flag on rclone, metadata gets preserved, so i think the problem is specific to azure

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.1.0-1008-oem (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Azure blob storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

touch file_with_metadata
setfattr -n user.iwashere -v "frank" file_with_metadata 
rclone copy --metadata file_with_metadata myazure:determined/
rclone copy --metadata  myazure:determined/file_with_metadata /tmp/
getfattr /tmp/file_with_metadata

The rclone config contents with secrets removed.

[myazure]
type = azureblob
account = reunstex001
key = XXXX

A log from the command with the -vv flag

<7>DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "-vv" "copy" "--metadata" "file_with_metadata" "myazure:determined/"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "file_with_metadata"
<7>DEBUG : Using config file from "/home/kervel/.config/rclone/rclone.conf"
<7>DEBUG : fs cache: adding new entry for parent of "file_with_metadata", "/home/kervel"
<7>DEBUG : Creating backend with remote "myazure:determined/"
<7>DEBUG : fs cache: renaming cache item "myazure:determined/" to be canonical "myazure:determined"
<7>DEBUG : file_with_metadata: Need to transfer - File not found at Destination
<7>DEBUG : o.meta = map[]
<7>DEBUG : o.meta = map[mtime:2023-08-03T11:38:57.924036184+02:00]
<7>DEBUG : file_with_metadata: md5 = d41d8cd98f00b204e9800998ecf8427e OK
<6>INFO  : file_with_metadata: Copied (new)
<6>INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.2s
2

I think this is correct. azure blob does not support extended attributes (at least not rclone implementation)

3

Hello,

In azure blob it seems rclone still sets one "user" metadata attribute, mtime (also apparent from the debug output)

image
image753×272 9 KB

this is apart from the system metadata on the file (like content-type) that is also set.

greetings,
Frank

4

Here you can see metadata support for all remotes.

Actually AFAIK extended attributes only work for local remotes.

It is on the list... for years. Waiting for somebody to implement it.

5

Azure blob doesn't yet support metadata. It could though.

You could have a go at this or maybe your company would like to sponsor us to implement it. If so check out https://rclone.com and contact us from there.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.