Metadata from linux local -> s3: Failed to copy: failed to read metadata from source object: function not implemented

What is the problem you are having with rclone?

I am very excited to use the new metadata features, but when attempting to sync data w/ metadata from my local, centos7 box to aws s3, i get the below error. I saw this post from a few weeks ago where the poster seemed to be doing exactly what I want to do and got a lot farther than i did, and I can't figure out why my attempt would be any different than his. I went as far as trying @ncw fix-xattr-notsup branch to see if that helps but i get the same error. basically no matter what I do:

failed to read metadata from source object: function not implemented

the dir itself I'm testing locally:

[zack.ramjan@submit admin]$ ls -laR testmd
testmd:
total 18
drwxrwx---+  2 zack.ramjan hpcusers               4096 Sep 26 12:23 .
drwxrwx---+ 62 zack.ramjan it infrastructure team 8192 Sep 26 13:46 ..
-rw-rwx---+  1 zack.ramjan hpcadmins                 6 Sep 26 12:22 test.withacls

Run the command 'rclone version' and share the full output of the command.

I've tested with the following 3 versions

[zack.ramjan@submit admin]$  ~/software/rclone/rclone-v1.59.2-linux-amd64/rclone version
rclone v1.59.2
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.15.2.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: none

[zack.ramjan@submit admin]$  ~/software/rclone/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone version
rclone v1.60.0-beta.6462.7e547822d
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.15.2.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.1
- go/linking: static
- go/tags: none

zack.ramjan@submit admin]$  ~/software/rclone/rclone-v1.60.0-beta.6410.fdfa07be1.fix-xattr-notsup-linux-amd64/rclone version
rclone v1.60.0-beta.6410.fdfa07be1.fix-xattr-notsup
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.15.2.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3, although it appears local is the issue

The command you were trying to run (eg rclone copy /tmp remote:tmp)

 ~/software/rclone/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone sync -vvvv --metadata testmd  aws:motuz-hot/testmd/```

The rclone config contents with secrets removed.

[aws]
type = s3
provider = AWS
access_key_id = *
secret_access_key = *
region = us-east-2
location_constraint = us-east-2
acl = private

A log from the command with the -vv flag

[zack.ramjan@submit admin]$  ~/software/rclone/rclone-v1.59.2-linux-amd64/rclone sync -vvvv --metadata testmd/test.withacls  aws:motuz-hot/testmd/
2022/09/26 13:48:22 DEBUG : rclone: Version "v1.59.2" starting with parameters ["/home/zack.ramjan/software/rclone/rclone-v1.59.2-linux-amd64/rclone" "sync" "-vvvv" "--metadata" "testmd/test.withacls" "aws:motuz-hot/testmd/"]
2022/09/26 13:48:22 DEBUG : Creating backend with remote "testmd/test.withacls"
2022/09/26 13:48:22 DEBUG : Using config file from "/home/zack.ramjan/.config/rclone/rclone.conf"
2022/09/26 13:48:22 DEBUG : fs cache: adding new entry for parent of "testmd/test.withacls", "/varidata/research/admin/testmd"
2022/09/26 13:48:22 DEBUG : Creating backend with remote "aws:motuz-hot/testmd/"
2022/09/26 13:48:22 DEBUG : fs cache: renaming cache item "aws:motuz-hot/testmd/" to be canonical "aws:motuz-hot/testmd"
2022/09/26 13:48:22 DEBUG : test.withacls: Need to transfer - File not found at Destination
2022/09/26 13:48:22 ERROR : test.withacls: Failed to copy: failed to read metadata from source object: function not implemented
2022/09/26 13:48:22 ERROR : Attempt 1/3 failed with 1 errors and: failed to read metadata from source object: function not implemented
2022/09/26 13:48:22 DEBUG : test.withacls: Need to transfer - File not found at Destination
2022/09/26 13:48:22 ERROR : test.withacls: Failed to copy: failed to read metadata from source object: function not implemented
2022/09/26 13:48:22 ERROR : Attempt 2/3 failed with 1 errors and: failed to read metadata from source object: function not implemented
2022/09/26 13:48:22 DEBUG : test.withacls: Need to transfer - File not found at Destination
2022/09/26 13:48:22 ERROR : test.withacls: Failed to copy: failed to read metadata from source object: function not implemented
2022/09/26 13:48:22 ERROR : Attempt 3/3 failed with 1 errors and: failed to read metadata from source object: function not implemented
2022/09/26 13:48:22 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.3s

2022/09/26 13:48:22 DEBUG : 5 go routines active
2022/09/26 13:48:22 Failed to sync: failed to read metadata from source object: function not implemented```

Just an update:

I tested with the same rclone binary on the same mounted FS (we are using gpfs), using an RHEL8.4 box and it worked correctly.

So it seems RHEL7/Centos7 will not detect the local metadata capability for linux...

This kind of implies that the local file system does not support reading xattrs

What does

attr -l file

do on the local system?

I'm guessing that RHEL7/Centos 7 isn't advertising xattr support for gpfs but I know very little about either so that is a guess only!

I tested on both systems and the output is the same

[zack.ramjan@submit admin]$ cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[zack.ramjan@submit admin]$ attr -l testmd
[zack.ramjan@submit admin]$
[zack.ramjan@dtn1 zack.ramjan]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)
[zack.ramjan@dtn1 zack.ramjan]$ attr -l test2
[zack.ramjan@dtn1 zack.ramjan]$

Its almost like centos7 is not being detected as linux.

Can you try this on the file too - rclone will print the attrs if available

rclone lsf -F pM /path/to/file

Also what do these commands do on both systems?

echo hello | rclone rcat -M --metadata-set greeting=hello file.txt
attr -l file.txt 
rclone lsf -F pM file.txt

on centos7

[zack.ramjan@submit zack.ramjan]$ ~/software/rclone/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone  lsf -F pM test2/
2022/09/29 12:50:08 ERROR : test: Failed to read metadata: function not implemented
test;{}
[zack.ramjan@submit zack.ramjan]$ echo hello | ~/software/rclone/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone rcat -M --metadata-set greeting=hello file2.txt
[zack.ramjan@submit zack.ramjan]$ attr -l file2.txt
Attribute "greeting" has a 5 byte value for file2.txt
[zack.ramjan@submit zack.ramjan]$ ~/software/rclone/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone  lsf -F pM file2.txt
2022/09/29 12:58:32 ERROR : file2.txt: Failed to read metadata: function not implemented
file2.txt;{}

on centos8:

[root@dtn1 zack.ramjan]# /tmp/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone lsf -F pM test2/
test;{"atime":"2022-09-26T16:46:26.512094557-04:00","btime":"1969-12-31T19:00:00-05:00","gid":"1351814256","mode":"100644","mtime":"2022-09-26T15:53:12.849183-04:00","uid":"1351813916"}
[zack.ramjan@dtn1 zack.ramjan]$ echo hello | /tmp/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone  rcat -M --metadata-set greeting=hello file.txt
[zack.ramjan@dtn1 zack.ramjan]$ attr -l file.txt
Attribute "greeting" has a 5 byte value for file.txt
[zack.ramjan@dtn1 zack.ramjan]$ /tmp/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone   lsf -F pM file.txt
file.txt;{"atime":"2022-09-29T12:56:47.340127-04:00","btime":"1969-12-31T19:00:00-05:00","gid":"1351800513","greeting":"hello","mode":"100644","mtime":"2022-09-29T12:56:47.34185-04:00","uid":"1351813916"}

The filesystem is mounted on both centos7 and centos8, so when I read the file2.txt created on centos7 using centos8 i see the metadata, so its getting set via rclone, just not able to read.

[zack.ramjan@dtn1 zack.ramjan]$ /tmp/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone   lsf -F pM file2.txt
file2.txt;{"atime":"2022-09-29T12:57:24.799037-04:00","btime":"1969-12-31T19:00:00-05:00","gid":"1351800513","greeting":"hello","mode":"100644","mtime":"2022-09-29T12:57:24.79979-04:00","uid":"1351813916"}

OK, so the OS supports xattrs as does the filing system, rclone can set them on both platforms but only read them on one.... Strange!

can you try this on centos7 (which fails with function not implemented) and post (or attach) the result?

strace -f ~/software/rclone/rclone-v1.60.0-beta.6462.7e547822d-linux-amd64/rclone  lsf -F pM test2/

That will show us which syscall is failing and what rclone passed to it.

Rclone uses this library for read/writing the xattrs: Issues · pkg/xattr · GitHub

I've attached the strace output from both the working (rhel8) and not working (centos7) systems. It appears that the process sends itself a SIGURG (line 718 in centos7 trace)

strace.rhel8.out.txt (42.7 KB)
strace.centos7.out.txt (53.2 KB)

Ah ha.

Try

export GODEBUG=asyncpreemptoff=1

To disable the preemption in the go runtime (it's this that causes the SIGURGs).

If this works then it works around a kernel bug.

disabling preemption got rid of the SIGURG, but rclone still gave the same error. Looking at the strace, seems like this is where it fails: (line 641)

statx(AT_FDCWD, "/varidata/researchtemp/hpctmp/zack.ramjan/test2/test", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_TYPE|STATX_MODE|STATX_UID|STATX_GID|STATX_ATIME|STATX_MTIME|STATX_CTIME|STATX_BTIME, 0xc000d26a98) = -1 ENOSYS (Function not implemented)
write(2, "2022/10/04 14:50:29 ERROR : test"..., 842022/10/04 14:50:29 ERROR : test: Failed to read metadata: function not implemented

strace.centos7.nopreempt.out.txt (49.0 KB)

Reading the man page for statx I see

VERSIONS
   statx() was added to Linux in kernel 4.11; library support was added in
   glibc 2.28.

What are the kernel versions for centos7/centos8?

4.11 was released in April 2017 so relatively recently. I think rclone should probably have a fallback if statx isn't implemented.

Can you try this which implements a fallback if statx() returns ENOTSUP. This means we can't read the birth time of the file but that shouldn't be too much of an issue.

v1.60.0-beta.6469.8b52544eb.fix-local-metadata on branch fix-local-metadata (uploaded in 15-30 mins)

That must be it. RHEL 7 uses kernel 3.10xxx, so way below the req'd 4.11 of statx. 3.10 is indeed very old. But on the other hand its branch is still actively being supported and maintained by Redhat/IBM for at least another couple of years, and still fairly popular. So I think your workaround will be of benefit to at a meaningful subset of the community, so thanks for making the fix. I will test shortly and let you know the results.

1 Like

So in testing, it did not seem to change the output:

[zack.ramjan@login02 test2]$  /tmp/rclone-v1.60.0-beta.6469.8b52544eb.fix-local-metadata-linux-amd64/rclone  lsf -F pM test
2022/10/06 10:15:18 ERROR : test: Failed to read metadata: function not implemented
test;{}

towards the end of the strace we have:

close(9)                                = 0
llistxattr("/varidata/researchtemp/hpctmp/zack.ramjan/test2/test", "", 0) = 0
syscall_332(0xffffffffffffff9c, 0xc000b754d8, 0, 0xfff, 0xc000be2a48, 0) = -1 (errno 38)
syscall_332(0xffffffffffffff9c, 0xc000be8100, 0x100, 0x8fb, 0xc000be2a68, 0) = -1 (errno 38)
write(2, "2022/10/06 10:18:59 ERROR : test"..., 842022/10/06 10:18:59 ERROR : test: Failed to read metadata: function not implemented
) = 84
write(1, "test;{}\n", 8test;{}
)                = 8
futex(0xc0005f0148, FUTEX_WAKE_PRIVATE, 1) = 1
exit_group(0)                           = ?
+++ exited with 0 +++

Thanks for testing.

I can see from the strace I used the wrong error number in the code - should have been ENOSYS not ENOTSUP.

Try this - hopefully it will work! You should see one call to syscall_332 returning ENOSYS (38) and then it should use fstatat after that.

v1.60.0-beta.6475.1c8639a79.fix-local-metadata on branch fix-local-metadata (uploaded in 15-30 mins)

[zack.ramjan@login02 test2]$ /tmp/rclone-v1.60.0-beta.6475.1c8639a79.fix-local-metadata-linux-amd64/rclone  lsf -F pM test
test;{"atime":"2022-10-06T13:10:02.520922349-04:00","gid":"1351814256","mode":"100644","mtime":"2022-09-26T15:53:12.849183-04:00","uid":"1351813916"}

seems to be working now on centos7 RHEL7, great!

I am gearing up to move multiple petabytes and hundreds of millions of files of research data so have been doing various testing. I encountered another metadata issue which I just created a new post for. Thanks for all your hard work!

Thanks for testing. I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.60

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.