UID/GID not restored for Objects Stored in Glacier

Suspected Bug
1

What is the problem you are having with rclone?

UID/GID is NOT Being restored for Objects Stored GLACIER. They ARE being restored for Objects stored in S3 Standard. The object in glacier has already been "restored" in the Amazon console so that the data can be copied. The --metadata flag is set. Same behavior with sync or copy.

Other information (Modified time) IS correct

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2-DEV
- os/version: rocky 8.8 (64 bit)
- os/kernel: 4.18.0-477.15.1.el8_8.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.4
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config /root/rcs3/POC/config/rclone.conf --s3-shared-credentials-file /root/rcs3/POC/config/credentials --metadata --links --transfers 4 --checkers 32 --log-level INFO --log-file /tmp/testdir.log --filter-from /tmp/testdir.filter **sync** s3-backup:pub-all/dfs8/pub/ppapadop /tmp/ppapadop

The rclone config contents with secrets removed.

s3-native]
type = s3
provider = AWS
env_auth = true
region = us-west-2
location_constraint = us-west-2
acl = bucket-owner-full-control
server_side_encryption = AES256
storage_class = STANDARD

[s3-backup]
type = alias
remote = s3-native:ppapadop-tmpstore1-uci-bkup-bucket

[s3-inventory]
type = alias
remote = s3-native:ppapadop-tmpstore1-uci-inventory

[s3-crypt]
type = crypt
remote = s3-native:ppapadop-tmpstore1-uci-bkup-bucket
filename_encryption = off
directory_name_encryption = false
password = <redacted>
password2 = <redacted>

A log from the command with the -vv flag

2023/11/14 17:15:09 DEBUG : rclone: Version "v1.64.2-DEV" starting with parameters ["rclone" "--config" "/root/rcs3/POC/config/rclone.conf" "--s3-shared-credentials-file" "/root/rcs3/POC/config/credentials" "-vv" "--metadata" "--links" "--transfers" "2" "--checkers" "32" "--log-file" "/tmp/testdir.log" "--filter-from" "/tmp/testdir.filter" "sync" "s3-backup:pub-all/dfs8/pub/ppapadop" "/tmp/ppapadop"]
2023/11/14 17:15:09 DEBUG : Creating backend with remote "s3-backup:pub-all/dfs8/pub/ppapadop"
2023/11/14 17:15:09 DEBUG : Using config file from "/root/rcs3/POC/config/rclone.conf"
2023/11/14 17:15:09 DEBUG : Creating backend with remote "s3-native:ppapadop-tmpstore1-uci-bkup-bucket/pub-all/dfs8/pub/ppapadop"
2023/11/14 17:15:09 DEBUG : s3-native: detected overridden config - adding "{DSqVk}" suffix to name
2023/11/14 17:15:09 DEBUG : fs cache: renaming cache item "s3-native:ppapadop-tmpstore1-uci-bkup-bucket/pub-all/dfs8/pub/ppapadop" to be canonical "s3-native{DSqVk}:ppapadop-tmpstore1-uci-bkup-bucket/pub-all/dfs8/pub/ppapadop"
2023/11/14 17:15:09 DEBUG : fs cache: renaming cache item "s3-backup:pub-all/dfs8/pub/ppapadop" to be canonical "s3-native{DSqVk}:ppapadop-tmpstore1-uci-bkup-bucket/pub-all/dfs8/pub/ppapadop"
2023/11/14 17:15:09 DEBUG : Creating backend with remote "/tmp/ppapadop"
2023/11/14 17:15:09 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2023/11/14 17:15:09 DEBUG : fs cache: renaming cache item "/tmp/ppapadop" to be canonical "local{b6816}:/tmp/ppapadop"
2023/11/14 17:15:10 DEBUG : .local: Excluded
2023/11/14 17:15:10 DEBUG : floop_va: Excluded
2023/11/14 17:15:10 DEBUG : foob: Excluded
2023/11/14 17:15:10 DEBUG : glibc-2.14: Excluded
2023/11/14 17:15:10 DEBUG : glibc-2.17: Excluded
2023/11/14 17:15:10 DEBUG : glibc: Excluded
2023/11/14 17:15:10 DEBUG : miniconda3: Excluded
2023/11/14 17:15:10 DEBUG : modulefiles: Excluded
2023/11/14 17:15:10 DEBUG : mytest3: Excluded
2023/11/14 17:15:10 DEBUG : phylo: Excluded
2023/11/14 17:15:10 DEBUG : results: Excluded
2023/11/14 17:15:10 DEBUG : sequences: Excluded
2023/11/14 17:15:10 DEBUG : sratest: Excluded
2023/11/14 17:15:10 DEBUG : tdlong-test: Excluded
2023/11/14 17:15:10 DEBUG : tdlong: Excluded
2023/11/14 17:15:10 DEBUG : ChaCha20.tar.gz: Excluded
2023/11/14 17:15:10 DEBUG : Hchar_Modeler.tar: Excluded
2023/11/14 17:15:10 DEBUG : Wunderlich.tar: Excluded
2023/11/14 17:15:10 DEBUG : glibc-2.17.tar.gz: Excluded
2023/11/14 17:15:10 DEBUG : install_pkg.sh: Excluded
2023/11/14 17:15:10 DEBUG : openfoam-docker: Excluded
2023/11/14 17:15:10 DEBUG : openfoam.sif: Excluded
2023/11/14 17:15:10 DEBUG : tdlong-testdir.tar: Excluded
2023/11/14 17:15:10 DEBUG : vg_small.tar: Excluded
2023/11/14 17:15:10 DEBUG : testdir/.bam: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L1-P05-TCGCAGT-ATAGAGAG-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L1-P05-TCGCAGT-ATAGAGAG-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L1-P08-CGCCATT-ATAGAGAG-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L1-P08-CGCCATT-ATAGAGAG-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L2-P08-AACTCCT-ATAGAGAG-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L2-P08-AACTCCT-ATAGAGAG-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L3-P03-CGATGGA-GCGATCTA-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L3-P03-CGATGGA-GCGATCTA-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L3-P08-AACTCCT-GCGATCTA-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L3-P08-AACTCCT-GCGATCTA-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L4-P10-GACTCTA-ATAGAGAG-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L4-P10-GACTCTA-ATAGAGAG-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L5-P04-CGAGTAG-ATAGAGAG-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L5-P04-CGAGTAG-ATAGAGAG-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L6-P03-TTCGACG-GCGATCTA-READ1-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/4R015-L6-P03-TTCGACG-GCGATCTA-READ2-Sequences.txt.gz: Excluded
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa: Excluded
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.amb: Excluded
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.bwt: Excluded
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.fai: Excluded
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.sa: Excluded
2023/11/14 17:15:10 DEBUG : testdir/test.in: Excluded
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.ann: Need to transfer - File not found at Destination
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: Need to transfer - File not found at Destination
2023/11/14 17:15:10 DEBUG : Local file system at /tmp/ppapadop: Waiting for checks to finish
2023/11/14 17:15:10 DEBUG : Local file system at /tmp/ppapadop: Waiting for transfers to finish
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: write buffer set to 131072
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: Starting multi-thread copy with 10 chunks of size 64Mi with 4 parallel streams
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 4/10 (201326592-268435456) size 64Mi starting
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 1/10 (0-67108864) size 64Mi starting
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 2/10 (67108864-134217728) size 64Mi starting
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 3/10 (134217728-201326592) size 64Mi starting
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.ann: md5 = 08726c6301a1b3e0973001600a852715 OK
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.ann.fapoxux4.partial: renamed to: testdir/pero.scaff.fa.ann
2023/11/14 17:15:10 INFO  : testdir/pero.scaff.fa.ann: Copied (new)
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 2
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 1
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 0
2023/11/14 17:15:10 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 3
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 2/10 (67108864-134217728) size 64Mi finished
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 5/10 (268435456-335544320) size 64Mi starting
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 4
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 4/10 (201326592-268435456) size 64Mi finished
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 6/10 (335544320-402653184) size 64Mi starting
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 3/10 (134217728-201326592) size 64Mi finished
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 7/10 (402653184-469762048) size 64Mi starting
2023/11/14 17:15:15 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 5
2023/11/14 17:15:16 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 6
2023/11/14 17:15:20 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 6/10 (335544320-402653184) size 64Mi finished
2023/11/14 17:15:20 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 8/10 (469762048-536870912) size 64Mi starting
2023/11/14 17:15:20 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 7
2023/11/14 17:15:21 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 5/10 (268435456-335544320) size 64Mi finished
2023/11/14 17:15:21 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 9/10 (536870912-603979776) size 64Mi starting
2023/11/14 17:15:21 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 8
2023/11/14 17:15:22 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 7/10 (402653184-469762048) size 64Mi finished
2023/11/14 17:15:22 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 10/10 (603979776-618791129) size 14.125Mi starting
2023/11/14 17:15:22 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: writing chunk 9
2023/11/14 17:15:22 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 1/10 (0-67108864) size 64Mi finished
2023/11/14 17:15:23 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 10/10 (603979776-618791129) size 14.125Mi finished
2023/11/14 17:15:25 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 8/10 (469762048-536870912) size 64Mi finished
2023/11/14 17:15:26 DEBUG : testdir/pero.scaff.fa.pac: multi-thread copy: chunk 9/10 (536870912-603979776) size 64Mi finished
2023/11/14 17:15:26 DEBUG : testdir/pero.scaff.fa.pac: Finished multi-thread copy with 10 parts of size 64Mi
2023/11/14 17:15:27 DEBUG : testdir/pero.scaff.fa.pac: md5 = b436224037c28753190c0628a7269eb0 OK
2023/11/14 17:15:27 DEBUG : testdir/pero.scaff.fa.pac.vazifun1.partial: renamed to: testdir/pero.scaff.fa.pac
2023/11/14 17:15:27 INFO  : testdir/pero.scaff.fa.pac: Multi-thread Copied (new)
2023/11/14 17:15:27 DEBUG : Waiting for deletions to finish
2023/11/14 17:15:27 INFO  : 
Transferred:   	  590.198 MiB / 590.198 MiB, 100%, 34.601 MiB/s, ETA 0s
Transferred:            2 / 2, 100%
Elapsed time:        18.1s

2023/11/14 17:15:27 DEBUG : 14 go routines active

**Output of ls -l AFTER transfer. Please note that owner of the pac file is root:root, ownership of ann file IS correct

# ls -l /tmp/ppapadop/testdir/
total 604368
-rw-rw-r-- 1 ppapadop ppapadop     76471 Apr  5  2019 pero.scaff.fa.ann
-rw-r--r-- 1 root     root     618791129 Apr  5  2019 pero.scaff.fa.pac

rclone lsf command illustrates that S3 has the correct UID/GID stored in metadata. The only real difference between the two files:

  • .STANDARD vs. GLACIER Tier
  • .GLACIER file has been restored in AWS console so that data can be copied
# rclone --config /root/rcs3/POC/config/rclone.conf --s3-shared-credentials-file /root/rcs3/POC/config/credentials --metadata --links --transfers 4 --checkers 32 **lsf --format="pTM"** s3-backup:pub-all/dfs8/pub/ppapadop/testdir | grep -e ann -e pac

**pero.scaff.fa.ann;**STANDARD;{"atime":"2023-03-30T07:36:30-07:00","btime":"2023-06-19T03:23:03Z","content-type":"application/octet-stream",**"gid":"1698224"**,"mode":"100664","mtime":"2019-04-05T12:37:55-07:00","tier":"STANDARD",**"uid":"1698224"**}
**pero.scaff.fa.pac;GLACIER**;{"atime":"2023-03-30T07:37:06-07:00","btime":"2023-06-19T03:23:15Z","content-type":"application/octet-stream",**"gid":"1698224"**,"mode":"100664","mtime":"2019-04-05T12:37:55-07:00","tier":"GLACIER",**"uid":"1698224"**}

The md5sum of the restored file and the original file are the same:

# md5sum /pub/ppapadop/testdir/pero.scaff.fa.pac /tmp/ppapadop/testdir/pero.scaff.fa.pac
b436224037c28753190c0628a7269eb0  /pub/ppapadop/testdir/pero.scaff.fa.pac
b436224037c28753190c0628a7269eb0  /tmp/ppapadop/testdir/pero.scaff.fa.pac

Any insights would be appreciated. While it's possible to "fix" the ownership afterwards - seems that rclone shouldn't care if data is in Glacier or Standard. It might be an artifact that these are really two different objects (the original in Glacier and the cache version) and that querying the metadata of the cached version is not giving the same results (By cached, I mean the temporary copy of the object in S3 standard that is there as a result of the 'restore object' operation inside of S3.).. And rclone is using the ID of the cached object (pointing to a possible S3 bug).

1 Like
2

I believe that this is a genuine BUG and it looks like it has something do do with multi-part.

When I set the multi-thread-cutoff to be above the size of the file (killing performance in the process), UID/GID is restored properly.

Command with --multi-thread-cutoff set

rclone --config /root/rcs3/POC/config/rclone.conf --s3-shared-credentials-file /root/rcs3/POC/config/credentials --metadata --links --transfers 2 --checkers 32 --log-level INFO --log-file /tmp/testdir.log --filter-from /tmp/testdir.filter --multi-thread-cutoff 1024Mi copy s3-backup:pub-all/dfs8/pub/ppapadop /tmp/ppapadop

Tail of the log file showing just single threaded copy (time marker 08:06)

# tail /tmp/testdir.log 
Elapsed time:        18.1s

2023/11/14 17:15:27 DEBUG : 14 go routines active
2023/11/15 08:06:19 INFO  : testdir/pero.scaff.fa.pac: Copied (new)
2023/11/15 08:06:19 INFO  : 
Transferred:   	  590.125 MiB / 590.125 MiB, 100%, 12.426 MiB/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:        47.5s

And the file (*pac) properly restored with UID and GID set as expected.

# ls -l /tmp/ppapadop/testdir/
total 604368
-rw-rw-r-- 1 ppapadop ppapadop     76471 Apr  5  2019 pero.scaff.fa.ann
-rw-rw-r-- 1 ppapadop ppapadop 618791129 Apr  5  2019 pero.scaff.fa.pac
1 Like
3

I don't believe this has anything to do with the object being stored in Glacier. I just uploaded the same file to S3 standard and then downloaded via a threaded download. Here's how to reproduce with a file above the multi-thread-cutoff threshold.

  1. Upload
# rclone --config /root/rcs3/POC/config/rclone.conf --s3-shared-credentials-file /root/rcs3/POC/config/credentials --metadata --links --transfers 2 --checkers 32 --progress --include pero.scaff.fa.pac copy /pub/ppapadop/testdir s3-backup:testing
Transferred:      590.125 MiB / 590.125 MiB, 100%, 45.393 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        13.3s
  1. Checking metadata on S3, Object is stored in standard. Other metadata is correct
# rclone --config /root/rcs3/POC/config/rclone.conf --s3-shared-credentials-file /root/rcs3/POC/config/credentials --metadata --links --transfers 2 --checkers 32 lsf --format "ptTM" s3-backup:testing
pero.scaff.fa.pac;2019-04-05 12:37:55;STANDARD;{"atime":"2023-06-09T11:34:03-07:00","btime":"2023-11-15T16:34:30Z","content-type":"application/octet-stream","gid":"1698224","mode":"100644","mtime":"2019-04-05T12:37:55-07:00","tier":"STANDARD","uid":"1698224"}
  1. Downloading file without multi-part-cutoff specified
# rclone --config /root/rcs3/POC/config/rclone.conf --s3-shared-credentials-file /root/rcs3/POC/config/credentials --metadata --links --transfers 2 --checkers 32  --include pero.scaff.fa.pac  copy s3-backup:testing /tmp/ppapadop
  1. checking ownership (incorrect), should be ppapadop:ppapadop
# ls -l /tmp/ppapadop
total 604292
-rw-r--r-- 1 root root 618791129 Apr  5  2019 pero.scaff.fa.pac
drwxr-xr-x 2 root root        56 Nov 15 08:06 testdir
  1. Redoing by erasing local copy, downloading with the --multi-thread-cutoff 1024Mi and things are now correct
# rm /tmp/ppapadop/pero.scaff.fa.pac 
rm: remove regular file '/tmp/ppapadop/pero.scaff.fa.pac'? y
# rclone --config /root/rcs3/POC/config/rclone.conf --s3-shared-credentials-file /root/rcs3/POC/config/credentials --metadata --links --transfers 2 --checkers 32 --multi-thread-cutoff 1024Mi --progress --include pero.scaff.fa.pac  copy s3-backup:testing /tmp/ppapadop
Transferred:      590.125 MiB / 590.125 MiB, 100%, 14.972 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        40.6s

# ls -l /tmp/ppapadop
total 604292
-rw-r--r-- 1 ppapadop ppapadop 618791129 Apr  5  2019 pero.scaff.fa.pac
drwxr-xr-x 2 root     root            56 Nov 15 08:06 testdir
4

Yes this looks like a bug. Multi thread uploads missing metadata.

Can you make an issue on GitHub with a simple way to reproduce and I'll have a go at fixing.

Thank you.

5

Thanks, Nick. I've created the issue in Github
Metadata (UID/GID) not set properly for multipart downloads from S3 to Unix file system · Issue #7424 · rclone/rclone (github.com)

As an aside - rclone is an awesome tool. Many thanks to you and the many developers from a very grateful user.

1 Like
6

Thank you for a great report and an awesome amount of details :slight_smile:

1 Like
7

Thank you - I managed to reproduce.

This won't be a quick fix unfortunately as I'll have to change one of rclone's internal interfaces which will require modding a lot of code :frowning: Will try to get it done for 1.65 - further updates on the issue.

8

Thanks for your quick review of this -- please let me know if I can help test in any way. once you have a candidate update.

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.