Copying from S3 where files have been restored from Glacier

[oceanthrsty (Matt)]

updated: user error. I did not actually restore all the files. It was working as it should and only failing on the images that weren't actually restored. Sorry.

I have copied files up to S3 with rclone using a standard storage. And using s3's lifecycles rules I have pushed images into Glacier. Using the AWS console online I initiated a restore of 3 folders of images:
Events/01/674/001/HiRes
Events/01/674/001/Previews
Events/01/674/001/Thumbs

And when I check the object in the console it says "Restoration Complete, expires Aug 11 2021"

But when I try to copy them using rclone I get this message:
2021-07-14 15:22:23 ERROR : Thumbs/1180nr_190728_674_001_0597.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: Events/01/674/001/Thumbs/1180nr_190728_674_001_0597.jpg

What is your rclone version (output from rclone version)

rclone v1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

debian linux

Which cloud storage system are you using? (eg Google Drive)

s3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy gfevents:gfevents/Events/01/674/001 /media/storage/Events/01/674/001 -P --transfers 12

[asdffdsa (jojothehumanmonkey)]

hi,
you can see the storage class using
rclone lsf gfevents:gfevents/Events/01/674/001 --format=pT -vv

and please post full debug logs, not just snippets.

[oceanthrsty (Matt)]

rclone lsf gfevents:gfevents/Events/01/674/001 --format=pT -vv
2021/07/14 15:56:17 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2021/07/14 15:56:17 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "lsf" "gfevents:gfevents/Events/01/674/001" "--format=pT" "-vv"]
2021/07/14 15:56:17 DEBUG : Creating backend with remote "gfevents:gfevents/Events/01/674/001"
HiRes/;
Preview/;
Thumbs/;
2021/07/14 15:56:17 DEBUG : 4 go routines active

There are thousands of images in the folder so I am just posting a snippet of my attempt of copying the first handful of images.

rclone copy gfevents:gfevents/Events/01/674/001 /media/storage/Events/01/674/001 -P --transfers 12 -vvv
2021/07/14 15:56:50 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2021/07/14 15:56:50 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "gfevents:gfevents/Events/01/674/001" "/media/storage/Events/01/674/001" "-P" "--transfers" "12" "-vvv"]
2021/07/14 15:56:50 DEBUG : Creating backend with remote "gfevents:gfevents/Events/01/674/001"
2021/07/14 15:56:50 DEBUG : Creating backend with remote "/media/storage/Events/01/674/001"
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0001.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0001.jpg: Unchanged skipping
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0310.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0310.jpg"
2021-07-14 15:56:51 DEBUG : Local file system at /media/storage/Events/01/674/001: Waiting for checks to finish
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0309.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0309.jpg"
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0312.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0312.jpg"
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0006.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0006.jpg: Unchanged skipping
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0301.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0301.jpg"
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0308.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0308.jpg"
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0306.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0306.jpg"
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0004.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0004.jpg: Unchanged skipping
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0002.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0002.jpg: Unchanged skipping
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0307.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0307.jpg"
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0005.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0005.jpg: Unchanged skipping
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0007.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021-07-14 15:56:51 DEBUG : Thumbs/1180nr_190728_674_001_0007.jpg: Unchanged skipping
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0305.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0305.jpg"
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0304.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0304.jpg"
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0313.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0313.jpg"
2021-07-14 15:56:51 ERROR : Thumbs/1180nr_190728_674_001_0315.jpg: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="gfevents", key="Events/01/674/001/Thumbs/1180nr_190728_674_001_0315.jpg"

[asdffdsa (jojothehumanmonkey)]

sorry, add --recursive the the rclone lsf command and post that debug log

and post the config file, redact id/secret

[oceanthrsty (Matt)]

Hmmm yeah they all still say Glacier. Interesting

rclone lsf gfevents:gfevents/Events/01/674/001 --format=pT -vv --recursive
2021/07/14 16:14:04 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2021/07/14 16:14:04 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "lsf" "gfevents:gfevents/Events/01/674/001" "--format=pT" "-vv" "--recursive"]
2021/07/14 16:14:04 DEBUG : Creating backend with remote "gfevents:gfevents/Events/01/674/001"
HiRes/1180nr_190728_674_001_0001.jpg;GLACIER
HiRes/1180nr_190728_674_001_0002.jpg;GLACIER
HiRes/1180nr_190728_674_001_0003.jpg;GLACIER
HiRes/1180nr_190728_674_001_0004.jpg;GLACIER
HiRes/1180nr_190728_674_001_0005.jpg;GLACIER
HiRes/1180nr_190728_674_001_0006.jpg;GLACIER
HiRes/1180nr_190728_674_001_0007.jpg;GLACIER
HiRes/1180nr_190728_674_001_0008.jpg;GLACIER
HiRes/1180nr_190728_674_001_0009.jpg;GLACIER
HiRes/1180nr_190728_674_001_0010.jpg;GLACIER
HiRes/1180nr_190728_674_001_0011.jpg;GLACIER
HiRes/1180nr_190728_674_001_0012.jpg;GLACIER
... this goes on for every file

config file

[gfevents]
type = s3
provider = AWS
env_auth = false
access_key_id = xxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxxxx
region = us-east-1

And here's a screenshot from the AWS s3 console

But.. if I try to download the image from the AWS console it DOES download. So they are restored.

[m4n63 (Magnus)]

Does not a glacier restore copy them to a new bucket? So maybe you need a new path (or even a new remote) to access the restored files.

[oceanthrsty (Matt)]

that's only if you're storing files directly in Glacier. These files are stored in s3 and just their storage class has changed.

[asdffdsa (jojothehumanmonkey)]

this worked for me

1. rclone copy two files from local to a remote bucket using storage_class = GLACIER
2. use aws s3 website to restore one of the two files
3. rclone copy from that remote bucket to local.

and here is the condensed debug log

rclone copy new-bucket-e69eb904: C:\data\rclone\scripts\rr\other\aws\dest  -vv 
ERROR : 02.txt: Failed to copy: failed to open source object: Object in GLACIER, restore first: bucket="new-bucket-e69eb904", key="test/02.txt"
DEBUG : 01.txt: MD5 = c4ca4238a0b923820dcc509a6f75849b OK
INFO  : 01.txt: Copied (new)
Transferred:   	         1 / 1 Bytes, 100%, 9 Bytes/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 2 / 2, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

[oceanthrsty (Matt)]

Well... I will admit where I screwed up. I'm sorry to waste your time.
The copy WAS working. I just didn't successfully restore all the files.

Sorry everybody.

The worst part was. I guess when I ran rclone the first time I didn't see that they were successful. So after when I was troubleshooting it would skip the successful ones and the only fail on the dead ones. So I never thought to check for the successful transfer.