Can't restore files from AWS Glacier Deep - only directories

Help and Support
1

What is the problem you are having with rclone?

Can't restore a single file from AWS Glacier Deep, but the same command pointing to a directory works fine.

when pointed to a file it just says: Failed to backend: is a file not a directory
when pointed to a directory it correctly outputs json with retrieval status of all files in the folder and web gui shows that retrieval has been started.

Using aws cli to restore that single file works fine:

aws s3api restore-object --bucket <REDACTED> --key crypt/q6vf4q0/9j0ql9r58i0 --restore-request '{"Days":1,"GlacierJobParameters":{"Tier":"Standard"}}'

I am however using a different IAM account (different permissions) for aws cli than for rclone, but if rclone manages to retrieve directories already this shouldn't be the problem.

And rclone documention mentions that it should be possible to retrieve single objects.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: arch "rolling" (64 bit)
- os/kernel: 6.3.9-arch1-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.4
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS Glacier Deep

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone backend restore glacier-deep-bucket:<BUCKET>/crypt/q6vf83g94q0/9j0qlm1jprgiv69r58i0 -o lifetime=1 -o priority=Standard

it's been encrypted, I've truncated the names slightly

The rclone config contents with secrets removed.

[glacier-deep-bucket]
type = s3
provider = AWS
access_key_id = <REDACTED>
secret_access_key = <REDACTED>
region = eu-west-1
location_constraint = eu-west-1
acl = private
storage_class = DEEP_ARCHIVE

A log from the command with the -vv flag

<7>DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "backend" "restore" "glacier-deep-bucket:<REDACTED>/crypt/q6vf83g94q0/9j0qlm1jprgiv69r58i0" "-o" "lifetime=1" "-o" "priority=Standard" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : 4 go routines active
Failed to backend: is a file not a directory
2

Have a look at this post:

3

Thanks, but I'm not restoring via the encrypted remote, but directly from the aws glacier. To get the correct name run the

rclone cryptdecode glacier-encrypted: memes/cat.png --reverse

on the encrypted remote and use that path to restore it from glacier: . Used it with aws cli and it worked fine.

4

use --include "filename.txt"

1 Like
5

oh wow, it worked. Thank you. I haven't thought about that.

Should it be added it to the documentation?

6

Yes it should:)

1 Like
7

Submitted:

1 Like
8

Do not forget to create PR from this change - I think you did not do it.

9

oops, thank you. It should be ready now (hopefully):

10

I was looking at the PR above and through the code and I think this is actually a bug and it should work just fine restoring a single file.

Can you try this please restoring a single file like you tried originally?

v1.63.0-beta.7079.35b976c42.fix-backend-file on branch fix-backend-file (uploaded in 15-30 mins)

2 Likes
11

thank you, I got rclone-v1.63.0-beta.7079.35b976c42.fix-backend-file-linux-amd64.zip and run it:
./rclone backend restore glacier-deep-bucket:<REDACTED>/crypt/q6vf...4q0/cm189...0bs0o --config =(gpg -d /home/.../rclone.conf.gpg) -o lifetime=1 -o priority=Standard -vv

sadly still Failed to backend: is a file not a directory, but it had an extra DEBUG line:

<7>DEBUG : name = "glacier-deep-bucket", root = "<REDACTED>/crypt/q6...q0/cm1...s0o", opt = &s3.Options{Provider:"AWS", <REDACTED>}

it showed a lot of internal information, so I've trucated it, but if there could be anything useful there please let me know. Other debug lines were the same (except that 6 go routines run instead of 4)

12

I looked at the code and what I did wasn't nearly sufficient to make it work.

In fact if I did make it work it would probably render other backend commands unsafe to use when given a file.

So I think I'm going to ditch this approach and go with your patch to the docs!

Thank you for testing.

1 Like
13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.