Storj s3 union “Failed to copy: object not found”

What is the problem you are having with rclone?

I am trying a UNION remote with two "storj s3 backends". Copying a file to union works fine, but copying nested folders leads to an error "Failed to copy: object not found"

Run the command 'rclone version' and share the full output of the command.

# rclone --version
rclone v1.59.0-beta.6139.8e5e230b8
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.13.0-1021-oracle (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.18.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Storj s3 backend

The command you were trying to run (eg rclone copy /tmp remote:tmp)


# tree testfolder
testfolder
└── subfolder
    └── testfile

1 directory, 1 file

# rclone copy testfolder s3:
2022/05/13 15:22:51 ERROR : subfolder/testfile: Failed to copy: object not found
2022/05/13 15:22:51 ERROR : Attempt 1/3 failed with 1 errors and: object not found
2022/05/13 15:22:52 ERROR : subfolder/testfile: Failed to copy: object not found
2022/05/13 15:22:52 ERROR : Attempt 2/3 failed with 1 errors and: object not found
2022/05/13 15:22:54 ERROR : subfolder/testfile: Failed to copy: object not found
2022/05/13 15:22:54 ERROR : Attempt 3/3 failed with 1 errors and: object not found
2022/05/13 15:22:54 Failed to copy: object not found

The rclone config contents with secrets removed.


[s2]
type = s3
provider = Storj
access_key_id = REDACTED
secret_access_key = REDACTED
endpoint = https://gateway.ap1.storjshare.io

[s3]
type = union
upstreams = s2:whyred s1:whyred
create_policy = eprand

[s1]
type = s3
provider = Storj
env_auth = false
access_key_id = REDACTED
secret_access_key = REDACTED
endpoint = https://gateway.ap1.storjshare.io

whyred is the bucket name in both s1: and s2:

A log from the command with the -vv flag


2022/05/13 15:08:29 DEBUG : rclone: Version "v1.59.0-beta.6139.8e5e230b8" starting with parameters ["rclone" "copy" "testfolder" "s3:" "--log-file" "log.txt" "-vv"]
2022/05/13 15:08:29 DEBUG : Creating backend with remote "testfolder"
2022/05/13 15:08:29 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/05/13 15:08:29 DEBUG : fs cache: renaming cache item "testfolder" to be canonical "/root/testfolder"
2022/05/13 15:08:29 DEBUG : Creating backend with remote "s3:"
2022/05/13 15:08:29 DEBUG : Creating backend with remote "s2:whyred"
2022/05/13 15:08:29 DEBUG : Creating backend with remote "s1:whyred"
2022/05/13 15:08:29 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpRand, searchPolicy = *policy.FF
2022/05/13 15:08:30 DEBUG : union root '': Waiting for checks to finish
2022/05/13 15:08:30 DEBUG : union root '': Waiting for transfers to finish
2022/05/13 15:08:31 ERROR : subfolder/testfile: Failed to copy: object not found
2022/05/13 15:08:31 ERROR : Attempt 1/3 failed with 1 errors and: object not found
2022/05/13 15:08:31 DEBUG : union root '': Waiting for checks to finish
2022/05/13 15:08:31 DEBUG : union root '': Waiting for transfers to finish
2022/05/13 15:08:32 ERROR : subfolder/testfile: Failed to copy: object not found
2022/05/13 15:08:32 ERROR : Attempt 2/3 failed with 1 errors and: object not found
2022/05/13 15:08:32 DEBUG : union root '': Waiting for checks to finish
2022/05/13 15:08:32 DEBUG : union root '': Waiting for transfers to finish
2022/05/13 15:08:33 ERROR : subfolder/testfile: Failed to copy: object not found
2022/05/13 15:08:33 ERROR : Attempt 3/3 failed with 1 errors and: object not found
2022/05/13 15:08:33 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         4.0s

2022/05/13 15:08:33 DEBUG : 7 go routines active
2022/05/13 15:08:33 Failed to copy: object not found

I can replicate this problem. It isn't a problem with the Storj backend in particular, it is a problem with bucket based backends which don't store directories. So when the union backend makes a directory and then tries to read it, it isn't there.

I can't think of an easy way to fix this as the union backend seems to rely on directories being created...

Can you create an issue on GitHub about this please?

Sure
https://github.com/rclone/rclone/issues/6170

1 Like