Issue using copyto for single files

Ron · June 7, 2023, 5:38pm

What is the problem you are having with rclone?

I'm unable to copy single files from a local directory to an s3 bucket. If I move a single file into a directory, and use copy to recursively copy the directory, it works.

Run the command 'rclone version' and share the full output of the command.

rclone --version
rclone v1.62.2-DEV

os/version: centos 7.9.2009 (64 bit)
os/kernel: 3.10.0-1160.88.1.el7.x86_64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.18
go/linking: dynamic
go/tags: none

yes

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

$ rclone ls s3-standard:dr-storage-systems/test/
<empty>

$ rclone -vvv --log-file=./rclone-copyto.log copyto ./test-me.txt s3-standard:dr-storage-systems/test/test-me.txt
<fails, see log>
$ rclone ls s3-standard:dr-storage-systems/test/
<empty>

$ mv ./test-me.txt ./testdir/

$ rclone -vvv --log-file=./rclone-copy.log  copy ./testdir  s3-standard:dr-storage-systems/test
<works see log>

$ rclone ls s3-standard:dr-storage-systems/test/
 5 test-me.txt

The rclone config contents with secrets removed.

[s3-standard]
type = s3
provider = AWS
env_auth = true
region = us-east-1
endpoint = <my s3 endpoint>
sts_endpoint = <my sts endpoint>
storage_class = STANDARD

A log from the command with the `-vv` flag

rclone-copyto.log (fails to copy individual file)
2023/06/07 13:05:05 DEBUG : rclone: Version "v1.62.2-DEV" starting with parameters ["rclone" "-vvv" "--log-file=./rclone-copyto.log" "copyto" "./test-me.txt" "s3-standard:dr-storage-systems/test/test-me.txt"]
2023/06/07 13:05:05 DEBUG : Creating backend with remote "./test-me.txt"
2023/06/07 13:05:05 DEBUG : Using config file from "/home/<user>/.config/rclone/rclone.conf"
2023/06/07 13:05:05 DEBUG : fs cache: adding new entry for parent of "./test-me.txt", "/home/<user>"
2023/06/07 13:05:05 DEBUG : Creating backend with remote "s3-standard:dr-storage-systems/test/"
2023/06/07 13:05:05 DEBUG : Resolving service "s3" region "us-east-1"
2023/06/07 13:05:05 DEBUG : fs cache: renaming cache item "s3-standard:dr-storage-systems/test/" to be canonical "s3-standard:dr-storage-systems/test"
2023/06/07 13:05:06 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 13:05:06 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: 12DYE0NGV2TY4MKG, host id: B0NWeASwN+7ELt5DXA8sMjBX02gzKu2/+23DNpDAU8t0DxjrkhF/dXF30RgQ7t6VxewsoB3a3SY=
2023/06/07 13:05:06 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
        status code: 403, request id: 12DYE0NGV2TY4MKG, host id: B0NWeASwN+7ELt5DXA8sMjBX02gzKu2/+23DNpDAU8t0DxjrkhF/dXF30RgQ7t6VxewsoB3a3SY=
2023/06/07 13:05:06 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 13:05:06 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: 12DK7JG55RX18ZCZ, host id: vOnd3q3DjVfUQj14vMFQR6TkV5FOG2OmLKDBMRKcrdfaMmu1lD9fLFYgOBisZ7IYOxGyvPokkTw=
2023/06/07 13:05:06 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
        status code: 403, request id: 12DK7JG55RX18ZCZ, host id: vOnd3q3DjVfUQj14vMFQR6TkV5FOG2OmLKDBMRKcrdfaMmu1lD9fLFYgOBisZ7IYOxGyvPokkTw=
2023/06/07 13:05:06 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 13:05:06 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: 12DRNF5FK300DNHX, host id: uwEWZDKq2lrf+4U5qO/ApouC1bMhp8F5QMk5eoqVQCTulPo5M+4MpzQc7/R2Tv6jHKvgSl2rXu4=
2023/06/07 13:05:06 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
        status code: 403, request id: 12DRNF5FK300DNHX, host id: uwEWZDKq2lrf+4U5qO/ApouC1bMhp8F5QMk5eoqVQCTulPo5M+4MpzQc7/R2Tv6jHKvgSl2rXu4=
2023/06/07 13:05:06 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.3s

2023/06/07 13:05:06 DEBUG : 6 go routines active
2023/06/07 13:05:06 Failed to copyto: AccessDenied: Access Denied
        status code: 403, request id: 12DRNF5FK300DNHX, host id: uwEWZDKq2lrf+4U5qO/ApouC1bMhp8F5QMk5eoqVQCTulPo5M+4MpzQc7/R2Tv6jHKvgSl2rXu4=

rclone-copy.log (works)
2023/06/07 13:05:25 DEBUG : rclone: Version "v1.62.2-DEV" starting with parameters ["rclone" "-vvv" "--log-file=./rclone-copy.log" "copy" "./testdir" "s3-standard:dr-storage-systems/test"]
2023/06/07 13:05:25 DEBUG : Creating backend with remote "./testdir"
2023/06/07 13:05:25 DEBUG : Using config file from "/home/<user>/.config/rclone/rclone.conf"
2023/06/07 13:05:25 DEBUG : fs cache: renaming cache item "./testdir" to be canonical "/home/<user>/testdir"
2023/06/07 13:05:25 DEBUG : Creating backend with remote "s3-standard:dr-storage-systems/test"
2023/06/07 13:05:25 DEBUG : Resolving service "s3" region "us-east-1"
2023/06/07 13:05:25 DEBUG : S3 bucket ncbi-dr-storage-systems path test: Waiting for checks to finish
2023/06/07 13:05:25 DEBUG : S3 bucket ncbi-dr-storage-systems path test: Waiting for transfers to finish
2023/06/07 13:05:25 DEBUG : test-me.txt: md5 = d8e8fca2dc0f896fd7cb4cb0031ba249 OK
2023/06/07 13:05:25 INFO  : test-me.txt: Copied (new)
2023/06/07 13:05:25 INFO  : 
Transferred:              5 B / 5 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.2s

2023/06/07 13:05:25 DEBUG : 5 go routines active

kapitainsky · June 7, 2023, 6:19pm

I did quick test myself:

$ rclone ls s3:test-bucket-kptsky/test
<empty>

$ rclone copyto ./test-me.txt s3:test-bucket-kptsky/test/test-me.txt

$ rclone ls s3:test-bucket-kptsky/test
2023/06/07 19:16:25 NOTICE: S3 bucket test-bucket-kptsky: Switched region to "eu-west-3" from "us-east-1"
        2 test-me.txt

So there is no obvious bug in rclone.

and in your case it clearly states:

2023/06/07 13:05:06 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied

can you double check permissions?

kapitainsky · June 7, 2023, 6:25pm

Could you also try copyto from your ./testdir? where copy works

kapitainsky · June 7, 2023, 6:38pm

and lastly I see that you are using some beta version. It is always better to stick to release one. Who knows in what stage of some testing your beta was

Ron · June 7, 2023, 7:05pm

I downloaded the latest binary - but see the same problem. I don't understand the access denied error. A recursive copy using rclone copy from my home directory (testdir) works, but rclone copyto with a single file fails with access denied. Here's the same test with the latest binary

$ ./rclone --version
rclone v1.62.2

os/version: centos 7.9.2009 (64 bit)
os/kernel: 3.10.0-1160.88.1.el7.x86_64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: none

copyto fails with access denied

./rclone -vvv copyto /home//test-me.txt s3-standard:dr-storage-systems/test/test-me.txt
2023/06/07 14:50:25 DEBUG : rclone: Version "v1.62.2" starting with parameters ["./rclone" "-vvv" "copyto" "/home//test-me.txt" "s3-standard:dr-storage-systems/test/test-me.txt"]
2023/06/07 14:50:25 DEBUG : Creating backend with remote "/home//test-me.txt"
2023/06/07 14:50:25 DEBUG : Using config file from "/home//.config/rclone/rclone.conf"
2023/06/07 14:50:25 DEBUG : fs cache: adding new entry for parent of "/home//test-me.txt", "/home/"
2023/06/07 14:50:25 DEBUG : Creating backend with remote "s3-standard:dr-storage-systems/test/"
2023/06/07 14:50:25 DEBUG : fs cache: renaming cache item "s3-standard:dr-storage-systems/test/" to be canonical "s3-standard:dr-storage-systems/test"
2023/06/07 14:50:25 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 14:50:25 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: YW3KXB9A4JGZJDNY, host id: 3fl92TdrlIcDxALIsfwHi9Ml/7Xu3vqLV78wyGC24Tox5FUH314vHRj0WiWKBUvKCA8vvGYYuy7866r4nku6bA==
2023/06/07 14:50:25 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: YW3KXB9A4JGZJDNY, host id: 3fl92TdrlIcDxALIsfwHi9Ml/7Xu3vqLV78wyGC24Tox5FUH314vHRj0WiWKBUvKCA8vvGYYuy7866r4nku6bA==
2023/06/07 14:50:25 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 14:50:25 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: YW3N87T7XFR8PZ4X, host id: AyrqgNzIH/IiWSD5qXoquJSCeOx8gDxq37PFtbBGxx3sYtwr8TwcaHSrfgLsPptj9xDZ/fxnmjBNT19mBjLYPQ==
2023/06/07 14:50:25 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: YW3N87T7XFR8PZ4X, host id: AyrqgNzIH/IiWSD5qXoquJSCeOx8gDxq37PFtbBGxx3sYtwr8TwcaHSrfgLsPptj9xDZ/fxnmjBNT19mBjLYPQ==
2023/06/07 14:50:25 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 14:50:25 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: YW3GW6VQABDR3RSC, host id: Gx6TbQbiOZLsr3Q4xZz96IaC9HT4LVUma232kwiEfg2o7xx+852YjlUsDgZx7vx3+SUb4wjfy6wNbjWh6csiyw==
2023/06/07 14:50:25 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: YW3GW6VQABDR3RSC, host id: Gx6TbQbiOZLsr3Q4xZz96IaC9HT4LVUma232kwiEfg2o7xx+852YjlUsDgZx7vx3+SUb4wjfy6wNbjWh6csiyw==
2023/06/07 14:50:25 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.2s

Move the file to testdir in my home directory, and use copy of the testdir - works without access denied

./rclone -vvv copy /home//testdir s3-standard:dr-storage-systems/test/
2023/06/07 14:51:55 DEBUG : rclone: Version "v1.62.2" starting with parameters ["./rclone" "-vvv" "copy" "/home//testdir" "s3-standard:dr-storage-systems/test/"]
2023/06/07 14:51:55 DEBUG : Creating backend with remote "/home//testdir"
2023/06/07 14:51:55 DEBUG : Using config file from "/home//.config/rclone/rclone.conf"
2023/06/07 14:51:55 DEBUG : Creating backend with remote "s3-standard:dr-storage-systems/test/"
2023/06/07 14:51:55 DEBUG : fs cache: renaming cache item "s3-standard:dr-storage-systems/test/" to be canonical "s3-standard:dr-storage-systems/test"
2023/06/07 14:51:55 DEBUG : S3 bucket dr-storage-systems path test: Waiting for checks to finish
2023/06/07 14:51:55 DEBUG : S3 bucket dr-storage-systems path test: Waiting for transfers to finish
2023/06/07 14:51:55 DEBUG : test-me.txt: md5 = d8e8fca2dc0f896fd7cb4cb0031ba249 OK
2023/06/07 14:51:55 INFO : test-me.txt: Copied (new)
2023/06/07 14:51:55 INFO :
Transferred: 5 B / 5 B, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.3s

2023/06/07 14:51:55 DEBUG : 5 go routines active

kapitainsky · June 7, 2023, 7:07pm

so can you try now copyto from the same dir where copy work?

I am not saying I know what the issue is but we have to gather some info:)

kapitainsky · June 7, 2023, 7:16pm

actually the plot thickens:

2023/06/07 13:05:06 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: 12DYE0NGV2TY4MKG, host id: B0NWeASwN+7ELt5DXA8sMjBX02gzKu2/+23DNpDAU8t0DxjrkhF/dXF30RgQ7t6VxewsoB3a3SY=

it looks like it is S3 AccessDenied

Ron · June 7, 2023, 7:33pm

I don't understand why attempts to copy a single file, using either copyto or copy, fail with AccessDenied, while copying the same file using a recursive rclone copy succeeds. Also aws cli succeeds copying the same file to the same path.

Here rclone copy, failing to copy the same file non-recursively with AccessDenied. This works if I copy the directory, not the file.

$ ./rclone -vvv copy ./testdir/test-me.txt s3-standard:dr-storage-systems/test/test-me.txt
2023/06/07 15:23:56 DEBUG : rclone: Version "v1.62.2" starting with parameters ["./rclone" "-vvv" "copy" "./testdir/test-me.txt" "s3-standard:dr-storage-systems/test/test-me.txt"]
2023/06/07 15:23:56 DEBUG : Creating backend with remote "./testdir/test-me.txt"
2023/06/07 15:23:56 DEBUG : Using config file from "/home//.config/rclone/rclone.conf"
2023/06/07 15:23:56 DEBUG : fs cache: adding new entry for parent of "./testdir/test-me.txt", "/home//testdir"
2023/06/07 15:23:56 DEBUG : Creating backend with remote "s3-standard:dr-storage-systems/test/test-me.txt"
2023/06/07 15:23:56 DEBUG : Resolving service "s3" region "us-east-1"
2023/06/07 15:23:56 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 15:23:57 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: CDDEK5FS2422B6DR, host id: 8DwM8IFHGet0Mtm0xvguVRAUdlbbPMJ2B7DqOFFRi7Osh40/kFyl/R2pfKIFhpKxH55uSBa5CvI=
2023/06/07 15:23:57 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: CDDEK5FS2422B6DR, host id: 8DwM8IFHGet0Mtm0xvguVRAUdlbbPMJ2B7DqOFFRi7Osh40/kFyl/R2pfKIFhpKxH55uSBa5CvI=
2023/06/07 15:23:57 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 15:23:57 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: CDDF0RTS8JR0J1J3, host id: USrcPf4J8Gl6irVY4YJ8YfD98W3M+34h0FWEFv3SeCQK0+8himuR7SeNULd9fc0FPcv9xn2mGhs=
2023/06/07 15:23:57 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: CDDF0RTS8JR0J1J3, host id: USrcPf4J8Gl6irVY4YJ8YfD98W3M+34h0FWEFv3SeCQK0+8himuR7SeNULd9fc0FPcv9xn2mGhs=
2023/06/07 15:23:57 DEBUG : test-me.txt: Need to transfer - File not found at Destination
2023/06/07 15:23:57 ERROR : test-me.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: CDD1BXMYSMS76SVW, host id: eUZYd3jQGXZ2jo2nEPzlM+zG+GreXBbk74gzVuuirhrzkPl0dYDbmuTj7M5VIVs81UYmz3RMnjA=
2023/06/07 15:23:57 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: CDD1BXMYSMS76SVW, host id: eUZYd3jQGXZ2jo2nEPzlM+zG+GreXBbk74gzVuuirhrzkPl0dYDbmuTj7M5VIVs81UYmz3RMnjA=
2023/06/07 15:23:57 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.3s

2023/06/07 15:23:57 DEBUG : 5 go routines active
2023/06/07 15:23:57 Failed to copy: AccessDenied: Access Denied
status code: 403, request id: CDD1BXMYSMS76SVW, host id: eUZYd3jQGXZ2jo2nEPzlM+zG+GreXBbk74gzVuuirhrzkPl0dYDbmuTj7M5VIVs81UYmz3RMnjA=

Here's aws cp successfully copying the same file to same bucket/path:

$ aws s3 cp ./testdir/test-me.txt s3://dr-storage-systems/test/test-me.txt
upload: testdir/test-me.txt to s3://dr-storage-systems/test/test-me.txt

$ aws s3 ls s3://ncbi-dr-storage-systems/test/
2023-06-05 10:42:19 0
2023-06-07 15:26:17 5 test-me.txt

kapitainsky · June 7, 2023, 7:36pm

can you try:

./rclone -vvv copyto /home/testdir/test-me.txt s3-standard:dr-storage-systems/test/test-me.txt

asdffdsa · June 7, 2023, 7:36pm

hello and welcome to the forum,

for a more detailed look, run the copyto with --dump=headers --retries=1
and as a test, might try --s3-no-check-bucket --s3-no-head

Ron · June 7, 2023, 7:46pm

adding --s3-no-check-bucket worked! Thank you. I'm not sure why it's required for copyto (or single files) and not required for a recursive copy, but this is good to know.

kapitainsky · June 7, 2023, 7:49pm

I am puzzled... too

asdffdsa · June 7, 2023, 7:56pm

welcome.
in my case for every rclone command, i always have to use that flag; due to the way i lock down s3 bucket policies.

if you run the copyto command twice, we can compare the output

--dump=headers --retries=1
--dump=headers --retries=1 --s3-no-check-bucket

ncw · June 8, 2023, 10:21am

If rclone lists the bucket then it knows the bucket is ok and doesn't need checking. This is what happens in rclone copy. However rclone copyto is just copying a single file so doesn't list the bucket and therefore needs to check the bucket is ok.

I could make everything use the list bucket method to check the bucket is ok but that would use extra transactions.

Perhaps --s3-no-list-bucket should be the default...

system · July 8, 2023, 10:22am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.