What is the problem you are having with rclone?
The --s3-profile flag is failing (hang then dump a fail) in rclone copy operations -- again -- in newer beta, and in a whole new way. This was fixed in past
Run the command 'rclone version' and share the full output of the command.
$ rclone-beta version
rclone v1.60.0-beta.6422.85eb9776b
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.76.1.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone-beta copy ~/testbed/testfile s3-standard:x20220908-rpc-1 --s3-profile dr_systems_profile
The rclone config contents with secrets removed.
[s3-standard]
type = s3
provider = AWS
env_auth = true
region = us-east-1
endpoint = https://bucket.vpce-MYVPCE.s3.us-east-1.vpce.amazonaws.com
storage_class = STANDARD
A log from the command with the -vv flag
$ rclone-beta copy ~/testbed/testfile s3-standard:x20220908-rpc-1 --s3-profile dr_systems_profile -vv
2022/09/08 15:06:54 DEBUG : rclone: Version "v1.60.0-beta.6422.85eb9776b" starting with parameters ["/mypath/rclone-beta/rclone" "copy" "/home/pattcornerri/testbed/testfile" "s3-standard:x20220908-rpc-1" "--s3-profile" "dr_systems_profile" "-vv"]
2022/09/08 15:06:54 DEBUG : Creating backend with remote "/...testfile"
2022/09/08 15:06:54 DEBUG : Using config file from "/...rclone.conf"
2022/09/08 15:06:54 DEBUG : fs cache: adding new entry for parent of "/..., "/.../testbed"
2022/09/08 15:06:54 DEBUG : Creating backend with remote "s3-standard:x20220908-rpc-1"
2022/09/08 15:06:54 DEBUG : s3-standard: detected overridden config - adding "{ypUXU}" suffix to name
2022/09/08 15:06:54 DEBUG : fs cache: renaming cache item "s3-standard:x20220908-rpc-1" to be canonical "s3-standard{ypUXU}:x20220908-rpc-1"
2022/09/08 15:07:37 ERROR : Attempt 1/3 failed with 1 errors and: SerializationError: failed to unmarshal error message
status code: 400, request id:
caused by: UnmarshalError: failed to unmarshal error message
00000000 3c 3f 78 6d 6c 20 76 65 72 73 69 6f 6e 3d 22 31 |<?xml version="1|
00000010 2e 30 22 20 65 6e 63 6f 64 69 6e 67 3d 22 55 54 |.0" encoding="UT|
00000020 46 2d 38 22 3f 3e 0a 3c 45 72 72 6f 72 3e 3c 43 |F-8"?>.<Error><C|
00000030 6f 64 65 3e 49 6e 76 61 6c 69 64 52 65 71 75 65 |ode>InvalidReque|
00000040 73 74 3c 2f 43 6f 64 65 3e 3c 4d 65 73 73 61 67 |st</Code><Messag|
00000050 65 3e 4d 69 73 73 69 6e 67 20 72 65 71 75 69 72 |e>Missing requir|
00000060 65 64 20 68 65 61 64 65 72 20 66 6f 72 20 74 68 |ed header for th|
00000070 69 73 20 72 65 71 75 65 73 74 3a 20 78 2d 61 6d |is request: x-am|
00000080 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 |z-content-sha256|
00000090 3c 2f 4d 65 73 73 61 67 65 3e 3c 52 65 71 75 65 |</Message><Reque|
000000a0 73 74 49 64 3e 51 35 39 58 33 38 56 47 54 43 35 |stId>Q59X38VGTC5|
000000b0 4d 47 52 52 45 3c 2f 52 65 71 75 65 73 74 49 64 |MGRRE</RequestId|
000000c0 3e 3c 48 6f 73 74 49 64 3e 52 78 75 4c 69 34 32 |><HostId>RxuLi42|
000000d0 63 67 5a 6f 52 2b 35 7a 77 52 39 42 2b 70 30 78 |cgZoR+5zwR9B+p0x|
000000e0 54 68 47 45 33 78 59 33 57 63 6a 46 74 48 39 69 |ThGE3xY3WcjFtH9i|
000000f0 47 47 70 46 6a 59 70 76 64 48 67 64 5a 41 62 50 |GGpFjYpvdHgdZAbP|
00000100 58 6c 4e 4a 65 59 36 48 5a 44 45 6e 61 39 2b 6e |XlNJeY6HZDEna9+n|
00000110 68 6c 44 49 3d 3c 2f 48 6f 73 74 49 64 3e 3c 2f |hlDI=</HostId></|
00000120 45 72 72 6f 72 3e |Error>|
caused by: unknown error response tag, {{ Error} []}
The bug report in GitHub is at `--s3-profile` option does not work with `role_arn` when AWS_PROFILE does · Issue #5468 · rclone/rclone · GitHub
What works
CLI works fine with same profile:
$ aws s3 cp ~/testbed/testfile s3://x20220908-rpc-1 --profile dr_systems_profile
upload: testbed/testfile to s3://x20220908-rpc-1/testfile
The Profile
[dr_systems_profile]
role_arn = arn:aws:iam::XXXXXXXXXXXX:role/SOMEGOODROLE
credential_source = Environment
Other
This was fixed a couple times in past, but appears broken now. The fail is atypical ... the dump you see in log appears after about a 30 second hang.
Older versions simply fail with 403, but they probably predate the first round of fixes