Setting up S3 remote

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I'm unable to run commands copy, move or lsd on AWS S3

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: darwin 12.7.4 (64 bit)

  • os/kernel: 21.6.0 (x86_64)

  • os/type: darwin

  • os/arch: amd64

  • go/version: go1.22.4

  • go/linking: dynamic

  • go/tags: cmount

rclone lsd aws-s3:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-east-1
endpoint = https://us-east-1.console.aws.amazon.com/
location_constraint = us-east-2
sse_kms_key_id = XXX
bucket_acl = private
acl = private
env_auth = true
server_side_encryption = aws:kms
storage_class = STANDARD

[drive]
type = drive
scope = drive
token = XXX
team_drive = 
root_folder_id = XXX

A log from the command that you were trying to run with the -vv flag

2024/06/23 18:06:08 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "lsd" "aws-s3:" "-vv"]
2024/06/23 18:06:08 DEBUG : Creating backend with remote "aws-s3:"
2024/06/23 18:06:08 DEBUG : Using config file from "/Users/beatrice/.config/rclone/rclone.conf"
2024/06/23 18:06:08 DEBUG : Resolving service "s3" region "us-east-1"
2024/06/23 18:06:58 ERROR : : error listing: SerializationError: failed to decode REST XML response
	status code: 200, request id: 7c07a541412523eddb735b9bc0657495
caused by: XML syntax error on line 16: invalid character entity & (no semicolon)
2024/06/23 18:06:58 DEBUG : 6 go routines active
2024/06/23 18:06:58 Failed to lsd with 2 errors: last error was: SerializationError: failed to decode REST XML response
	status code: 200, request id: 7c07a541412523eddb735b9bc0657495
caused by: XML syntax error on line 16: invalid character entity & (no semicolon)
2

welcome to the forum,

as a test, try something simple, get that working first.
for example, here is a working remote for aws

[aws]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
3

Sorry, what command are you suggesting I run?

4

the same command that you ran before.

5

I'm running rclone lsd aws: and here's the response

2024/06/23 18:33:11 ERROR : : error listing: SerializationError: failed to decode REST XML response
	status code: 200, request id: 0bd418525a632c87e271128d75953bfd
caused by: XML syntax error on line 16: invalid character entity & (no semicolon)
2024/06/23 18:33:11 Failed to lsd with 2 errors: last error was: SerializationError: failed to decode REST XML response
	status code: 200, request id: 0bd418525a632c87e271128d75953bfd
caused by: XML syntax error on line 16: invalid character entity & (no semicolon)
6

need to post a full debug log, not snippets, ok?

i already shared an example of a working aws remote.
make your remote look like my very simple, example remote.

7

That's worked now, thank you and command lsd is returning the correct file directory. But when I try to copy from Google Drive, here's the response.

remote config:

[aws]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX

[drive]
type = drive
scope = drive
token = XXX
team_drive = 
root_folder_id = XXX

command rclone lsd aws:

2024/06/23 18:46:38 ERROR : Google drive root 'path': error reading source root directory: directory not found

2024/06/23 18:46:38 ERROR : Attempt 1/3 failed with 1 errors and: directory not found

2024/06/23 18:46:38 ERROR : Google drive root 'path': error reading source root directory: directory not found

2024/06/23 18:46:38 ERROR : Attempt 2/3 failed with 1 errors and: directory not found

2024/06/23 18:46:39 ERROR : Google drive root 'path': error reading source root directory: directory not found

2024/06/23 18:46:39 ERROR : Attempt 3/3 failed with 1 errors and: directory not found

2024/06/23 18:46:39 Failed to copy: directory not found

8

need to run the command with -vv, need to post the full output, not snippets, ok?

and for testing commands, use --dry-run

9

Oops. sorry again. Here it is...

rclone copy drive:path aws:path --dry-run -vv
2024/06/23 18:55:40 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "copy" "drive:path" "aws:path" "--dry-run" "-vv"]
2024/06/23 18:55:40 DEBUG : Creating backend with remote "drive:path"
2024/06/23 18:55:40 DEBUG : Using config file from "/Users/beatrice/.config/rclone/rclone.conf"
2024/06/23 18:55:42 DEBUG : Creating backend with remote "aws:path"
2024/06/23 18:55:43 ERROR : Google drive root 'path': error reading source root directory: directory not found
2024/06/23 18:55:43 DEBUG : S3 bucket path: Waiting for checks to finish
2024/06/23 18:55:43 DEBUG : S3 bucket path: Waiting for transfers to finish
2024/06/23 18:55:43 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2024/06/23 18:55:43 ERROR : Google drive root 'path': error reading source root directory: directory not found
2024/06/23 18:55:43 DEBUG : S3 bucket path: Waiting for checks to finish
2024/06/23 18:55:43 DEBUG : S3 bucket path: Waiting for transfers to finish
2024/06/23 18:55:43 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2024/06/23 18:55:44 ERROR : Google drive root 'path': error reading source root directory: directory not found
2024/06/23 18:55:44 DEBUG : S3 bucket path: Waiting for checks to finish
2024/06/23 18:55:44 DEBUG : S3 bucket path: Waiting for transfers to finish
2024/06/23 18:55:44 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2024/06/23 18:55:44 NOTICE: 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.5s

2024/06/23 18:55:44 DEBUG : 9 go routines active
2024/06/23 18:55:44 Failed to copy: directory not found
10

that is the error. need to fix that.

11

Okay, I'd reset the directory and try again

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.