Understanding Concurrent S3 Credential Update and File Operations in Rclone with "Backend Set"

What is the problem you are having with rclone?

I'm encountering a scenario with Rclone related to the Amazon S3 remote. My workflow involves updating the credentials of an S3 remote dynamically using Rclone's backend set command. This process takes place while other operations, particularly file transfers like copying, are concurrently active on the same remote.

My primary concern is understanding the implications of this concurrent operation:

  • How does Rclone manage an ongoing file operation (e.g., copying a file) when the credentials of the S3 remote are updated in the middle of the process?

  • Is there a risk of interruption or failure for the file operation due to the credential update?

  • What would be the expected behavior or error handling mechanism from Rclone in such a situation?

I'm looking to understand whether the file operation would continue using the old credentials until its completion, or if it would be halted or fail.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

  • os/version: Microsoft Windows Server 2016 Datacenter 1607 (64 bit)

  • os/kernel: 10.0.14393.6167 (x86_64)

  • os/type: windows

  • os/arch: amd64

  • go/version: go1.21.3

  • go/linking: static

  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)


rclone rcd --rc-addr=localhost:$rcdPort --rc-user=$User --rc-pass=$RandomPassword --log-level=$debug_level --log-file=$rcdLogPath

rclone rc mount/mount fs=s3:${S3Mount} mountPoint=${LocalMount} --rc-addr=localhost:$rcdPort --rc-user=$User --rc-pass=$RandomPassword

rclone rc backend/command command=set fs=${S3Mount} -o access_key_id=$accessKeyId -o secret_access_key=$secretAccessKey -o session_token=$sessionToken --rc-addr=localhost:$rcdPort --rc-user=$User --rc-pass=$RandomPassword

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.


[s3]

type = s3

provider = AWS

env_auth = false

access_key_id = <- Not redacted is actual empty value

secret_access_key = <- Not redacted is actual empty value

session_token = <- Not redacted is actual empty value

region = eu-central-1

A log from the command that you were trying to run with the -vv flag


n/a

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.