What is the problem you are having with rclone?
I am running rclone in a script. I would like my script to validate that the endpoint and path are valid before kicking off a copy of a large amount of files/objects, since the retries on each file/object take a while. I am attempting to do this by running "rclone lsd" before the copy. This works well in many cases. However, in the specific case of a valid S3 endpoint and invalid path as a non-existent bucket path made up of any number of slashes like "//", I get a list of all buckets at the endpoint. Looks like it is by design but I'd like to understand the details (is it only empty strings and slashes that do this?) and whether there is a better option for determining the validity of an endpoint path. My current idea is to keep using "lsd" for the validation, but strip the leading and trailing backward and forward slashes from the path string and check that it is not empty. Then I will use the non-empty stripped path for lsd and copy. Does this sound good?
Run the command 'rclone version' and share the full output of the command.
>rclone version
rclone v1.59.0
- os/version: Microsoft Windows 10 Enterprise 21H2 (64 bit)
- os/kernel: 10.0.19044.1826 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.3
- go/linking: static
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Datacore Swarm (S3 storage type)
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
>rclone lsd swarm-mb1-93://
-1 2022-06-03 15:53:18 -1 cl-source
-1 2022-06-03 17:25:55 -1 destination
-1 2022-06-29 14:55:02 -1 destination2
-1 2022-06-21 10:21:57 -1 hackathon
-1 2022-06-07 15:25:51 -1 smallbucket
-1 2022-06-16 15:33:30 -1 uniquebucket
The rclone config contents with secrets removed.
[swarm-mb1-93]
type = s3
access_key_id = <secret removed>
secret_access_key = <secret removed>
endpoint = http://domain.mb1-93.tx.caringo.com:8082
chunk_size = 100M
provider = Wasabi
A log from the command with the -vv
flag
>rclone lsd swarm-mb1-93:// -vv
2022/07/15 13:24:25 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "lsd" "swarm-mb1-93://" "-vv"]
2022/07/15 13:24:25 DEBUG : Creating backend with remote "swarm-mb1-93://"
2022/07/15 13:24:25 DEBUG : Using config file from "C:\\Users\\kmcmurtrey\\AppData\\Roaming\\rclone\\rclone.conf"
2022/07/15 13:24:25 DEBUG : fs cache: renaming cache item "swarm-mb1-93://" to be canonical "swarm-mb1-93:"
-1 2022-06-03 15:53:18 -1 cl-source
-1 2022-06-03 17:25:55 -1 destination
-1 2022-06-29 14:55:02 -1 destination2
-1 2022-06-21 10:21:57 -1 hackathon
-1 2022-06-07 15:25:51 -1 smallbucket
-1 2022-06-16 15:33:30 -1 uniquebucket
2022/07/15 13:24:26 DEBUG : 4 go routines active