Endpoint validation with lsd

What is the problem you are having with rclone?

I am running rclone in a script. I would like my script to validate that the endpoint and path are valid before kicking off a copy of a large amount of files/objects, since the retries on each file/object take a while. I am attempting to do this by running "rclone lsd" before the copy. This works well in many cases. However, in the specific case of a valid S3 endpoint and invalid path as a non-existent bucket path made up of any number of slashes like "//", I get a list of all buckets at the endpoint. Looks like it is by design but I'd like to understand the details (is it only empty strings and slashes that do this?) and whether there is a better option for determining the validity of an endpoint path. My current idea is to keep using "lsd" for the validation, but strip the leading and trailing backward and forward slashes from the path string and check that it is not empty. Then I will use the non-empty stripped path for lsd and copy. Does this sound good?

Run the command 'rclone version' and share the full output of the command.

>rclone version
rclone v1.59.0
- os/version: Microsoft Windows 10 Enterprise 21H2 (64 bit)
- os/kernel: 10.0.19044.1826 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Datacore Swarm (S3 storage type)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

>rclone lsd swarm-mb1-93://
          -1 2022-06-03 15:53:18        -1 cl-source
          -1 2022-06-03 17:25:55        -1 destination
          -1 2022-06-29 14:55:02        -1 destination2
          -1 2022-06-21 10:21:57        -1 hackathon
          -1 2022-06-07 15:25:51        -1 smallbucket
          -1 2022-06-16 15:33:30        -1 uniquebucket

The rclone config contents with secrets removed.

[swarm-mb1-93]
type = s3
access_key_id = <secret removed>
secret_access_key = <secret removed>
endpoint = http://domain.mb1-93.tx.caringo.com:8082
chunk_size = 100M
provider = Wasabi

A log from the command with the -vv flag

>rclone lsd swarm-mb1-93:// -vv
2022/07/15 13:24:25 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "lsd" "swarm-mb1-93://" "-vv"]
2022/07/15 13:24:25 DEBUG : Creating backend with remote "swarm-mb1-93://"
2022/07/15 13:24:25 DEBUG : Using config file from "C:\\Users\\kmcmurtrey\\AppData\\Roaming\\rclone\\rclone.conf"
2022/07/15 13:24:25 DEBUG : fs cache: renaming cache item "swarm-mb1-93://" to be canonical "swarm-mb1-93:"
          -1 2022-06-03 15:53:18        -1 cl-source
          -1 2022-06-03 17:25:55        -1 destination
          -1 2022-06-29 14:55:02        -1 destination2
          -1 2022-06-21 10:21:57        -1 hackathon
          -1 2022-06-07 15:25:51        -1 smallbucket
          -1 2022-06-16 15:33:30        -1 uniquebucket
2022/07/15 13:24:26 DEBUG : 4 go routines active

Rclone ignores leading and trailing slashes on most backends (exceptions Dropbox business and sftp) so stripping them is a good idea.

If an s3 remote has no bucket it will list all buckets of it has permission.

1 Like

Thank you! I will strip the leading and trailing slashes and whitespace before using it for lsd and copy. If it is empty after the stripping (no bucket), that's an invalid name. If not empty, lsd should be a great way to further check the bucket name is not invalid (spelled wrong, no permission, etc) before proceeding to copy.

(And if type is dropbox or sftp, skip trimming the slashes)

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.