Cloudsmith.io hosted content

Help and Support
1

What is the problem you are having with rclone?

I'm not able to sync content in a publicly available location hosted by cloudsmith.io
It appears the directory listing isn't available for rclone to gather the remote source's content. Interestingly enough I get a 404 if I try to list out the directory contents with curl. However, the files hosted after the URL in the rclone config are available to download using curl (with the full path) or via a browser.

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.0
- os/version: darwin 15.2 (64 bit)
- os/kernel: 24.2.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.23.4
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Cloudsmith

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync ISC-KEA: ~/tmp/isc-kea/ --log-file=/Users/kix/var/log/isc-kea.rclone.log -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[ISC-KEA]
type = http
url = https://dl.cloudsmith.io/public/isc/kea-2-4/rpm/el/8/x86_64/
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

[2025-01-27 10:11:05] 
2025/01/27 10:12:29 DEBUG : rclone: Version "v1.69.0" starting with parameters ["rclone" "sync" "ISC-KEA:" "/Users/kix/tmp/isc-kea/" "--log-file=/Users/kix/var/log/isc-kea.rclone.log" "-vv"]
2025/01/27 10:12:29 DEBUG : Creating backend with remote "ISC-KEA:"
2025/01/27 10:12:29 DEBUG : Using config file from "/Users/kix/.config/rclone/rclone.conf"
2025/01/27 10:12:29 DEBUG : Root: https://dl.cloudsmith.io/public/isc/kea-2-4/rpm/el/8/x86_64/
2025/01/27 10:12:29 DEBUG : Creating backend with remote "/Users/kix/tmp/isc-kea/"
2025/01/27 10:12:29 DEBUG : fs cache: renaming cache item "/Users/kix/tmp/isc-kea/" to be canonical "/Users/kix/tmp/isc-kea"
2025/01/27 10:12:30 ERROR : https://dl.cloudsmith.io/public/isc/kea-2-4/rpm/el/8/x86_64/: error reading source root directory: error listing "": directory not found
2025/01/27 10:12:30 DEBUG : Local file system at /Users/kix/tmp/isc-kea: Waiting for checks to finish
2025/01/27 10:12:30 DEBUG : Local file system at /Users/kix/tmp/isc-kea: Waiting for transfers to finish
2025/01/27 10:12:30 ERROR : Local file system at /Users/kix/tmp/isc-kea: not deleting files as there were IO errors
2025/01/27 10:12:30 ERROR : Local file system at /Users/kix/tmp/isc-kea: not deleting directories as there were IO errors
2025/01/27 10:12:30 ERROR : Attempt 1/3 failed with 1 errors and: error listing "": directory not found
2025/01/27 10:12:30 ERROR : https://dl.cloudsmith.io/public/isc/kea-2-4/rpm/el/8/x86_64/: error reading source root directory: error listing "": directory not found
2025/01/27 10:12:30 DEBUG : Local file system at /Users/kix/tmp/isc-kea: Waiting for checks to finish
2025/01/27 10:12:30 DEBUG : Local file system at /Users/kix/tmp/isc-kea: Waiting for transfers to finish
2025/01/27 10:12:30 ERROR : Local file system at /Users/kix/tmp/isc-kea: not deleting files as there were IO errors
2025/01/27 10:12:30 ERROR : Local file system at /Users/kix/tmp/isc-kea: not deleting directories as there were IO errors
2025/01/27 10:12:30 ERROR : Attempt 2/3 failed with 1 errors and: error listing "": directory not found
2025/01/27 10:12:30 ERROR : https://dl.cloudsmith.io/public/isc/kea-2-4/rpm/el/8/x86_64/: error reading source root directory: error listing "": directory not found
2025/01/27 10:12:30 DEBUG : Local file system at /Users/kix/tmp/isc-kea: Waiting for checks to finish
2025/01/27 10:12:30 DEBUG : Local file system at /Users/kix/tmp/isc-kea: Waiting for transfers to finish
2025/01/27 10:12:30 ERROR : Local file system at /Users/kix/tmp/isc-kea: not deleting files as there were IO errors
2025/01/27 10:12:30 ERROR : Local file system at /Users/kix/tmp/isc-kea: not deleting directories as there were IO errors
2025/01/27 10:12:30 ERROR : Attempt 3/3 failed with 1 errors and: error listing "": directory not found
2025/01/27 10:12:30 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.7s

2025/01/27 10:12:30 DEBUG : 5 go routines active
2025/01/27 10:12:30 NOTICE: Failed to sync: error listing "": directory not found
2 
$ curl https://dl.cloudsmith.io/public/isc/kea-2-4/rpm/el/8/x86_64/
<!DOCTYPE html>
    <html lang="en">
    <head>
      <meta charset="utf-8">
      <title>Not Found</title>
      <meta name="viewport" content="width=device-width, initial-scale=1.0">
    </head>
    <body>
      <h1>Not Found</h1>
      <p>
        Could not find the requested resource.
      </p>
    </body>
    </html>

This URL is not accessible.

The same when trying web browser:

image
image2798×1048 133 KB

So rclone fails as expected:

I am not sure what you expect from rclone here?

3

Thank you @kapitainsky, I appreciate you taking the time to read and respond to my post. Using curl, I am able to download a file past that directory location:

 curl -O https://dl.cloudsmith.io/public/isc/kea-2-4/rpm/el/8/x86_64/isc-kea-ctrl-agent-2.4.0-isc20230630120747.el8.x86_64.rpm                                                [2025-01-25 16:46:31] 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  161k  100  161k    0     0   543k      0 --:--:-- --:--:-- --:--:--  545k

ll isc-kea-ctrl-agent-2.4.0-isc20230630120747.el8.x86_64.rpm                                                                                                                 [2025-01-27 10:47:52] 
-rw-r--r--  1 ---  ----  165224 Jan 27 10:47 isc-kea-ctrl-agent-2.4.0-isc20230630120747.el8.x86_64.rpm

I understand this is more of a question for Cloudsmith (I've also reached out to ISC and Cloudsmith support). Posting here in case someone has experience copying/syncing content from Cloudsmith using rclone. Or perhaps there is an option in the utility that could be used to successfully gather a directory listing.

Thank you,
Enrique

4

rclone has no problem with doing it. But not when it is blocked and not available.

There is no magic built into rclone to guess what files URLs are - either website list them or you have to know them and then you can use rclone to download them one by one.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.