Combine: Inconsistent handling of (/) for HTTP upstream(s)

What is the problem you are having with rclone?

Given a remote xx: with the http backend, a lsf command to a path has the same output with or without a trailing slash provided to the command. Given a remote zz: with the combine backend and an upstream path mapped to the xx: remote, a lsf command to a mapped path has incorrect results when the trailing slash is omitted from the command argument.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0
- os/version: ubuntu 22.10 (64 bit)
- os/kernel: 5.19.0-46-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

HTTP, combine

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsf remote:ubuntu-releases/23.04

The rclone config contents with secrets removed.

All commands configured by environment variables as follows.

$ RCLONE_CONFIG_XX_TYPE=http \
  RCLONE_CONFIG_XX_URL=https://ftp.osuosl.org/pub \
  RCLONE_CONFIG_YY_TYPE=http \
  RCLONE_CONFIG_YY_URL=https://ftp2.osuosl.org/pub RCLONE_CONFIG_ZZ_TYPE=combine \
  RCLONE_CONFIG_ZZ_UPSTREAMS="ubuntu-releases=xx:ubuntu-releases/" \
  <command>

For completeness, user rclone.conf file:

[B2]
type = b2
account = ...
key = ...

[shr...]
type = b2
account = ...
key = ...

[son...]
type = http
url = ...

[uoc]
type = http
url = ...

[moz...]
type = http
url = ...

[ani...]
type = s3
provider = Other
env_auth = false
access_key_id = ...
secret_access_key = ...
endpoint = ...
acl = public-read

[xmi...]
type = ftp
host = ...
user = anonymous
pass = ...

[fun...]
type = ftp
host = ...
user = anonymous
pass = ...

[git...]
type = b2
account = ...
key = ...
hard_delete = true

[lfs...]
type = http
url = ...
headers = Referer,...,Sec-Fetch-Site,same-origin,Sec-Fetch-Mode,navigate,Sec-Fetch-Dest,document,Sec-Fetch-User,?1,Sec-GPC,1
no_head = true

[dri...]
type = drive
client_id = ...
client_secret = ...
export_formats = odt,ods,odp,svg,pdf,docx
acknowledge_abuse = true
token = ...
team_drive = 

[osl1]
type = http
url = https://ftp.osuosl.org/pub

[osl2]
type = http
url = https://ftp2.osuosl.org/pub

[osl]
type = combine
upstreams = 1060=osl1:1060/ 1060.org=osl1:1060.org/ CPAN=osl1:CPAN/ almalinux=osl1:almalinux/ apache=osl1:apache/ archlinux=osl1:archlinux/ blfs=osl1:blfs/ centos=osl1:centos/ centos-altarch=osl1:centos-altarch/ centos-stream=osl1:centos-stream/ cinc=osl1:cinc/ cpan=osl1:cpan/ cran=osl1:cran/ cryoem=osl1:cryoem/ cygwin=osl1:cygwin/ debian=osl1:debian/ debian-cdimage=osl1:debian-cdimage/ deluge=osl1:deluge/ elrepo=osl1:elrepo/ faif-oggcast=osl1:faif-oggcast/ fedora=osl1:fedora/ fedora-epel=osl1:fedora-epel/ finnix=osl1:finnix/ fosdem=osl1:fosdem/ freedroid=osl1:freedroid/ ftp.debian.org=osl1:ftp.debian.org/ gentoo=osl1:gentoo/ gnome=osl1:gnome/ haiku=osl1:haiku/ jenkins=osl1:jenkins/ kernel=osl1:kernel/ lfs=osl1:lfs/ lfs-livecd=osl1:lfs-livecd/ libpng=osl1:libpng/ manulix=osl1:manulix/ mariadb=osl1:mariadb/ midnightcommander=osl1:midnightcommander/ msys2=osl1:msys2/ musescore=osl1:musescore/ musescore-nightlies=osl1:musescore-nightlies/ musicbrainz=osl1:musicbrainz/ mutt=osl1:mutt/ mxlinux-iso=osl1:mxlinux-iso/ netwinder=osl1:netwinder/ open-ce=osl1:open-ce/ open-lldp=osl1:open-lldp/ openstreetmap=osl1:openstreetmap/ osgeo=osl1:osgeo/ osl=osl1:osl/ pentoo=osl1:pentoo/ php=osl1:php/ plan9=osl1:plan9/ portablepython=osl1:portablepython/ ros=osl1:ros/ ros-shadow-fixed=osl1:ros-shadow-fixed/ ros-ubuntu=osl1:ros-ubuntu/ ros2=osl1:ros2/ ros2-testing=osl1:ros2-testing/ rpm=osl1:rpm/ sftpgo=osl1:sftpgo/ slackware=osl1:slackware/ slackware-iso=osl1:slackware-iso/ tails=osl1:tails/ tdf=osl1:tdf/ trk=osl1:trk/ ubuntu=osl1:ubuntu/ ubuntu-releases=osl1:ubuntu-releases/ videolan=osl1:videolan/ xbmc=osl1:xbmc/ xiph=osl1:xiph/ aqsis=osl2:aqsis/ clfs=osl2:clfs/ cybercomplex=osl2:cybercomplex/ eclipse=osl2:eclipse/ ganeti-rpm=osl2:ganeti-rpm/ grml=osl2:grml/ gsb=osl2:gsb/ hackage=osl2:hackage/ manjaro-arm=osl2:manjaro-arm/ monitoring-plugins=osl2:monitoring-plugins/ mxlinux=osl2:mxlinux/ mythtv=osl2:mythtv/ netbeans=osl2:netbeans/ openefs=osl2:openefs/ openvz=osl2:openvz/ osmc=osl2:osmc/ parrot=osl2:parrot/ parrotos=osl2:parrotos/ ppc64el=osl2:ppc64el/ qemu=osl2:qemu/ rebornos=osl2:rebornos/ replicant=osl2:replicant/ salix=osl2:salix/ teachingopensource=osl2:teachingopensource/ trisquel=osl2:trisquel/

A log from the command with the -vv flag

Redundant repetition of environment variables has been trimmed. All commands were run prefixed by the environment assignments above.

Listed below in the order: (1) HTTP/trailing slash, (2) combine/trailing slash, (3) HTTP/no trailing slash, (4) combine/no trailing slash.

(1) HTTP (trailing slash)

$ rclone lsf -vv xx:ubuntu-releases/23.04/
<7>DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "lsf" "-vv" "xx:ubuntu-releases/23.04/"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "xx:ubuntu-releases/23.04/"
<7>DEBUG : Using config file from "/home/maxwell/.config/rclone/rclone.conf"
<7>DEBUG : Setting type="http" for "xx" from environment variable RCLONE_CONFIG_XX_TYPE
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : xx: detected overridden config - adding "{WCD74}" suffix to name
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : Root: https://ftp.osuosl.org/pub/ubuntu-releases/23.04/
<7>DEBUG : fs cache: renaming cache item "xx:ubuntu-releases/23.04/" to be canonical "xx{WCD74}:ubuntu-releases/23.04/"
FOOTER.html
HEADER.html
SHA256SUMS
SHA256SUMS.gpg
netboot/
ubuntu-23.04-desktop-amd64.iso
ubuntu-23.04-desktop-amd64.iso.torrent
ubuntu-23.04-desktop-amd64.iso.zsync
ubuntu-23.04-desktop-amd64.list
ubuntu-23.04-desktop-amd64.manifest
ubuntu-23.04-live-server-amd64.iso
ubuntu-23.04-live-server-amd64.iso.torrent
ubuntu-23.04-live-server-amd64.iso.zsync
ubuntu-23.04-live-server-amd64.list
ubuntu-23.04-live-server-amd64.manifest
ubuntu-23.04-netboot-amd64.tar.gz
<7>DEBUG : 34 go routines active

(2) combine (trailing slash)

$ rclone lsf -vv zz:ubuntu-releases/23.04/
<7>DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "lsf" "-vv" "zz:ubuntu-releases/23.04/"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "zz:ubuntu-releases/23.04/"
<7>DEBUG : Using config file from "/home/maxwell/.config/rclone/rclone.conf"
<7>DEBUG : Setting type="combine" for "zz" from environment variable RCLONE_CONFIG_ZZ_TYPE
<7>DEBUG : Setting upstreams="ubuntu-releases=xx:ubuntu-releases/" for "zz" from environment variable RCLONE_CONFIG_ZZ_UPSTREAMS
<7>DEBUG : zz: detected overridden config - adding "{Mo6oa}" suffix to name
<7>DEBUG : Setting upstreams="ubuntu-releases=xx:ubuntu-releases/" for "zz" from environment variable RCLONE_CONFIG_ZZ_UPSTREAMS
<7>DEBUG : Creating backend with remote "xx:ubuntu-releases/"
<7>DEBUG : Setting type="http" for "xx" from environment variable RCLONE_CONFIG_XX_TYPE
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : xx: detected overridden config - adding "{WCD74}" suffix to name
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : Root: https://ftp.osuosl.org/pub/ubuntu-releases/
<7>DEBUG : fs cache: renaming cache item "xx:ubuntu-releases/" to be canonical "xx{WCD74}:ubuntu-releases/"
<7>DEBUG : fs cache: renaming cache item "zz:ubuntu-releases/23.04/" to be canonical "zz{Mo6oa}:ubuntu-releases/23.04"
FOOTER.html
HEADER.html
SHA256SUMS
SHA256SUMS.gpg
netboot/
ubuntu-23.04-desktop-amd64.iso
ubuntu-23.04-desktop-amd64.iso.torrent
ubuntu-23.04-desktop-amd64.iso.zsync
ubuntu-23.04-desktop-amd64.list
ubuntu-23.04-desktop-amd64.manifest
ubuntu-23.04-live-server-amd64.iso
ubuntu-23.04-live-server-amd64.iso.torrent
ubuntu-23.04-live-server-amd64.iso.zsync
ubuntu-23.04-live-server-amd64.list
ubuntu-23.04-live-server-amd64.manifest
ubuntu-23.04-netboot-amd64.tar.gz
<7>DEBUG : 32 go routines active

(3) HTTP (no trailing slash)

$ rclone lsf -vv xx:ubuntu-releases/23.04
<7>DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "lsf" "-vv" "xx:ubuntu-releases/23.04"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "xx:ubuntu-releases/23.04"
<7>DEBUG : Using config file from "/home/maxwell/.config/rclone/rclone.conf"
<7>DEBUG : Setting type="http" for "xx" from environment variable RCLONE_CONFIG_XX_TYPE
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : xx: detected overridden config - adding "{WCD74}" suffix to name
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : Assuming path is a directory as HEAD response is redirect (301 Moved Permanently) to a path that ends with '/': https://ftp.osuosl.org/pub/ubuntu-releases/23.04/
<7>DEBUG : To avoid the initial HEAD request add a trailing '/' to the path
<7>DEBUG : Root: https://ftp.osuosl.org/pub/ubuntu-releases/23.04/
<7>DEBUG : fs cache: renaming cache item "xx:ubuntu-releases/23.04" to be canonical "xx{WCD74}:ubuntu-releases/23.04"
FOOTER.html
HEADER.html
SHA256SUMS
SHA256SUMS.gpg
netboot/
ubuntu-23.04-desktop-amd64.iso
ubuntu-23.04-desktop-amd64.iso.torrent
ubuntu-23.04-desktop-amd64.iso.zsync
ubuntu-23.04-desktop-amd64.list
ubuntu-23.04-desktop-amd64.manifest
ubuntu-23.04-live-server-amd64.iso
ubuntu-23.04-live-server-amd64.iso.torrent
ubuntu-23.04-live-server-amd64.iso.zsync
ubuntu-23.04-live-server-amd64.list
ubuntu-23.04-live-server-amd64.manifest
ubuntu-23.04-netboot-amd64.tar.gz
<7>DEBUG : 38 go routines active

(4) combine (no trailing slash)

$ rclone lsf -vv zz:ubuntu-releases/23.04
<7>DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "lsf" "-vv" "zz:ubuntu-releases/23.04"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "zz:ubuntu-releases/23.04"
<7>DEBUG : Using config file from "/home/maxwell/.config/rclone/rclone.conf"
<7>DEBUG : Setting type="combine" for "zz" from environment variable RCLONE_CONFIG_ZZ_TYPE
<7>DEBUG : Setting upstreams="ubuntu-releases=xx:ubuntu-releases/" for "zz" from environment variable RCLONE_CONFIG_ZZ_UPSTREAMS
<7>DEBUG : zz: detected overridden config - adding "{Mo6oa}" suffix to name
<7>DEBUG : Setting upstreams="ubuntu-releases=xx:ubuntu-releases/" for "zz" from environment variable RCLONE_CONFIG_ZZ_UPSTREAMS
<7>DEBUG : Creating backend with remote "xx:ubuntu-releases/"
<7>DEBUG : Setting type="http" for "xx" from environment variable RCLONE_CONFIG_XX_TYPE
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : xx: detected overridden config - adding "{WCD74}" suffix to name
<7>DEBUG : Setting url="https://ftp.osuosl.org/pub" for "xx" from environment variable RCLONE_CONFIG_XX_URL
<7>DEBUG : Root: https://ftp.osuosl.org/pub/ubuntu-releases/
<7>DEBUG : fs cache: renaming cache item "xx:ubuntu-releases/" to be canonical "xx{WCD74}:ubuntu-releases/"
<7>DEBUG : fs cache: adding new entry for parent of "zz:ubuntu-releases/23.04", "zz{Mo6oa}:ubuntu-releases"
<7>DEBUG : 14.04.6: Excluded
<7>DEBUG : 14.04: Excluded
<7>DEBUG : 16.04.7: Excluded
<7>DEBUG : 16.04: Excluded
<7>DEBUG : 18.04.6: Excluded
<7>DEBUG : 18.04: Excluded
<7>DEBUG : 20.04.6: Excluded
<7>DEBUG : 20.04: Excluded
<7>DEBUG : 22.04.2: Excluded
<7>DEBUG : 22.04: Excluded
<7>DEBUG : 22.10: Excluded
<7>DEBUG : 23.04: Excluded
<7>DEBUG : bionic: Excluded
<7>DEBUG : cdicons: Excluded
<7>DEBUG : focal: Excluded
<7>DEBUG : include: Excluded
<7>DEBUG : jammy: Excluded
<7>DEBUG : kinetic: Excluded
<7>DEBUG : lunar: Excluded
<7>DEBUG : project: Excluded
<7>DEBUG : releases: Excluded
<7>DEBUG : streams: Excluded
<7>DEBUG : trusty: Excluded
<7>DEBUG : xenial: Excluded
<7>DEBUG : FOOTER.html: Excluded
<7>DEBUG : HEADER.html: Excluded
<7>DEBUG : favicon.ico: Excluded
<7>DEBUG : robots.txt: Excluded
<7>DEBUG : 12 go routines active

I had not been familiar with the union backend before, but it works nicely around this issue in my case.

RCLONE_CONFIG_XX_TYPE=http RCLONE_CONFIG_XX_URL=https://ftp.osuosl.org/pub \
RCLONE_CONFIG_YY_TYPE=http RCLONE_CONFIG_YY_URL=https://ftp2.osuosl.org/pub \
RCLONE_CONFIG_ZZ_TYPE=union RCLONE_CONFIG_ZZ_UPSTREAMS="xx::ro yy::ro" \
RCLONE_CONFIG_ZZ_SEARCH_POLICY=epall \
    rclone lsf zz:ubuntu-releases/23.04
FOOTER.html
HEADER.html
SHA256SUMS
SHA256SUMS.gpg
netboot/
ubuntu-23.04-desktop-amd64.iso
ubuntu-23.04-desktop-amd64.iso.torrent
ubuntu-23.04-desktop-amd64.iso.zsync
ubuntu-23.04-desktop-amd64.list
ubuntu-23.04-desktop-amd64.manifest
ubuntu-23.04-live-server-amd64.iso
ubuntu-23.04-live-server-amd64.iso.torrent
ubuntu-23.04-live-server-amd64.iso.zsync
ubuntu-23.04-live-server-amd64.list
ubuntu-23.04-live-server-amd64.manifest
ubuntu-23.04-netboot-amd64.tar.gz

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.