Can I skip the connection when the combine contains unavailable remotes?

maiya · October 31, 2023, 1:03pm

What is the problem you are having with rclone?

I have a combine where I added some FTP remotes, and I need to serve it with a server. However, these FTP remotes are not guaranteed to be online at all times. If one of them is offline, the entire combine becomes unavailable.
I would like to avoid establishing connections for unused items in the combine when using it., so that I can use other available items. Is there a way to achieve this?

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
os/kernel: 10.0.19045.3324 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.21.1
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

ftp

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone lsf index:
rclone serve ftp index:
rclone serve webdav index:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[@cwd]
type = alias
remote = .

[ftp1]
type = ftp
host = XXX
user = XXX
port = 2121
pass = XXX

[ftp2]
type = ftp
host = XXX
user = XXX
port = 2121
pass = XXX

[index]
type = combine
upstreams = ftp1=ftp1: ftp2=ftp2: @cwd=@cwd
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the `-vv` flag

2023/10/31 12:52:04 DEBUG : rclone: Version "1.64.0-termux" starting with parameters ["rclone" "--config" "exp.conf" "-vv" "serve" "webdav" "index:"]
2023/10/31 12:52:04 DEBUG : Creating backend with remote "index:"
2023/10/31 12:52:04 DEBUG : Using config file from "/data/data/com.termux/files/home/exp.conf"
2023/10/31 12:52:04 DEBUG : Creating backend with remote "@cwd"
2023/10/31 12:52:04 DEBUG : Creating backend with remote "ftp2:"
2023/10/31 12:52:04 NOTICE: "@cwd" refers to a local folder, use "@cwd:" to refer to your remote or "./@cwd" to hide this warning
2023/10/31 12:52:04 DEBUG : Creating backend with remote "ftp1:"
2023/10/31 12:52:04 DEBUG : fs cache: renaming cache item "@cwd" to be canonical "/data/data/com.termux/files/home/@cwd"
2023/10/31 12:52:04 DEBUG : ftp://192.168.31.31:2121: Connecting to FTP server
2023/10/31 12:52:04 DEBUG : ftp://192.168.31.31:2121: dial("tcp","192.168.31.31:2121")
2023/10/31 12:52:04 DEBUG : ftp://192.168.31.30:2121: Connecting to FTP server
2023/10/31 12:52:04 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:52:04 DEBUG : ftp://192.168.31.31:2121: > dial: conn=*fshttp.timeoutConn, err=<nil>
2023/10/31 12:52:11 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:52:11 DEBUG : pacer: low level retry 1/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:52:11 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2023/10/31 12:52:11 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:52:18 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:52:18 DEBUG : pacer: low level retry 2/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:52:18 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2023/10/31 12:52:18 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:52:25 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:52:25 DEBUG : pacer: low level retry 3/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:52:25 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2023/10/31 12:52:25 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:52:32 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:52:32 DEBUG : pacer: low level retry 4/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:52:32 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2023/10/31 12:52:32 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:52:39 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:52:39 DEBUG : pacer: low level retry 5/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:52:39 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2023/10/31 12:52:39 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:52:46 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:52:46 DEBUG : pacer: low level retry 6/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:52:46 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2023/10/31 12:52:46 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:52:53 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:52:53 DEBUG : pacer: low level retry 7/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:52:53 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2023/10/31 12:52:53 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:53:00 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:53:00 DEBUG : pacer: low level retry 8/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:53:00 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/10/31 12:53:00 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:53:04 DEBUG : ftp://192.168.31.31:2121: closing 1 unused connections
2023/10/31 12:53:08 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:53:08 DEBUG : pacer: low level retry 9/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:53:08 DEBUG : ftp://192.168.31.30:2121: dial("tcp","192.168.31.30:2121")
2023/10/31 12:53:15 DEBUG : ftp://192.168.31.30:2121: > dial: conn=<nil>, err=dial tcp 192.168.31.30:2121: connect: connection timed out
2023/10/31 12:53:15 DEBUG : pacer: low level retry 10/10 (error dial tcp 192.168.31.30:2121: connect: connection timed out)
2023/10/31 12:53:15 Failed to create file system for "index:": failed to create upstream "ftp1:": NewFs: failed to make FTP connection to "192.168.31.30:2121": dial tcp 192.168.31.30:2121: connect: connection timed out

kapitainsky · October 31, 2023, 2:27pm

It is not possible with rclone on its own.

As a workaround you can create multiple config files - with or without ftp server and start your rclone serve with whichever configuration is fully working - you will need some script wrapper to "ping" ftp servers, determine which one is live or not and then apply appropriate config.

system · November 30, 2023, 2:27pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.