I keep failing to connect to a specific sftp

[Boaz_Fishman]

Hi all.

I've been using rclone in order to copy files from multiple client SFTP and until now it worked perfectly.
Our new client got me perplexed and on top of it, every 3 wrong tries it locks my account

The command I'm using is:

rclone copy client_sftp:Inbox client_s3:prefix --log-level DEBUG --use-json-log --transfers 16 --config rclone.conf --sftp-use-insecure-cipher --sftp-user username --sftp-pass pass --sftp-host mftprod.host.com

the logs are:

{"level":"debug","msg":"Version \"v1.56.2\" starting with parameters [\"rclone\" REDACTED ","object":"rclone","objectType":"string","source":"cmd/cmd.go:408","time":"2023-02-07T23:31:14.123196+02:00"}
{"level":"debug","msg":"Creating backend with remote \"REDACTED\"","source":"fs/newfs.go:28","time":"2023-02-07T23:31:14.123474+02:00"}
{"level":"debug","msg":"Using config file from \"/REDACTED/rclone.conf\"","source":"config/config.go:356","time":"2023-02-07T23:31:14.124071+02:00"}
{"level":"debug","msg":"detected overridden config - adding \"{wkom7}\" suffix to name","object":"REDACTED","objectType":"string","source":"fs/newfs.go:45","time":"2023-02-07T23:31:14.124203+02:00"}
{"level":"debug","msg":"New connection REDACTED to \"SSH-2.0-Maverick_SSHD\"","object":"REDACTED","objectType":"*sftp.Fs","source":"sftp/sftp.go:330","time":"2023-02-07T23:31:15.137979+02:00"}
{"level":"debug","msg":"low level retry 1/10 (error couldn't initialise SFTP: EOF)","object":"pacer","objectType":"string","source":"fs/pacer.go:90","time":"2023-02-07T23:31:15.652606+02:00"}
{"level":"debug","msg":"Rate limited, increasing sleep to 200ms","object":"pacer","objectType":"string","source":"fs/pacer.go:46","time":"2023-02-07T23:31:15.652682+02:00"}
.
.
.
.
.
{"level":"debug","msg":"low level retry 4/10 (error couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain)","object":"pacer","objectType":"string","source":"fs/pacer.go:90","time":"2023-02-07T23:31:25.228663+02:00"}
.
.
.
{"level":"debug","msg":"low level retry 10/10 (error couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain)","object":"pacer","objectType":"string","source":"fs/pacer.go:90","time":"2023-02-07T23:31:37.045686+02:00"}
2023/02/07 23:31:37 Failed to create file system for "REDACTED": NewFs: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

the conf file is pretty redundant:

[client_s3]
type = s3
provider = AWS
env_auth = true

[client_sftp]
type = sftp
host =
user =
md5sum_command = none
sha1sum_command = none

I would appreciate any suggestion or an idea how to debug this issue.

Thanks!!!

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

@Boaz_Fishman

Version \"v1.56.2\"

please update to latest stable, v1.61.1 and test again.
https://rclone.org/downloads/#script-download-and-install

[Boaz_Fishman]

Thanks, I will do it the minute my account will get unlocked.
A quick question, does the logs from running lsjson instead of copy are suffice? (with the sftp authorization)

thanks!!!!

[Boaz_Fishman]

After updating to 1.61.1

{"level":"debug","msg":"Version \"v1.61.1\" starting with parameters [\"rclone\" \"lsjson\" \"client_sftp:Inbox\" \"--retries\" \"1\" \"--log-level\" \"DEBUG\" \"--use-json-log\" \"--transfers\" \"16\" \"--config\" \"rclone.conf\" \"--sftp-use-insecure-cipher\" \"--sftp-user\" \"REDACTED\" \"--sftp-pass\" \"REDACTED\" \"--sftp-host\" \"mftprod.REDACTED.com\"]","object":"rclone","objectType":"string","source":"cmd/cmd.go:414","time":"2023-02-08T09:05:17.703038+02:00"}
{"level":"debug","msg":"Creating backend with remote \"REDACTED:Inbox\"","source":"fs/newfs.go:27","time":"2023-02-08T09:05:17.703369+02:00"}
{"level":"debug","msg":"Using config file from \"REDACTED/rclone.conf\"","source":"config/config.go:356","time":"2023-02-08T09:05:17.706409+02:00"}
{"level":"debug","msg":"detected overridden config - adding \"{wkom7}\" suffix to name","object":"REDACTED","objectType":"string","source":"fs/newfs.go:47","time":"2023-02-08T09:05:17.706538+02:00"}
{"level":"debug","msg":"New connection REDACTED:65299-\u003eREDACTED:22 to \"SSH-2.0-Maverick_SSHD\"","object":"sftp://REDACTED@mftprod.REDACTED.com:22/Inbox","objectType":"*sftp.Fs","source":"sftp/sftp.go:459","time":"2023-02-08T09:05:19.919756+02:00"}
{"level":"debug","msg":"low level retry 1/10 (error couldn't initialise SFTP: EOF)","object":"pacer","objectType":"string","source":"fs/pacer.go:90","time":"2023-02-08T09:05:20.512123+02:00"}
{"level":"debug","msg":"Rate limited, increasing sleep to 200ms","object":"pacer","objectType":"string","source":"fs/pacer.go:46","time":"2023-02-08T09:05:20.512351+02:00"}
{"level":"debug","msg":"New connection REDACTED:65300-\u003eREDACTED:22 to \"SSH-2.0-Maverick_SSHD\"","object":"sftp://REDACTED@mftprod.REDACTED.com:22/Inbox","objectType":"*sftp.Fs","source":"sftp/sftp.go:459","time":"2023-02-08T09:05:22.19103+02:00"}
{"level":"debug","msg":"low level retry 2/10 (error couldn't initialise SFTP: EOF)","object":"pacer","objectType":"string","source":"fs/pacer.go:90","time":"2023-02-08T09:05:23.931835+02:00"}
{"level":"debug","msg":"Rate limited, increasing sleep to 400ms","object":"pacer","objectType":"string","source":"fs/pacer.go:46","time":"2023-02-08T09:05:23.931952+02:00"}
{"level":"debug","msg":"New connection REDACTED:65301-\u003eREDACTED:22 to \"SSH-2.0-Maverick_SSHD\"","object":"sftp://REDACTED@mftprod.REDACTED.com:22/Inbox","objectType":"*sftp.Fs","source":"sftp/sftp.go:459","time":"2023-02-08T09:05:27.429262+02:00"}
{"level":"debug","msg":"low level retry 3/10 (error couldn't initialise SFTP: EOF)","object":"pacer","objectType":"string","source":"fs/pacer.go:90","time":"2023-02-08T09:05:30.373791+02:00"}
...
{"level":"debug","msg":"low level retry 10/10 (error couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain)","object":"pacer","objectType":"string","source":"fs/pacer.go:90","time":"2023-02-08T09:05:49.049397+02:00"}
2023/02/08 09:05:49 Failed to create file system for "REDACTED:Inbox": NewFs: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

please notice that after 3 retries we get locked so after the 3rd retry the error message isn't relevant to the issue.

I can successfully connect via filezilla, can I use it to debug the issue? (If I'm only missing a flag maybe I can deduce it from filezilla logs)

Thank you very much for the time and patience.

[ncw (Nick Craig-Wood)]

@Boaz_Fishman

error couldn't initialise SFTP: EOF

That is not the most helpful error message ever, but I think it means that the remote end just shut the TCP connection. This is usually caused by some kind of firewall I think.

Is that possible?

This thread may be useful

NewFs: couldn't initialise SFTP: EOF

What is the problem you are having with rclone? I am able to SSH into my remote machine by ssh 10.1.0.12 \ -l root \ -i ~/.ssh/id_ed25519 \ -p 22 However, I am not be able to use Rclone, for example, to list files. I did search online, but didn't find anything related. (Couldn't initialize SFTP: EOF does not help, as I am able to use SSH in my case.) Any guide to resolve the error NewFs: couldn't initialise SFTP: EOF would be appreciate! Thanks! Run the command 'rclone vers…

Rclone will make multiple connections to the backend which might be a problem though I don't see that in your log.

[Boaz_Fishman]

Thanks @ncw

@ncw

This is usually caused by some kind of firewall I think.

I'm guessing that if I can connect using FileZilla then it's not a firewall issue ... is there any "identification" mechanism such as user agent in sftp?

I'll read the thread and see if it helps, Thank you!

[ncw (Nick Craig-Wood)]

@Boaz_Fishman

I'm guessing that if I can connect using FileZilla then it's not a firewall issue

Probably not. Check the port is correct.

@Boaz_Fishman

is there any "identification" mechanism such as user agent in sftp?

There is in the underlying SSH protocol.

You can change what rclone uses with --user-agent I think. I doubt it will have any effect though.

@Boaz_Fishman

[client_sftp]
type = sftp
host =
user =
md5sum_command = none
sha1sum_command = none

You don't have any Auth in here. Are you using an ssh agent? Or is there a password line? If so make sure it is obscured with rclone obscure

[Boaz_Fishman]

Thanks,

It seems the port is correct.
I passed the password obscured both via the CLI and configurations, same result.

I investigated further:

• It doesn't matter if I use an invalid password or a valid one, I get the same log output.
• I can ssh to the machine and it works fine.

@ncw How can I debug rclone? I have small experience with go and hashicorp's systems.

Is there a starters guide for debugging? couldn't find one on github

[albertony]

@Boaz_Fishman

Is there a starters guide for debugging? couldn't find one on github

Don't think there is. Which os? I'm using Visual Studio Code with the go extension (golang.go), and really happy with that. Here's some pointers to rclone-specific configuration I use:

For mount sftp,why right click on exe file is so slow until it freezes

Using VSCode? If so, some real quick pointers based on my setup, if it helps... In launch.json (for debug): "name": "Launch", "type": "go", "request": "launch", "mode": "debug", "program": "${workspaceFolder}", "buildFlags": "-tags cmount", "env": { "CGO_ENABLED": "0" }, In settings.json (for intellisense etc) { "go.buildTags": "cmount" } In tasks.json (for build): "label": "Go build task", "type": "shell", "options": { "env": { "CGO_ENABLED": "0" } }, "command": "go build -x -ta…

Edit: And to debug specific commands, add arguments to rclone like this in launch.json:

"args": [
    "serve", "http", "C:\\Temp", "-vv"
]

[Boaz_Fishman]

Thanks @albertony it actually wet smoother than I thought
I passed the same args I passed to cli and I'm debugging the issue.

I how I'll get a better idea

[Boaz_Fishman]

Ok after debugging I found the issue.

The password ended with an '&'.
obscuring the password without "" dropped the & character.

Thank you all for your patience

[ncw (Nick Craig-Wood)]

@Boaz_Fishman

The password ended with an '&'.
obscuring the password without "" dropped the & character.

Ah ha! Shell escaping strikes again.

Well done for finding the issue.