Rclone 1.39 ssh-agent in a cron job


#1

I am trying to consolidate my cron jobs so that everything goes through rclone instead of a mashup of plexdrive, unionfs, rclone and rsync, but I think I have run into a bug.

I am running rclone v1.39 in Ubuntu 16.04.3 LTS and trying to sync to an SSH target using ssh-agent with a password-protected private key.

On boot I run:

ssh-agent -s > ${AGENT}
. ${AGENT}
ssh-add

My cron job calls a bash script that imports ${AGENT}
The bash script successfully sets the environmental variables (which are echoed to a log file):

SSH_AGENT_PID=19645
SSH_AUTH_SOCK=/tmp/ssh-p9QnHM5NAc2X/agent.19645

However, rclone throws this error when called from a cron job: NewFs: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

If I call the script from the command line, it runs without error, so it seems like the environmental variables are somehow not seen by rclone when called from a cron job.

The reason I think that this is a bug and not a weird shell issue is because rsync runs perfectly from the cron job. That is, I can run rsync -rP /path/ sshtarget:/path/ and rclone copy /path/ sshtarget:/path/ in the same script. When the script is called from an interactive session both commands run perfectly, but when called from cron only rsync runs; rclone throws an error.


#2

I managed to reproduce this with a simple crontab

# test
* * * * * /home/ncw/z.sh

And a simple script

#!/bin/bash

export SSH_AUTH_SOCK=/tmp/ssh-0dY6RXF7sLcL/agent.3510
export SSH_AGENT_PID=3612

rclone lsf --dirs-only box: >/tmp/z-log 2>&1

Which works on the command line but not in the crontab…

I put a debug in rclone to see if it was connecting to the agent and it is and I can see it getting the signers from the agent…

However if I add export USER=ncw to the script it works…

Crontab doesn’t set the USER env var it sets LOGNAME instead. rclone should really be reading the username a different way I think…

I’ll try this and post a beta shortly…


#3

Thanks, at least it is an easy fix for now!


#4

I made a fix here

https://beta.rclone.org/v1.39-015-g28480c05/ (uploaded in 15-30 mins)

Let me know how it goes!


#5

No go, unfortunately, but settings the $USER environmental variable in the crontab did the trick, so at least it is fixed for the time being.


#6

Hmm, that definitely fixes it for me … can you check you used the new binary? (put rclone version in the script?) Thanks!


#7

This is the version information:

$ rclone --version
rclone v1.39-015-g28480c05β
- os/arch: linux/amd64
- go version: go1.9.2

I ran the test script from cron without USER set explicitly and got this:

2018/01/08 07:10:01 DEBUG : rclone: Version "v1.39-015-g28480c05β" starting with parameters ["rclone" "lsf" "-vv" "--dirs-only" "sshtarget:"]
2018/01/08 07:10:01 Failed to create file system for "sshtarget:": NewFs: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain

I don’t think there is anything unusual about my setup:

$ cat /etc/issue
Ubuntu 16.04.3 LTS \n \l

#8

Thanks for double checking

Can you show the contents of your sshtarget config please? (redact things you don’t want public is fine - I’m mostly interested in which fields are filled in and which aren’t)


#9

Ah-hah!
This works with or without $USER set:

[sshtarget]
type = sftp
host = sshtarget.com
user = chezmojo
port = 22
pass =
key_file =
use_insecure_cipher = false
disable_hashcheck =

But this fails if $USER is not set explicitly:

[sshtarget]
type = sftp
host = sshtarget.com
user =
port = 22
pass =
key_file =
use_insecure_cipher = false
disable_hashcheck =

When I created the config I just left username blank since I use the same username everywhere.


#10

Hmmm, if you try making a new sftp, then what does it print at this point as your user name?

Storage> sftp
SSH host to connect to
Choose a number from below, or type in your own value
 1 / Connect to example.com
   \ "example.com"
host> example.com
SSH username, leave blank for current username, ncw <<<<<< is this your username?
user> 

And does it do the same thing if you do unset USER first in the shell?

Does your script fail if you run unset USER in the shell BTW?

Can you look in the logs of the remote machine and see what user it is connecting as when it fails under the crontab?


#11

BTW I noticed a silly error in the patch I made which probably won’t affect you but it might do - who knows!

https://beta.rclone.org/v1.39-016-g052c8863/ (uploaded in 15-30 mins)


#12

With $USER set:
SSH username, leave blank for current username, chezmojo

Without $USER set:
SSH username, leave blank for current username, chezmojo

It looks like rclone is trying to connect with an empty user name (when user = in the config file and $USER is unset):

Jan  7 06:55:03 sshtarget sshd[18562]: Invalid user  from xxx.xxx.xxx.xxx
Jan  7 06:55:03 sshtarget sshd[18562]: input_userauth_request: invalid user  [preauth]
Jan  7 06:55:03 sshtarget sshd[18562]: Connection closed by xxx.xxx.xxx.xxx [preauth]

For example, this is what it looks like when random bots fish for default accounts:

sshd[18528]: Failed password for invalid user debian from 91.121.153.218 port 36352 ssh2

#13

That looks like the code to read the username is working properly even when $USER isn’t set

But that looks like it isn’t…

It should be the same code running to read the username so I’m puzzled!

What do you think is different about the two environments?

Did you try the revised version above?


#14

How is it determining the username? I am not very familiar with go, but if you could point me at the source code perhaps I could do some tests myself.

I don’t know, but this is far from the first time I have encountered mysterious issues with cron jobs. Ubuntu tends to spread config files all over the place making it hard to figure out exactly what cron vacuums up to configure itself before a run. I did dump env from cron and at least the environmental variables are all fine (e.g., setting $LOGNAME correctly).

Come to think of it, I ran into the same problem trying to get duplicity to backup to an SSH target (all on Ubuntu machines, too). Somehow Python was not passing the username into the backend. I was not able to solve that one and ended up calling it from a daemon spawned from an interactive session.

Currently the link (still) only shows Windows builds, so I have not been able to try it (yet).


#15

Here is the code which reads the username: https://github.com/ncw/rclone/blob/master/sftp/sftp.go#L120

That is the new code I put in to replace just reading $USER from the environment.

I gave the CI a kick - hopefully it will build shortly!


#16

This version:

$ rclone --version
rclone v1.39-016-g052c8863β
- os/arch: linux/amd64
- go version: go1.9.2

works as expected.
Thanks!


#17

That is great. Thanks for being patient and trying all the different versions. And have a :star: for making a very clear and well thought out bug report :slight_smile:


#18

Happy to be able to help; rclone is a fantastic project!