SFTP, Update SetModTime failed

lyutech · March 17, 2021, 12:52pm

What is the problem you are having with rclone?

Error when sending files to sftp.

What is your rclone version (output from `rclone version`)

rclone v1.54.1

os/arch: linux/amd64
go version: go1.15.8

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS Linux release 7.8.2003
x86_64

Which cloud storage system are you using? (eg Google Drive)

sftp

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone move ~/test.file master:/test/upload/ -vv --transfers 1 --checkers 0 --sftp-set-modtime=false --ignore-size --retries 1 --error-on-no-transfer --ignore-checksum --ignore-times --no-traverse

The rclone config contents with secrets removed.

[master]
type = sftp
host = xxx.xxx.xxx.xxx
user = user
port = 22
use_insecure_cipher = false
disable_hashcheck = true
set_modtime=false
key_file = ~/.ssh/id_rsa

A log from the command with the `-vv` flag

$ rclone move ~/test.file master:/test/upload/ -vv --transfers 1 --checkers 0 --sftp-set-modtime=false --ignore-size --retries 1 --error-on-no-transfer --ignore-checksum --ignore-times --no-traverse
2021/03/17 14:33:31 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "move" "/home/master/test.file" "master:/test/upload/" "-vv" "--transfers" "1" "--checkers" "0" "--sftp-set-modtime=false" "--ignore-size" "--retries" "1" "--error-on-no-transfer" "--ignore-checksum" "--ignore-times" "--no-traverse"]
2021/03/17 14:33:31 DEBUG : Creating backend with remote "/home/master/test.file"
2021/03/17 14:33:31 DEBUG : Using config file from "/home/master/.config/rclone/rclone.conf"
2021/03/17 14:33:31 DEBUG : fs cache: adding new entry for parent of "/home/master/test.file", "/home/master"
2021/03/17 14:33:31 DEBUG : Creating backend with remote "master:/test/upload/"
2021/03/17 14:33:32 DEBUG : sftp://user@xxx.xxx.xxx.xxx:/test/upload/: New connection xx.xx.xx.xx:57856->xxx.xxx.xxx.xxx:22 to "SSH-2.0-SSHD"
2021/03/17 14:33:33 DEBUG : test.file: Need to transfer - File not found at Destination
2021/03/17 14:33:34 ERROR : test.file: Failed to copy: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/17 14:33:34 ERROR : test.file: Not deleting source as copy failed: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/17 14:33:34 ERROR : Attempt 1/1 failed with 1 errors and: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/17 14:33:34 INFO  :
Transferred:             5 / 5 Bytes, 100%, 5 Bytes/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 1 / 1, 100%
Elapsed time:         2.8s

2021/03/17 14:33:34 DEBUG : 12 go routines active
2021/03/17 14:33:34 Failed to move: Update SetModTime failed: SetModTime stat failed: object not found

jwink3101 · March 17, 2021, 3:24pm

First, while it is likely not the problem, I would reduce your flags to the bare minimum. It makes it that much easier to isolate the issue.

Second, what is serving the sftp? Some servers have odd quirks. Alternatively, are you sure you have permissions on the sftp server? It certainly sounds like a permissions issue

ncw · March 17, 2021, 3:57pm

What SFTP server are you using?

You can disable setting the modtime here: SFTP

lyutech · March 18, 2021, 7:25am

Thank you for your answer
First - reduce flags
no flags:
rclone move ~/test.file master:/test/upload/-vv
2021/03/18 09:09:13 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "move" "/home/master/test.file" "master:/test/upload" "-vv"]
2021/03/18 09:09:13 DEBUG : Creating backend with remote "/home/master/test.file"
2021/03/18 09:09:13 DEBUG : Using config file from "/home/master/.config/rclone/rclone.conf"
2021/03/18 09:09:13 DEBUG : fs cache: adding new entry for parent of "/home/master/test.file", "/home/master"
2021/03/18 09:09:13 DEBUG : Creating backend with remote "master:/test/upload/"
2021/03/18 09:09:14 DEBUG : sftp://user@xxx.xxx.xxx.xxx:/test/upload/: New connection xx.xx.xx.xx:36436->xxx.xxx.xxx.xxx:22 to "SSH-2.0-SSHD"
2021/03/18 09:09:15 DEBUG : test.file: Need to transfer - File not found at Destination
2021/03/18 09:09:16 ERROR : test.file: Failed to copy: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:16 ERROR : test.file: Not deleting source as copy failed: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:16 ERROR : Attempt 1/3 failed with 1 errors and: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:16 DEBUG : test.file: Need to transfer - File not found at Destination
2021/03/18 09:09:17 ERROR : test.file: Failed to copy: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:17 ERROR : test.file: Not deleting source as copy failed: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:17 ERROR : Attempt 2/3 failed with 1 errors and: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:17 DEBUG : test.file: Need to transfer - File not found at Destination
2021/03/18 09:09:18 ERROR : test.file: Failed to copy: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:18 ERROR : test.file: Not deleting source as copy failed: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:18 ERROR : Attempt 3/3 failed with 1 errors and: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:09:18 INFO :
Transferred: 15 / 15 Bytes, 100%, 5 Bytes/s, ETA 0s
Errors: 1 (retrying may help)
Checks: 3 / 3, 100%
Elapsed time: 5.1s

2021/03/18 09:09:18 DEBUG : 11 go routines active
2021/03/18 09:09:18 Failed to move: Update SetModTime failed: SetModTime stat failed: object not found

and only with: --sftp-set-modtime=false

$ rclone move ~/test.file master:/test/upload/ -vv --sftp-set-modtime=false
2021/03/18 09:13:41 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "move" "/home/master/test.file" "master:/test/upload/" "-vv" "--sftp-set-modtime=false"]
2021/03/18 09:13:41 DEBUG : Creating backend with remote "/home/master/test.file"
2021/03/18 09:13:41 DEBUG : Using config file from "/home/masterd/.config/rclone/rclone.conf"
2021/03/18 09:13:41 DEBUG : fs cache: adding new entry for parent of "/home/master/test.file", "/home/master"
2021/03/18 09:13:41 DEBUG : Creating backend with remote "master:/test/upload/"
2021/03/18 09:13:42 DEBUG : sftp://user@xxx.xxx.xxx.xxx:/test/upload/: New connection xx.xx.xx.xx:36460->xxx.xxx.xxx.xxx:22 to "SSH-2.0-SSHD"
2021/03/18 09:13:43 DEBUG : test.file: Need to transfer - File not found at Destination
2021/03/18 09:13:44 ERROR : test.file: Failed to copy: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:44 ERROR : test.file: Not deleting source as copy failed: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:44 ERROR : Attempt 1/3 failed with 1 errors and: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:44 DEBUG : test.file: Need to transfer - File not found at Destination
2021/03/18 09:13:45 ERROR : test.file: Failed to copy: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:45 ERROR : test.file: Not deleting source as copy failed: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:45 ERROR : Attempt 2/3 failed with 1 errors and: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:45 DEBUG : test.file: Need to transfer - File not found at Destination
2021/03/18 09:13:46 ERROR : test.file: Failed to copy: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:46 ERROR : test.file: Not deleting source as copy failed: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:46 ERROR : Attempt 3/3 failed with 1 errors and: Update SetModTime failed: SetModTime stat failed: object not found
2021/03/18 09:13:46 INFO  :
Transferred:            15 / 15 Bytes, 100%, 5 Bytes/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:         5.2s

2021/03/18 09:13:46 DEBUG : 11 go routines active
2021/03/18 09:13:46 Failed to move: Update SetModTime failed: SetModTime stat failed: object not found

Second:
Most definitely, I do not have permission to read from this directory and I cannot influence it.

lyutech · March 18, 2021, 7:37am

Thank you for your interest in my problem.

I do not know what kind of SFTP is and I cannot find out.
Above I gave the config, the parameter is set there.

ncw · March 18, 2021, 8:19am

What is happening here is rclone uploads the object.

It doesn't try to set the modtime.

However it does stat the object after the upload to see if it is OK - got the right size etc.

At this point the server returns object not found.

So I think this server must take the file away immediately it has been uploaded.

As this reading of the stat happens in the upload, there are no flags which will fix this.

Try this - I made this ignore the error from the stat call if --sftp-set-modtime=false which I think should solve your problem.

v1.55.0-beta.5315.4397f8a44.fix-sftp-no-modtime on branch fix-sftp-no-modtime (uploaded in 15-30 mins)

lyutech · March 18, 2021, 12:49pm

Very grateful!
Working!

ncw · March 18, 2021, 4:32pm

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.55

system · March 21, 2021, 4:32pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.