[SOLVED] - Server Modification Time Not Working on SFTP Server

Help and Support
1

What is the problem you are having with rclone?

I am trying to list files and check the upload time using the --use-server-modtime flag, but it doesn't work on one SFTP server

Run the command 'rclone version' and share the full output of the command.

rclone 1.65.0
- os/version: nixos 24.05 (Uakari) (64 bit)
- os/kernel: 6.1.69 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.5
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

SFTP & Local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vv Downloads/test/2MB.mp4 sftp:/others/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[sftp]
type = sftp
host = XXXXXX
user = admin
pass = verysecretpassword
use_insecure_cipher = true
shell_type = none

A log from the command that you were trying to run with the -vv flag

rclone copy -vv Downloads/test/2MB.mp4 sftp:/others/
<7>DEBUG : rclone: Version "1.65.0" starting with parameters ["/etc/profiles/per-user/user/bin/rclone" "copy" "-vv" "Downloads/test/2MB.mp4" "sftp:/others/"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "Downloads/test/2MB.mp4"
<7>DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
<7>DEBUG : fs cache: adding new entry for parent of "Downloads/test/2MB.mp4", "/home/user/Downloads/test"
<7>DEBUG : Creating backend with remote "sftp:/others/"
<7>DEBUG : sftp://admin@host.com:22//others/: New connection XXX.XXX.XXX.XXX:34554->XXX.XXX.XXX.XXX:22 to "SSH-2.0-WeOnlyDo-wodFTPD 3.2.2.395"
<7>DEBUG : sftp://admin@host.com:22//others/: Shell type "none" from config
<7>DEBUG : sftp://admin@host.com:22//others/: Using root directory "/others/"
<7>DEBUG : 2MB.mp4: Size and modification time the same (differ by 0s, within tolerance 1s)
<7>DEBUG : 2MB.mp4: Unchanged skipping
<6>INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         2.3s

<7>DEBUG : 13 go routines active
<7>DEBUG : sftp://admin@host.com:22//others/: Closing 1 unused connections
~ took 2s ❯ rclone copy -vv Downloads/test/2MB.mp4 sftp:/others/
<7>DEBUG : rclone: Version "1.65.0" starting with parameters ["/etc/profiles/per-user/user/bin/rclone" "copy" "-vv" "-u" "Downloads/test/2MB.mp4" "sftp:/others/"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "Downloads/test/2MB.mp4"
<7>DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
<7>DEBUG : fs cache: adding new entry for parent of "Downloads/test/2MB.mp4", "/home/user/Downloads/test"
<7>DEBUG : Creating backend with remote "sftp:/others/"
<7>DEBUG : sftp://admin@host.com:22//others/: New connection XXX.XXX.XXX.XXX:37512->XXX.XXX.XXX.XXX:22 to "SSH-2.0-WeOnlyDo-wodFTPD 3.2.2.395"
<7>DEBUG : sftp://admin@host.com:22//others/: Shell type "none" from config
<7>DEBUG : sftp://admin@host.com:22//others/: Using root directory "/others/"
<7>DEBUG : 2MB.mp4: Need to transfer - File not found at Destination
<3>ERROR : sftp://admin@host.com:22//others/: Move: Failed to remove existing file "/others/2MB.mp4": sftp: "Bad message" (SSH_FX_BAD_MESSAGE)
<7>DEBUG : 2MB.mp4.yezigek2.partial: renamed to: 2MB.mp4
<6>INFO  : 2MB.mp4: Copied (new)
<6>INFO  : 
Transferred:        1.943 MiB / 1.943 MiB, 100%, 497.405 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         7.2s

<7>DEBUG : 15 go routines active
<7>DEBUG : sftp://admin@host.com:22//others/: Closing 1 unused connections

So if I execute rclone lsjson -u sftp:/others/ --use-server-modtime | jq, the modification time of the .partial file is correct, but as soon as the file is fully transferred to the SFTP server and renamed to the original filename, its modification time is back to 2021, which is incorrect.

[
  {
    "Path": "2MB.mp4.rayirav8.partial",
    "Name": "2MB.mp4.rayirav8.partial",
    "Size": 0,
    "MimeType": "video/mp4",
    "ModTime": "2024-01-03T16:14:45+01:00",
    "IsDir": false
  }
]

and this is when the file is fully transferred to the SFTP server.

[  
  {
    "Path": "2MB.mp4",
    "Name": "2MB.mp4",
    "Size": 2037627,
    "MimeType": "video/mp4",
    "ModTime": "2022-12-31T14:30:59+01:00",
    "IsDir": false
  }
]

My is that I would like to run filter where I provide --min-age and --max-age relative to the upload timestamp, but with this incorrect modtime I would be unable to do it.

2

welcome to the forum,

the use of .partial is a new feature, might be that it does not work correctly when using --use-server-modtime
might try --inplace

3

it works just fine with S3-compliant storage, but not on this SFTP. I tried to pass --inplace with my copy command, but the result was the same. The modification time of the fully transferred file is again 2021.

4

i think that is correct.
maybe, given that sftp has only one modtime, --use-server-modtime would not work.

as far as i know, rclone supports btime for S3 but not SFTP
Help": "Time of file birth (creation) read from Last-Modified header"

what is your goal, to have the dest file modtime set to the current time on the server?

5

Yes, exactly, I want to check and filter the files on the SFTP based on their upload date.

6

maybe try --no-update-modtime

maybe an issue when using shell_type = none

what other sftp clients have you tested that support the feature you want?

7

I tried all possible options:

  • shell
  • cmd
  • powershell

and it didn't work with any of them.

I also tried to use --no-update-modtime while copying the file to no avail, even though I am not sure this is the right flag:

--no-update-modtime - Don't update destination mod-time if files identical

If I upload the file directly using Filezilla, the modification time is set to NOW, so this is what I am trying to replicate.

8

Does it mean that it works on other sftp servers you tried and you have only problem with one?

9

that is a setting in filezilla
image

what is the sftp server, fzSftp or what?

10

No, I didn't try with other SFTP servers, only with this one. I don't have access to other SFTP servers unless I deploy one on a VM/docker myself. Plus I don't have any control whatsoever over the SFTP configuration.

11

Found the problem: SFTP if I change the SFTP config to include set_modtime = false the file uploaded is set to NOW.

Modification times and hashes
Modified times are stored on the server to 1-second precision.
Modified times are used in syncing and are fully supported.
Some SFTP servers disable setting/modifying the file modification time after upload (for example, certain configurations of ProFTPd with mod_sftp). If you are using one of these servers, you can set the option set_modtime = false in your RClone backend configuration to disable this behavior.

The default value of this option is true.
So now the output of rclone config show is:

[sftp]
type = sftp
host = XXXXXX
user = admin
pass = XXXXXXX
use_insecure_cipher = true
set_modtime = false

And now the file is with the current modification timestamp:

[
  {
    "Path": "2MB.mp4",
    "Name": "2MB.mp4",
    "Size": 2037627,
    "MimeType": "video/mp4",
    "ModTime": "2024-01-03T18:39:11+01:00",
    "IsDir": false
  }
]

[EDIT]: Thanks everyone for your help! Really amazing community!

1 Like
12

welcome and yeah, the rclone forum rocks!

good, in the end, i just keep trying different flags until it works as i need.
along the way, i learn more about rclone.

1 Like
13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.