FTP server reports file timestamps in UTC

rosco12 · August 31, 2021, 10:02am

What is the problem you are having with rclone?

When accessing Google Drive using FTP, file timestamps are shown in UTC timezone rather than my local timezone (UTC+12).

What is your rclone version (output from `rclone version`)

rclone version is 1.56.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 7 (64 bit)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Starting the FTP server (as root):

# rclone -vv serve ftp googledrive:/ &
2021/08/31 17:17:44 NOTICE: Google drive root '': Serving FTP on localhost:2121

Accessing files with FTP client:

# ftp 127.0.0.1 2121
Connected to 127.0.0.1 (127.0.0.1).
220 Welcome to Rclone v1.56.0 FTP Server
Name (127.0.0.1:root): anonymous
331 User name ok, password required
Password:
230 Password ok, continue
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir Notebook
227 Entering Passive Mode (127,0,0,1,120,181)
150 Opening ASCII mode data connection for file list
-rw-r--r-- 1 root root         5968 May 17 05:03 Notebook-48x48.png
-rw-r--r-- 1 root root    114717696 Aug 31 04:20 Notebook.ctb
-rw-r--r-- 1 root root    114717696 Aug 31 03:11 Notebook.ctb~
-rw-r--r-- 1 root root    114717696 Aug 31 02:44 Notebook.ctb~~
-rw-r--r-- 1 root root         3810 Aug  8 04:23 SafeNotes.ctx
-rw-r--r-- 1 root root         3442 Aug  8 04:23 SafeNotes.ctx~
-rw-r--r-- 1 root root         3442 Aug  8 02:23 SafeNotes.ctx~~
-rw-r--r-- 1 root root       954368 Aug 14 02:59 cherrytree_manual.ctb
-rw-r--r-- 1 root root           11 Aug 31 05:01 last_change

Accessing files using rclone lsl shows the correct timestamps (UTC+12)

# rclone lsl googledrive:/Notebook
       11 2021-08-31 17:01:01.724000000 last_change
114717696 2021-08-31 16:20:27.427000000 Notebook.ctb
114717696 2021-08-31 15:11:26.619000000 Notebook.ctb~
114717696 2021-08-31 14:44:26.272000000 Notebook.ctb~~
     3810 2021-08-08 16:23:55.580000000 SafeNotes.ctx
     3442 2021-08-08 16:23:55.398000000 SafeNotes.ctx~
     3442 2021-08-08 14:23:53.961000000 SafeNotes.ctx~~
   954368 2020-08-14 14:59:29.000000000 cherrytree_manual.ctb
     5968 2015-05-17 17:03:17.000000000 Notebook-48x48.png

The rclone config contents with secrets removed.

[googledrive]
type = drive
scope = drive
token = {"access_token"":"token_type":"Bearer","refresh_token":"","expiry":"2021-08-31T22:02:43.773401458+12:00"}
team_drive =

A log from the command with the `-vv` flag

2021/08/31 21:56:49 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "-vv" "serve" "ftp" "googledrive:/"]
2021/08/31 21:56:49 DEBUG : Creating backend with remote "googledrive:/"
2021/08/31 21:56:49 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/08/31 21:56:49 DEBUG : Google drive root '': 'root_folder_id = 0AJSAHpbQvFf9Uk9PVA' - save this in the config to speed up startup
2021/08/31 21:56:49 DEBUG : fs cache: renaming cache item "googledrive:/" to be canonical "googledrive:"
2021/08/31 21:56:50 NOTICE: Google drive root '': Serving FTP on localhost:2121
2021/08/31 21:56:50 INFO  : : Rclone FTP Server listening on 2121
2021/08/31 21:57:40 DEBUG : : NewDriver: Init driver
2021/08/31 21:57:40 DEBUG : : >NewDriver: 
2021/08/31 21:57:40 INFO  : 00f10aa90d8ac96f0d91: Connection Established
2021/08/31 21:57:40 INFO  : 00f10aa90d8ac96f0d91: < 220 Welcome to Rclone v1.56.0 FTP Server
2021/08/31 21:57:47 INFO  : 00f10aa90d8ac96f0d91: > USER anonymous
2021/08/31 21:57:47 INFO  : 00f10aa90d8ac96f0d91: < 331 User name ok, password required
2021/08/31 21:57:49 INFO  : 00f10aa90d8ac96f0d91: > PASS ****
2021/08/31 21:57:49 INFO  : 00f10aa90d8ac96f0d91: < 230 Password ok, continue
2021/08/31 21:57:49 INFO  : 00f10aa90d8ac96f0d91: > SYST 
2021/08/31 21:57:49 INFO  : 00f10aa90d8ac96f0d91: < 215 UNIX Type: L8
2021/08/31 21:57:50 DEBUG : Google drive root '': Checking for changes on remote
2021/08/31 21:58:10 INFO  : 00f10aa90d8ac96f0d91: > PASV 
2021/08/31 21:58:10 INFO  : 00f10aa90d8ac96f0d91: < 227 Entering Passive Mode (127,0,0,1,122,98)
2021/08/31 21:58:10 INFO  : 00f10aa90d8ac96f0d91: > LIST Notebook
2021/08/31 21:58:10 DEBUG : /Notebook: Stat: 
2021/08/31 21:58:10 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/08/31 21:58:10 DEBUG : pacer: Rate limited, increasing sleep to 1.726496986s
2021/08/31 21:58:11 DEBUG : pacer: Reducing sleep to 0s
2021/08/31 21:58:11 DEBUG : My Saved Places: Ignoring unknown document type "application/vnd.google-apps.map"
2021/08/31 21:58:11 DEBUG : /Notebook: >Stat: fi=&{FileInfo:Notebook/ mode:2147484141 owner:0 group:0}, err = <nil>
2021/08/31 21:58:11 DEBUG : /Notebook: ListDir: 
2021/08/31 21:58:13 DEBUG : /Notebook: >ListDir: err = <nil>
2021/08/31 21:58:13 INFO  : 00f10aa90d8ac96f0d91: < 150 Opening ASCII mode data connection for file list
2021/08/31 21:58:13 INFO  : 00f10aa90d8ac96f0d91: < 226 Closing data connection, sent 719 bytes
2021/08/31 21:58:50 DEBUG : Google drive root '': Checking for changes on remote
2021/08/31 21:59:50 DEBUG : Google drive root '': Checking for changes on remote

ivandeex · August 31, 2021, 11:21am

most ftp servers have no means to negotiate time zone with client so they return modtime as utc. that's not a problem, just a protocol detail.

rosco12 · September 1, 2021, 5:19am

In my experience FTP servers present timestamps in their local time, though obviously not rclone.
In any case I have swapped to using webdav which has fixed the problem.

ivandeex · September 1, 2021, 9:46am

Could you please give a name/version or public account of an FTP web server that provides timestamps in local time missed by rclone? I'd be happy to investigate.

rosco12 · September 1, 2021, 10:42pm

I'm not using rclone to access remote FTP servers. I'm using "rclone serve ftp" to create a local private FTP server through which I can mount cloud storage using "gio mount". (I tried "rclone mount" but it was too slow.)
Accessing Google Drive this way the timestamps are in UTC. However accessing MEGA this way the timestamps are my local timezone. It would be good if the rclone FTP server for Google Drive worked the same way as for MEGA.

ncw · September 4, 2021, 11:51am

That is interesting info...

If I try a local file system then I get UTC timestamps

rclone serve ftp --user test --pass test /tmp

And test with

rclone lsl --max-depth 1 ":ftp,host=localhost,port=2121,user=test,pass=gblWP10H-JrGe1M8PouITTcEeD8:"

Whereas if I try google drive, I get local time

$ rclone lsl --max-depth 1 TestDrive: --include "/test.txt"
        6 2021-03-30 15:52:25.503000000 test.txt
$ rclone lsl --max-depth 1 ":ftp,host=localhost,port=2121,user=test,pass=gblWP10H-JrGe1M8PouITTcEeD8:" --include "/test.txt"
        6 2021-03-30 15:52:00.000000000 test.txt

So there must be something whacky about rclone's ftp serving time.

Looking at the code, the FTP library is using the output of ModTime directly. Maybe it is doing something strange with the timezone.

I guess the options are

send an upstream patch to the FTP library fix its timezone handling
do the same in rclone

I had a go with 2) here - can you give it a test?

v1.57.0-beta.5651.07ba58e62.fix-ftp-serve-modtime on branch fix-ftp-serve-modtime (uploaded in 15-30 mins)

@ivandeex do you think this is the right fix, or should we be sending a patch to the ftp server library?

ivandeex · September 4, 2021, 5:51pm

FTP servers have 2 methods to return file time:

As a modified file property in reply to the MLSD command (extended listing). Again, RFC states that this should always be UTC (note that the the official response format does not have a timezone field at all).

This is supported by jlaffaye/ftp but the library returns times to the caller as zone-less values which is a bug.

Note that RFC does not prevent FTP server implementations from adding ad-hoc file facts including times with zone included. Rclone as a caller has no means to find out where the fact was parsed from, so it's responsibility of the protocol parser in jlaffaye/ftp to mark returned times properly.

In reply to the MDTM command. RFC3659 states that this should always be UTC.

This method is not yet supported by jlaffaye/ftp as of this time but it will when I find time to submit my FTP-related jlaffaye/ftp patches and rclone patches.
It will address the time zone issue too.

FTP servers returning time in local timezone should be considered buggy.
The implementation in rclone serve ftp obeys the standard precisely. The user here asks us to add a flag to serve ftp implementing a non-standard quirk for compatibility with something else (google?).

I don't understand what you are trying to do here. Mitigate a zone-less value returned by jlaffaye/ftp? We'd better fix it right there!
Note that the user talks about serve ftp, not the ftp client.

ncw · September 4, 2021, 10:25pm

Its a workaround for the rclone serve ftp code namely "goftp.io/server/core" ignoring the timezone of the ModTimes

This is the formatting code from the library in core/list_formatter.go and what it appears to be doing is ignoring the timezone completely of the ModTime() returned. So my little patch makes sure the ModTimes are in UTC and hence the output of the FTP server times are always in UTC.

// Detailed returns a string that lists the collection of files with extra
// detail, one per line
func (formatter listFormatter) Detailed() []byte {
	var buf bytes.Buffer
	for _, file := range formatter {
		fmt.Fprint(&buf, file.Mode().String())
		fmt.Fprintf(&buf, " 1 %s %s ", file.Owner(), file.Group())
		fmt.Fprint(&buf, lpad(strconv.FormatInt(file.Size(), 10), 12))
		if file.ModTime().Before(time.Now().AddDate(-1, 0, 0)) {
			fmt.Fprint(&buf, file.ModTime().Format(" Jan _2  2006 "))
		} else{
			fmt.Fprint(&buf, file.ModTime().Format(" Jan _2 15:04 "))
		}
		fmt.Fprintf(&buf, "%s\r\n", file.Name())
	}
	return buf.Bytes()
}

I guess doing it like that gives the user of the library control over exactly what time is displayed - if you want localtime, pass in a ModTime() with the correct timezone, whereas if you want UTC pass in a UTC modtime. So I think the patch for rclone is probably the correct thing to do.

What do you think?

ivandeex · September 5, 2021, 1:53am

Sorry, I mistook your serve/ftp change for backend/ftp. My bad! Your patch is good...

...until it's incomplete because rclone serve ftp does not support MLSD yet, just ancient LIST so for reasonably old files the time is returned by server with +/- a day precision making the whole timezone discussion a joke.

Some time ago I made a preliminary patch introducing MLSD and MDTM in goftp-server here, we discussed it on the tracker.

ncw · September 5, 2021, 10:24am

Eeek!

@rosco12 can you have a go with this and see if it fixes the problem for you?

rosco12 · September 5, 2021, 10:56am

I tried the following:

rclone serve ftp gcrypt:/ &
ftp localhost 2121
Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
220 Welcome to Rclone v1.57.0-beta.5651.07ba58e62.fix-ftp-serve-modtime FTP Server
Name (localhost:root): anonymous
331 User name ok, password required
Password:
230 Password ok, continue
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir Notebook/last_change
227 Entering Passive Mode (127,0,0,1,118,200)
150 Opening ASCII mode data connection for file list
-rw-r--r-- 1 root root           11 Sep  5 05:01 last_change
226 Closing data connection, sent 62 bytes

Then:

rclone lsl gcrypt:/Notebook/last_change
       11 2021-09-05 17:01:01.461000000 last_change

So the FTP server still reports UTC for Google Drive. So now does MEGA (was previously local time).

It would be good if there was an option to use localtime even if this is not strictly RFC compliant. The FTP server would then be consistent with the rclone WebDAV server and rclone lsl.

system · November 5, 2021, 6:56am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.