Incorrect File Date/Time from rclone serve http

alan113696 · May 4, 2020, 8:06pm

What is the problem you are having with rclone?

When using rclone serve http to serve a backend, and then using rclone copy to transfer files, the resulting files do not have the proper date/time. The date/time ends up being "Dec 31, 1969" for all files.

What is your rclone version (output from `rclone version`)

rclone v1.51.0

os/arch: linux/amd64
go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux, 64 bit
CentOS Linux release 7.8.2003 (Core)
3.10.0-1127.el7.x86_64

Which cloud storage system are you using? (eg Google Drive)

Occurs when backend is S3 or local

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone serve http local: --read-only
rclone copy --http-url http://localhost:8080 :http:test ./test -P

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

rclone copy --http-url http://localhost:8080 :http:test ./test -vv --dump requests

2020/05/04 15:52:48 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "copy" "--http-url" "http://localhost:8080" ":http:test" "./test" "-vv" "--dump" "requests"]
2020/05/04 15:52:48 DEBUG : Using config file from "/home/mydev/.config/rclone/rclone.conf"
2020/05/04 15:52:48 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : HTTP REQUEST (req 0xc00051a900)
2020/05/04 15:52:48 DEBUG : HEAD /test HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : HTTP RESPONSE (req 0xc00051a900)
2020/05/04 15:52:48 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 11
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 04 May 2020 19:52:48 GMT
Server: rclone/v1.51.0
X-Content-Type-Options: nosniff

2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : HTTP REQUEST (req 0xc00051af00)
2020/05/04 15:52:48 DEBUG : GET /test/ HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.51.0
Accept-Encoding: gzip

2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : HTTP RESPONSE (req 0xc00051af00)
2020/05/04 15:52:48 DEBUG : HTTP/1.1 200 OK
Content-Length: 252
Accept-Ranges: bytes
Content-Type: text/html; charset=utf-8
Date: Mon, 04 May 2020 19:52:48 GMT
Server: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : HTTP REQUEST (req 0xc00051b200)
2020/05/04 15:52:48 DEBUG : HEAD /test/file1.txt HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : HTTP REQUEST (req 0xc00051b400)
2020/05/04 15:52:48 DEBUG : HEAD /test/file2.txt HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : HTTP RESPONSE (req 0xc00051b200)
2020/05/04 15:52:48 DEBUG : HTTP/1.1 200 OK
Content-Length: 6
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 04 May 2020 19:52:48 GMT
Server: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : HTTP RESPONSE (req 0xc00051b400)
2020/05/04 15:52:48 DEBUG : HTTP/1.1 200 OK
Content-Length: 6
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 04 May 2020 19:52:48 GMT
Server: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 INFO : Local file system at /tmp/test: Waiting for checks to finish
2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : HTTP REQUEST (req 0xc00051b800)
2020/05/04 15:52:48 DEBUG : GET /test/file1.txt HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.51.0
Accept-Encoding: gzip

2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 DEBUG : HTTP REQUEST (req 0xc00051ba00)
2020/05/04 15:52:48 DEBUG : GET /test/file2.txt HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.51.0
Accept-Encoding: gzip

2020/05/04 15:52:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/05/04 15:52:48 INFO : Local file system at /tmp/test: Waiting for transfers to finish
2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : HTTP RESPONSE (req 0xc00051ba00)
2020/05/04 15:52:48 DEBUG : HTTP/1.1 200 OK
Content-Length: 6
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 04 May 2020 19:52:48 GMT
Last-Modified: Mon, 04 May 2020 19:33:30 GMT
Server: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 DEBUG : HTTP RESPONSE (req 0xc00051b800)
2020/05/04 15:52:48 DEBUG : HTTP/1.1 200 OK
Content-Length: 6
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 04 May 2020 19:52:48 GMT
Last-Modified: Mon, 04 May 2020 19:33:26 GMT
Server: rclone/v1.51.0

2020/05/04 15:52:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/05/04 15:52:48 INFO : file1.txt: Copied (new)
2020/05/04 15:52:48 INFO : file2.txt: Copied (new)
2020/05/04 15:52:48 INFO :
Transferred: 12 / 12 Bytes, 100%, 480 Bytes/s, ETA 0s
Transferred: 2 / 2, 100%
Elapsed time: 0.0s

2020/05/04 15:52:48 DEBUG : 9 go routines active
2020/05/04 15:52:48 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "copy" "--http-url" "http://localhost:8080" ":http:test" "./test" "-vv" "--dump" "requests"]

Animosity022 · May 4, 2020, 8:37pm

What happens if you rclone lsl on the remote you setup? Is the time right?

textere@seraphim ~ % rclone lsl GD:hosts
      266 2020-03-27 08:31:26.338000000 hosts

alan113696 · May 4, 2020, 8:43pm

rclone --http-url http://localhost:8080 lsl :http:test
    6 1969-12-31 19:00:00.000000000 file1.txt
    6 1969-12-31 19:00:00.000000000 file2.txt

Timestamps are wrong as well. Thoughts?

asdffdsa · May 4, 2020, 8:44pm

that is a well known date and time

Animosity022 · May 4, 2020, 8:44pm

What's your rclone.conf look like for that local entry?

alan113696 · May 4, 2020, 8:46pm

rclone lsl local:test
    6 2020-05-04 15:33:26.005404545 file1.txt
    6 2020-05-04 15:33:30.725462005 file2.txt

Here are the file timestamps I was expecting.

alan113696 · May 4, 2020, 8:48pm

Here's the rclone.conf:

cat ~/.config/rclone/rclone.conf 
[local]
type = local

Animosity022 · May 4, 2020, 8:55pm

Perfect. I can reproduce the same thing on my system as well as I wanted to make sure.

felix@gemini:~$ rclone --http-url http://192.168.1.30:5252 lsl :http:logs/rclone.log
2438728 1969-12-31 19:00:00.000000000 rclone.log

Seems like a bug to me..I'd file a bug report on the github as we have the steps to reproduce it.

alan113696 · May 4, 2020, 8:56pm

I appreciate the help in verifying that this could be a bug! I'll follow up with a bug report on Github.

Animosity022 · May 4, 2020, 8:57pm

From my search, I didn't find any issues like it. It seems like it's not getting the date/time from the file for some reason.

AlvordSky · May 5, 2020, 1:13am

The time quoted - 1969-12-31 19:00:00.000000000 - is suspiciously close to the UNIX/Linux epoch (time counter at 0 (zero)) - give or take a few time zones. Some software is failing to set the binary time_t date/time field in the file stat info to a value other than zero.

alan113696 · May 5, 2020, 1:25am

Posted issue on github.

ncw · May 5, 2020, 8:51am

I've posted a fix on the issue

Animosity022 · May 5, 2020, 10:40am

Awesome.

Having a great issue and how to reproduce makes a fix a bit easier as @ncw has to do less to figure things out and looks like he fixed already in the beta.

alan113696 · May 5, 2020, 1:05pm

I tested the fix and it works! Thanks @ncw for the help!

system · July 5, 2020, 9:05am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.