Timestamps accuracy

Help and Support
1

It is my understanding that the accuracy of the timestamps retrieved by lsjson depends on the storage system in use.
For example I have reports about webdav servers returning timestamps like “2019-01-25 00:15:06” while timestamps for other remotes (or for the local filesystems) are like “2019-01-25 00:15:06.000776”.
My problem with that is that I have troubles determining the timestamp difference between local and remote copies of a file (I need that for UpBack). In the aforementioned example I calculate the difference to be 776microseconds which makes no sense since the calculation should be performed with respect to the least accurate representation (so it should be zero).
And here lies my problem: how can I determine the timestamp accuracy for a storage system?
If I run rclone lsjson . I get results of the form:
“2019-01-30T00:56:23.4890265+01:00”
“2018-01-07T13:28:53+01:00”
so I cannot just look at one entry to determine the accuracy.
Any suggestion on how to proceed? Thanks.

2

That is correct…

Rclone knows the accuracy of each filesystem - it is referred to as the Precision of the backend, and two Precisions come together to make a modify window which is the duration difference to ignore between two times for two backends.

Generally speaking the precision for a remote won’t change. Webdav will always be 1s

What do you think I should do about this?

  • document the precisions in https://rclone.org/overview/ ?
  • add a command to dump the Precision for a remote?
  • add something to the lsf output? A precision field maybe?

Any other thoughts?

3

I would say: add a command to dump general info about the remote, among which the supported precision.
Even better, in my perspective, since I would have to parse the output of the command:
a) allow to obtain a json version of the output (rclone info --json remote:) or
b) allow to filter the fields in output (rclone info precision remote:).
Another option would be to add this info (and maybe other fields) to the output of the about command.
A fast hack would be to always use full precision in dumps, so if the backend supports milliseconds precision dump “2018-01-07T13:28:53.000000+01:00” rather than “2018-01-07T13:28:53+01:00”.
I hope this makes sense…

4

There are some other things I could dump here like supported hashes so this might be a good idea.

Not a bad idea…

Rclone will dump the full precision of the timestamp always, but it uses the least number of decimal places to represent that, eg

$ rclone lsjson /tmp/src/
[
2019/02/02 19:11:36 NOTICE: file: Can't follow symlink without -L/--copy-links
{"Path":"hello","Name":"hello","Size":6,"MimeType":"application/octet-stream","ModTime":"2019-01-27T17:11:08.084924217Z","IsDir":false},
{"Path":"hello2","Name":"hello2","Size":6,"MimeType":"application/octet-stream","ModTime":"2019-02-01T23:24:16.477039975Z","IsDir":false}
]

I could always dump 9 figures of time easily enough, but I’m not sure it helps you.

I was thinking of adding a Precision field would would be the granularity of the timestamp in nS so it would be 1 for local and 1000000000 for webdav backends etc. It might be better in floating point though rather than as an integer.

5

Well, the number of figures that are supported. I mean, if the timestamp is “2018-01-07T13:28:53” I know precision is seconds, if it is “2018-01-07T13:28:53.123000” I know precision is microseconds.

My only concern is about adding it to lsjson: that would be an additional field with the very same value repeated for each entry. As a field of info or about that would be fine.
I don’t know about “the granularity of the timestamp in nS”, that is going to be a power of ten anyway so the number of decimal places says the very same thing and you do not have to deal with loooong numbers.

6

Ah OK! I had a go at that here - what do you think?

https://beta.rclone.org/branch/v1.45-157-ga8d98187-lsjson-precision-beta/ (uploaded in 15-30 mins)

7

Looks cool. I do not have a webdav backend to test, but the behaviour for other backends looks fine.
Can I assume this will be merged in the master branch? Thanks.

8

I’ve merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.46 :smile:

1 Like