How to update the Modtime to the upload time?

Help and Support
1

What is the problem you are having with rclone?

When I copy a file from Linux to an AWS bucket the Modtime is the last modification time of my file on Linux. I would like change the Modtime during upload to AWS to the time when the file was uploaded to the bucket.
( similar as when I do the copy of the file to the bucket with the S3 Explorer from AWS).

This is what I have done:
rclone copy -vv test1.txt s3:newsflash2
after the upload the mod date is the old linux date: 2021-02-25
rclone lsf --format "tsp" s3:testbucketnils --include=test2.txt
2021-02-25 11:44:00;51000;test1.txt

Is there a flag that I can use in order to change the moddate during upload to AWS to date when the transfer took place (upload date to the bucket)?

I tried with all kinds of flag, but without any luck:
--update --refresh-times
--update --use-server-modtime
..

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: amazon 2018.03 (64 bit)
  • os/kernel: 4.14.238-125.422.amzn1.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

rclone copy -vv test1.txt s3:newsflash2

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here
2

No rclone.conf
No command you used.
No debug log.

You have the right flag but provided no actual logs or whatnot to help you out :frowning:

--update --use-server-modtime
Uses no extra transactions.
Modification time becomes the time the object was uploaded.
For many operations this is sufficient to determine if it needs uploading.
Using --update along with --use-server-modtime, avoids the extra API call and uploads files whose local modification time is newer than the time it was last uploaded.
Files created with timestamps in the past will be missed by the sync.
rclone sync --update --use-server-modtime /path/to/source s3:bucket
3

Thanks a lot for the support.
Here is the rclone.conf file:
Processing: rclone.conf...
rclone.log (175 Bytes)

This is the log file for the command:
rclone copy -vv --update --use-server-modtime test1.txt s3:newsflash2

$ rclone copy -vv --update --use-server-modtime test1.txt s3:newsflash2
2021/12/08 21:47:00 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "-vv" "--update" "--use-server-modtime" "test1.txt" "s3:newsflash2"]
2021/12/08 21:47:00 DEBUG : Creating backend with remote "test1.txt"
2021/12/08 21:47:00 DEBUG : Using config file from "/home/stonebranch/.config/rclone/rclone.conf"
2021/12/08 21:47:00 DEBUG : fs cache: adding new entry for parent of "test1.txt", "/home/stonebranch/demo/out"
2021/12/08 21:47:00 DEBUG : Creating backend with remote "s3:newsflash2"
2021/12/08 21:47:01 DEBUG : test1.txt: Need to transfer - File not found at Destination
2021/12/08 21:47:01 DEBUG : test1.txt: md5 = 878eaf2d901370e03056b8da9e0d7cc9 OK
2021/12/08 21:47:01 INFO : test1.txt: Copied (new)
2021/12/08 21:47:01 INFO :
Transferred: 49.805 KiB / 49.805 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 1.1s

2021/12/08 21:47:01 DEBUG : 5 go routines active

$ rclone lsf -vv --format "tsp" s3:newsflash2 --include=test1.txt
2021/12/08 21:47:17 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsf" "-vv" "--format" "tsp" "s3:newsflash2" "--include=test1.txt"]
2021/12/08 21:47:17 DEBUG : Creating backend with remote "s3:newsflash2"
2021/12/08 21:47:17 DEBUG : Using config file from "/home/stonebranch/.config/rclone/rclone.conf"
2021/12/08 21:47:18 DEBUG : bigdata.png: Excluded
2021/12/08 21:47:18 DEBUG : index.html: Excluded
2021-02-25 11:44:00;51000;test1.txt
2021/12/08 21:47:18 DEBUG : 4 go routines active

The mod date is still the date of file create in linux from February 25th.
I would like the modedate to be 2021/12/08 21:47:01 - the date when the file was uploaded to AWS S3

4

I'm not a huge S3 user so I could be mistaken, but I thought that is actually not stored in the "normal" spot.

This is where it is stored:

and this one:

5

Thanks for all the infos. So it seems it is not possible what I wanted to achieve.
My goal was to list in my AWS bucket all files which have been uploaded after a certain date. I thought I could use the modtime for this. This will not work, because it seems I cannot update the modtime. Reclone will always show the Linux last modification time ( e.g. the time when I created the file on my Linux server).
The strange think is that if I upload the same file with my Aws explorer ( or boto3) the mod time is the date when I uploaded the file to Aws and not the Linux creation date. This is why I thought it must also be possible with reclone copy to do this.

6

Perhaps @ncw or @asdffdsa might have some thoughts as they are much more familiar than I with S3 as I've never used it.

What happens if you do:

rclone lsf -vv --format "tsp" s3:newsflash2 --include=test1.txt --use-server-modtime

Does that do it?

1 Like
7

hello and welcome to the forum,

as far as i know, objects in s3 are atomic, once uploaded, cannot be modified, including metadata.

if i understand correctly, then rclone can do that.

rclone lsl D:\files\asdffdsamodtime 
        0 1971-01-01 00:00:00.000000000 file.txt

rclone copy D:\files\asdffdsamodtime wasabi01:asdffdsamodtime -v 
INFO  : file.txt: Copied (new)

rclone lsl wasabi01:asdffdsamodtime --use-server-modtime 
        0 2021-12-08 18:01:18.000000000 file.txt

rclone lsl wasabi01:asdffdsamodtime 
        0 1971-01-01 00:00:00.000000000 file.txt

rclone lsl wasabi01:asdffdsamodtime --max-age=24h --use-server-modtime 
        0 2021-12-08 18:01:18.000000000 file.txt

rclone lsl wasabi01:asdffdsamodtime --max-age=24h
8

Hi, really thanks a lot. This exactly what I was looking for. :grinning:

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.