I use Rclone to mount my IDrive Cloud S3 compatible storage in Windows 10, 64 bit. I utilize this S3 storage as a network share for my family and maintain a backup (used to be the original source) on a desktop hard drive. I recently noticed that last modified dates on the cloud are not retained in the S3 objects file properties - this date reflects the date of upload onto the cloud, not the last modified date if you were to copy a file in the file system. Understand this is an S3 property. Does Rclone mounting and then copy/paste or the Rclone sync commands keep the original file's last modified intact in metadata? I'd like to both effectively compare the S3 object to the original file on the hard drive for backup/sync purposes and also if I had to restore these files from S3 to a file system have this date correct once back off S3 for sorting files.
Rclone version 1.56 Windows 10, 64bit
Command line: .\rclone.exe mount :/ S:
yes, by default rclone does that. https://rclone.org/docs/#use-server-modtime
"Some object-store backends (e.g, Swift, S3) do not preserve file modification times (modtime). On these backends, rclone stores the original modtime as additional metadata on the object."
Here is how rclone stores the modtime - you'll be able to see it in the metadata for the object
Modified time
The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch accurate to 1 ns.
If the modification time needs to be updated rclone will attempt to perform a server side copy to update the modification if the object can be copied in a single part. In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archive storage the object will be uploaded rather than copied.
Note that reading this from the object takes an additional HEAD request as the metadata isn't returned in object listings.
Thank you @ncw & @asdffdsa for your replies. I got the same results that the modified date was retained utilizing both the command line as described and the windows GUI after mounting the drive.
I used Cyberduck for the majority of the 1.5 TB upload for initial seeding of my S3 storage, but it appears all date modification data was lost doing so. I attempted to download a few of the files utilizing Cyberduck and the modified date was still the upload date, not the source date.
I love rclone for the functionality - being able to mount my cloud storage so it is visible in Windows. Rclone obviously is able to work for copying a few files and folders to preserve dates, but can it handle a copy of a folder containing hundreds of thousands of files and TB's of data?
My aim is to seed the S3 cloud from a local disk source. Then utilize the S3 cloud drive as the primary across multiple devices, "streaming" the data. Then have a backup/sync schedule whether daily or weekly back to a local disk source to ensure a local copy is kept. Do you have software recommendations for either the initial seeding (I'm guessing I will need to upload the data again) and sync/backup?
there is a flag, --refresh-times that might help, tho not sure if it works well with s3.
as i have never used it, ncw can comment on that.
yes, i have a folder in wasabi, a s3 clone, with a set of folders/subfolders with over 1,000,000 files.
if the source and dest are exactly the same, so that no files needs to be copied, rclone sync takes 33 seconds to traverse the local and remote file systems.
and there are other rcloners with much large remotes.
Empty current bucket with non-date modified metadata.
rclone.exe copy the data back out to S3 from local drive (wait ~2 days while this completes)
Work in the cloud. Make mount the source, local drive the destination and run rclone.exe sync periodically to keep local data up to date.
I currently have a drive mounted with no VFS caching. For the setup (ex. editting an Excel file on the cloud) to work properly, it appears I need to use "--vfs-cache-mode writes". What happens though if I save the file, and put my computer asleep immediately? This could mean no write back to the mount correct? Or does rclone check for that upon waking and use and write old cached data that wasn't written yet to the mount, even if it's been a day or two days since the computer was powered on?
what is the total data size of all the files that you plan to store in cloud?
as i do not use rclone mount to edit excel and other such files, i cannot tell you the details.
in any office/location i deal with, there is always a file server and/or NAS.
sometimes that is local, sometimes it is over vpn.
i use veeam backup and rclone sync to backup that data to the cloud.
recent backups in stored in wasabi s3 and older backups are stored in aws s3 deep glacier.
