I am a novice with rclone but I use it for simple tasks. I work with weather data (NEXRAD) and I am having an issue finding data, that's why I am here.
The command I use to see the data is : rclone lsf chunks:unidata-nexrad-level2-chunks/KEWX/
So, the data I need is always being written in one of these folders. Is there a way to know what folder has the latest data being written? The files are just one scan of the sky and then it moves to a new folder to write the next data. Therefore I only have a few moments to determine which folder # the data is in.
Is there a way to list by latest file written inside unidata-nexrad-level2-chunks/KEWX/? This would give me the folder #, I would assume.
I'm not an AWS guy but I know on Google, there isn't anything like that to note if a folder/directory has been written to. I doubt that metadata is in AWS but I could be wrong as well.
In Linux, most things have atime (access time) by default turned on so you can usually figure it out.
Thank you. I am playing around with your advice, but still no luck. I can list every file in all 999 folders but that takes an insanely long time. Stumped.
If I run this :
rclone lsl chunks:unidata-nexrad-level2-chunks/KEWX/
It starts listing every single file with the latest time in all 999 folders. It never finishes (Well I am sure it does, but it takes longer than 5 minutes. By the time I finish listing, the radar is already writing files into a new folder.)
You probably want to atleast add --fast-list and see if that helps. The challenge that it doesn't keep any data so you are having to fetch full listings each time.
There is a request at some point to add a local metadata cache as that's in the works and would be great for situations like this. I'd avoid the cache backend since it's a bit buggy sadly as that would be a good fit as well.
Doesn’t Windows change the parent directory date when it writes to it? You could use RCline LSD to get just the directories I think and then parse the time?
The timestamps are all exactly the same, and I know only one is being written to at a time. The active directory changes every few minutes. It looks like its showing the time on my local PC when I executed the command (11:46, it's 11:50 now)
I am a Linux dude so bear with me. Files have an access time, creation time, modified time etc. can you use a Windows command, possibly dir with some option to see the other times? If one gives you what you want you could add that to your script that runs RClone.
Windows or Linux doesn't matter as it's pointing to a cloud remote and not a local file system. The remote has to support access time which I'm not aware of any that do.
Sorry when I saw C:\ I thought you were copying from a local system.
Is the "cloud remote" an S3 compatible bucket? If not, I would look at command tools by your provider to analyze for dates. This is not as much an RClone issue as a source provider question.
S3 compatible bucket files contain a "last modified date" which could be used but I have no idea what capabilities your source data has. If you search all files remotely, yes it will be slow. Here is an example of data S3 can provide you about the object.
The rclone lsd command is essential an aws s3 ls without recursion.
AWS bucket contents are stored by full name. A parent directory is known as a prefix. The prefix doesn't have a date like a Windows or Linux system have.
You may be one level down from where you should be. Your source data should have an API to get you the current file(s). If not you may have to explore how to get that and then figure out if one of the RClone filters will give you the parent prefix.
Also -- if the source is a real-time system you probably want to get the next-to-last item so you don't try and copy something partial that is being created.