--min-age not working for extracted files on a upload script. Any solution?

[tisurame]

Hi. I'm using the Linux version of Rclone and the following script to trigger a move command.

ajkis/scripts/blob/master/rclone/rclone-upload.sh

if find $FROM* -type f -mmin +15 | read

From what I understand, the line above will check if there are files in the folder older than 15 minutes. If not, it will exit (without triggering rclone), if there are files older than 15 minutes, it will execute the following Rclone command:

rclone move "$FROM" "$TO" --transfers=20 --checkers=20 --delete-after --min-age 15m --log-file=$LOGFILE

The reason to avoid files younger than 15 minutes is to wait until files are completely extracted to the folder before moving it to the cloud.

It makes sense, but it doesn't work as intended. Because " --min-age" for Rclone is the last modified time. And when extracting a packed file the last modified time of the file will retain the time when the file was originally packed.

So, it will (almost) NEVER be less than 15 minutes.

Is there any way to overcome this problem?

I thought about modifying the script command that checks if there are any files older than 15 minutes.

if find $FROM* -type f -mmin +15 | read

to

if find $FROM* -type f -amin +15 | read

So it will read the last time the file was accessed (which will be when the file was extracted). It works partially. Unfortunately if you are extracting multiple files it will trigger the move command from Rclone based on the first file last accessed time without giving enough time for the last file to be extracted.

Maybe I could overcome this problem allowing Rclone to only transfer one file per time, but I'm not sure which file it will choose first (the file already extracted or partially extracted).

Of course, I wouldn't have this problem if Rclone could read not only modified time, but also accessed time, but I guess it's not possible for now.

So, any ideas of how to solve this problem?

Thanks!

[ncw (Nick Craig-Wood)]

@tisurame

Is there any way to overcome this problem?

Why not "touch" the files after they've been extracted? That would solve the problem I think.

[tisurame]

@ncw

Why not "touch" the files after they've been extracted? That would solve the problem I think.

Yes. I thought about using two different scripts working at the same time. The first one will only work inside where the files are extracted, and move all files last accessed more than 15 minutes (to ensure they were really extracted). Something like this...

FROM="/home/user/intermediate_folder"
TO="/home/user/final_folder"

find $FROM* -amin +15 -exec mv "{}" $TO ;

The second script would work only where the files were moved to and trigger Rclone.

What do you think?

[ncw (Nick Craig-Wood)]

Separating the extract and upload directories (if I understand you correctly) is the right solution I think.