Synchronize a directory with an aws bucket but have it start syncing the newest file to the oldest.

Hi, I want to sync a directory with an aws bucket but have it start syncing the newest file to the oldest.
It's possible ?

It is currently not possible to sort the transfer list - sorry :frowning:

I am hoping to eventually push Nick to look into this because I have some ideas for it. Otherwise it's one of those coding tasks I might consider cutting my teeth on, because it should not be that hard to do (go already has the sorting functions needed after all).

The best alternative I can give you that works right now is one of these two...

(1) many commands with different tiers of --max-age in a script
rclone copy C:\SourceFolder AWS:\DestFolder -- max-age 4h
rclone copy C:\SourceFolder AWS:\DestFolder -- max-age 8h
rclone copy C:\SourceFolder AWS:\DestFolder -- max-age 12h
rclone copy C:\SourceFolder AWS:\DestFolder -- max-age 16h
rclone copy C:\SourceFolder AWS:\DestFolder -- max-age 20h
rclone copy C:\SourceFolder AWS:\DestFolder
(the last one doing "everything else")

This is messy and a little inefficient, but largely works I suppose...

(2) The better more advanced way, but also more complex
rclone lsf C:\SourceFolder --log-file=MyTransferlist.txt
Now we have a list of all files to be transfered
You can now sort this in some other program (many alternatives)
then...
rclone copy C:\SourceFolder AWS:\DestFolder --files-from "C:\rclone\MyTransferlist.txt"

This will accomplish your goal, and it's not that hard to script.
Actually, I'd love to find a good compact sorting exe for this on windows to make use of this until it can be integrated into rclone. So if you are on windows - let me know if you find something suitable. On Linux there should be loads of good commandline tools for the job already. Probably even suitable ones installed as stock - but I am not experienced enough to name the exact names...

Actually, now that I think about it - you would have to use rclone lsd to get the modtime data to sort by - instead of lsf (only files and directories).

then trunate away that info in script before using it in --files-from
maybe --files-from can handle the lsd format directly too , but I have not tested it

Thanks thestigma, I am using linux as there are over 5 million objects I will analyze option 2 already gave me a way forward.

Another correction (sorry):
not lsd, but lsl
I sometimes confuse these still
:slight_smile:

Did our discussions get made into an issue? I can't find it!

You probably don't want rclone lsd/lsl - if you only want files or directories with lsf you can use --files-only and --dirs-only

Oh, please tell me that lsl is not incompatible... I just scripted a lsl sorting function to sort by modtime and size lol. As far as I could see this was the only format the included modtime and size which I needed for the sorting.

Will I need to cull the fields to make the format like LSF (files only) for it to be readable with --files-from ? I guess I really should have checked this BEFORE programming haha.

No, I guess I never specifically asked you if you wanted one - and you didn't ask me, so I suppose it never happened. I pinged the relevant conversation for you though.
And in that conversation you did link another issue for sorting in general (which would be the basis for my own suggestion - and useful in it's own right)

The gist of my suggestion was just a special type of sorting where you would interleave the biggest with the smallest from the top (maybe favoring small files 2-to-1or 3to-1 ideally?). The benefit would be higher average throughput in mixed transfers for all backends that have limited amount of files that can be opened pr second (like Gdrive). I suppose the even more optimal method would be to dedicate 1 transfer to the big files - then let other transfers work from smallest to largest until they meet. Would be more efficient than a set ratio.

The only additional thing that this would require on top of "general sorting" is that it may need 2 lists if you want it to be optimized for speed (your suggestion), although a simple sorted insertion would probably work too - albeit slower.

Do you want me to create an issue for it? Or perhaps just make a note in the existing sorting thread to accommodate for the BIG/small sorting method specifically?

You can cut the bits off the lsl output you don't need so if you do that it will work just as well as lsf.

I'm traveling at the moment so can't look up the issue. Can you link it?

You probably saw this in the PM thread I pinged, but here it is again just in case:

And safe travels by the way! :slight_smile:

1 Like