yes that can work, since there are millions of files i wanted to see if there is solution that basically retry the copy few times, then when it successfully copy(to cache) it then upload it to destination.
with the 2 batch approach i will have to constantly create new batch files, and scan each batch logs for errors
I mean move ALL files to hot storage before running rclone.
Doing rclone retries you will choke it to death if run with full speed as you said you do not know how long restorations takes - seconds or minutes. Or it will be mega slow if you wait minutes between retries - will never finish for millions of files.
That is the challenge "move ALL files to hot storage before running rclone"
once its in hot storage its basic rclone usage
i might need to code a solution that will copy files to hot storage(copy-wait-verify file exists), once file is copied (downloaded) then i can run rclone
its just dont give me much advantages to use Rclone, if i copy(download) the file to local hot storage/local disk then i can now just write it to S3
its called DiskExtender (data stored in Tapes, for windows its look like \server\folder\file.cad ) the file is a pointer/link
once you try to copy/open it the system will go fetch the file
i think its same as Dropbox/OneDrive etc... when file is offline
I think you have to find a way to move deep to hot storage files sequentially as they are stored on tapes. So get all data from tape1 to hot - then rclone it to your preferred destination. Move to tape2.
Doing it randomly is not going to work for millions of files given tape latency. And most likely will kill tapes/drive.
So it requires some knowledge of how DiskExtender works in details. IMO you can not achieve all using rclone only.
yes you are correct, i do have list of the files and can sort it by order i want (i.e tape)
so i can build a batch file per tape
was just looking for a tool that i can run for this step 1 (deep to hot storage) and wait for step 1 to complete
then i can run step 2 rclone upload to S3
Talk to this tape solution company - they are only people who know what is the best way to do what you want. You might have to pay them to provide customised solution. No idea what your contract with them is.
rclone cloud storage transfer is the least problem here.
can Rclone do this? the try copy+wait
i will not run it on million of files but will split it to many batches
once batch complete download
will run batch to upload
so the tape reading can rest
they said i can try to copy/read 30 files in parallel without issue