Good day, I have been using rclone for a while now to sync backups to a storj endpoint. Some files started to fail and not syncing, but every time the syncs run again it retries the failed files. Is there a way to skip the failed files next time the rclone sync is run? Thanks in advance.
hello and welcome to the forum,
https://rclone.org/filtering/#exclude-from-read-exclude-patterns-from-file
Thank you. I have looked into this option but that means I have to provide the list of failed files to exclude. Is there a way to output the failed files from a sync to a text file that I can use then to exclude from the next run? As there are about 10,000 files that fails out of the 1million, because of special characters etc being dropped to upload to the S3 bucket. Thanks in advance.
for that sync, there will be a number of errors in the log file.
write a script to read the log file, extract the bad filenames into a text file and then use an exclude filter
i do a similar thing using a python script.
the scripts scans the log file looking for ERRORS and some other text.
it writes that info to a text fie and emails it to me.
perhaps try something like this
https://forum.rclone.org/t/raw-filename-exclude-best-way-to-exclude-specific-list-of-complex-file-names/26571/4
Great! Thank you, I will give that a try.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.