I have set up a personal backup solution using rclone and backblaze B2. RClone is a great tool and I'm very happy about it. But there is a problem which I have not managed so far.
My backup strategy is a file based solution (python script) where the hierarchy and filenames of the local folder are eliminated and each file is renamed and crypted. Data restore is done using a database which was created during backup. This works very nice. The only problem is deleting files in the remote folder. Deletion is done per-file. rclone delete hence reads the remote folder dir every time. The remote folder is very large, hence the operation takes appr. 1min. Which means, that only one file can be deleted per minute.
Is there any solution to keep the listing for deleting many files? Is there another way? Of course I could store a local copy of the remote bucket and use only a sync operation. May be this is faster? But much less elegant.
Many thanks in advance...
hmm, may be this is an option. I'll implement this and try.
My problem is, that I have no idea, where the long duration comes from. In the verbose output it hangs nearly 2mins with the statement "Waiting for deletions to finish". I'm not sure whether this can be solved with the list file. Because deletion starts fast and than hangs.
once again me. There seems to be a significant difference in time using delete or deletefile. deletefile is much faster. I do not understand why, but perhaps this is the solution.
If you add -vv --dump headers that will give you an idea as to what transactions rclone is doing with b2 during that 1 minute period. You can check the API docs to see what the calls mean, though the path names are pretty self explanatory.
thanks for this advice as well. I'll do a new backup run using deletefile instead of plain delete. If this is not the solution I'll try your suggestion. Thanks a lot for your support!