I am looking for beta testers on the rclone interface for my file-sync script. It is the rclone branch of PyFiSync. And specifically see the rclone readme and the B2 Setup Example (which would be the same for S3 with minor, noted, modifications).
I originally made the script for syncing macOS/Linux/Unix computers so a lot of design decisions were around that. I just added rclone as a remote so a lot of logic is repeated of what rclone can do (including exclusions, etc).
Some nice features are:
- move tracking when possible. Works especially well on macOS with inodes and file-create time. Works pretty well on Linux with inodes. When set up, works well with rclone hashes
- Safe operations. Before anything is written, it is backed up. This is totally needed with the way some bucket-based systems work but it is a very nice feature to have anyway
- Safe from interups. The sync is not atomic by any means but I carefully enumerate all situations in the FAQs
- Extensive testing. Every sync situations and edge case (and file tracking adds a ton) that I could think of is considered and tested. The rclone remote does have a few known issues but they have minor side affects. I could, of course, have missed some so let me know if you think of any
Where I need help:
- Real-world testing. While I’ve read the rclone documentation extensively, I still may have missed a quirk with some remotes or even different local systems (I tested it much more on macOS than linux for example). While I considered a lot in the test case, There is no match for battle-testing.
- Different remotes
- Real-world usage (of the rclone remote. I have extensively used the rsync one for years)
- Developer input. I started this project when I was earlier in my python learning. I have become a much better programmer but I don’t have the time to refactor the whole project.
What’s Missing:
- I think this is nearly finished. The biggest thing I am missing is unifying the rclone and the rsync documentation
- I have not yet deprecated anything with git-exclusions and they do not work with rclone
- Because this is not an rclone-first tool, it is only designed to work with the remote machine being rclone. It does not work rclone-remote to rclone-remote (nor do I plan to support this)