Bi-directional rclone solution


#1

Hello all,
I have written a bi-directional rclone sync wrapper in Python 2.7. My motivation was to have a stable method of syncing both services to a LAN share (Linux samba) since Google doesn’t offer a Linux client and Drobox’s Linux client quit on me recently (probably wants me to update), and there are hassles when using network drive spaces with their Windows clients.

It seems to work well on Google Drive and Dropbox. I have not tested it for other services. I have it setup as a cron jobs to run every 30min to sync both services to a Samba share drive on my LAN.

Where best to post this? I will post it to my personal Github space, unless it makes sense to post it within the rclone github space.

Key behaviors / operations
• Keeps an rclone lsl file list of the Local and Remote systems. On each run, checks for delta on Local and Remote
• Applies Remote delta to the Local filesystem, then rclone syncs the Local to the Remote file system.
• Handles change conflicts nondestructively by creating _LOCAL and _REMOTE versions
• Somewhat fail safe (lock file to prevent multiple simultaneous runs when taking a while, and file access health check)

Known bugs/limitations/pending enhancements:
# Cannot directly compare timestamps on remote and local. Modification time not retained when a file is transferred.
# rclone sync pushes files from local to remote based on size differences. If a local file is changed but same size it wont be pushed.
# the try/except blocks around rclone are not effective for catching rclone errors. It seems rclone always exits with 0 status
# Size difference identified but not handled
# Older version found but not handled
# If same timestamp file is added to both cloud and local then ???

[xxx@xxx RCloneSyncWD]$ ./RCloneSync.py --help
2017-08-05 12:11:49,328/RCloneSync//WARNING: ***** BiDirectional Sync for Cloud Services using RClone *****
usage: RCloneSync.py [-h] [–FirstSync] [–ExcludeListFile EXCLUDELISTFILE]
[–Verbose] [–DryRun]
{Dropbox:,GDrive:} LocalRoot

***** BiDirectional Sync for Cloud Services using RClone *****

positional arguments:
{Dropbox:,GDrive:} Name of remote cloud service
LocalRoot Path to local root

optional arguments:
-h, --help show this help message and exit
–FirstSync First run setup. WARNING: Local files may overwrite
Remote versions
–ExcludeListFile EXCLUDELISTFILE
File containing rclone file/path exclusions (Needed
for Dropbox)
–Verbose Event logging with per-file details (Python INFO level
- default is WARNING level)
–DryRun Go thru the motions - No files are copied/deleted


#2

Put it on your personal github and put a link to it in the rclone wiki in the 3rd party integrations section.

Rclone should exit with non zero status when there are errors. If it isn’t can you please make a new issue on github with an example please?

You might want to look at rclone lsjson for file listings with more info in a program readable format.

You could consider porting this to Go and making it a new rclone command?


#3

Thanks ncw. See the rclone wiki in the 3rd party integrations section.

Regards,
cjn


#4

Updated version posted - Reworked error checking to be functional, and cleaned up a couple issues related to file renaming (_LOCAL, _REMOTE), and added the --CheckAccess switch. See 3rd party integrations or RCloneSync on GitHub.

I’ve received a few requests for the Remote/cloud side to accept a path in order to support synching just a partial tree. More interest in this?


#5

Undated “final” version posted. https://github.com/cjnaz/RCloneSync

  • A path is now supported on the remote/cloud, which allows synching individual targets to subdirs on the remote/cloud side.
  • rclone commands now have 3x retries to improve fault tolerance.
  • Beautified the --Verbose mode output for readability, and split out rclone verbose mode switch control.

#6

The discussion re born at https://github.com/ncw/rclone/issues/118