Automating Bisync to gDrive accounts on Linux, for complete dummies

I had meant to post this as a follow-up to my question here, but was too slow and it got closed. Basically, I'm a new migrant to Linux from Windows, and wanted to duplicate the functionality of Google's native drive-syncing app. With the help I got elsewhere on this forum (thanks Ole Frost!), I was able to get it all working well, and I'm documenting what I did here, in case it helps any equally clueless newcomers.

1. Set up the first Remote:

  • First remote was created using the “rclone config” command in a terminal window.

  • The remote was named “gdrive-[XXX]1

  • All default options were accepted, except:

    --Chose “1 Full access” for scope
    --(under advanced config) Chose “true” for “skip_gocs”
    
  • An empty folder /media/[USERNAME]/Disk2/gdrive-[XXX]1 was created on a local drive.

  • All files in the target google drive directory were BACKED UP.

  • Manually ran:

    rclone bisync --resync gdrive-[XXX]1:/ /media/[USERNAME]/Disk2/gdrive-[XXX]1
    
  • (resync runs usually took 5-10 minutes)

  • Verified everything synchronized correctly.

  • Created/tested the script rclone_batch.sh for just the one sync pair (script given below*).

  • After creating, gave user account execute privileges by running “chmod u+x rclone_batch.sh

2. Set up additional Remotes:

  • Repeated steps above to create/test second and third remote+local pairs.

  • Note: After creating a first remote, the menu called by “rclone config” offers a new option for creation of additional remotes.

  • Added an entry in rclone_batch.sh for the additional sync pairs.

  • Tested/verified all working.

3. Set up automation:

  • Set up a cron job to run the script every 15 minutes by adding the following line to my crontab:

    */15 * * * * /home/[username]/bin/rclone_batch.sh
    
  • Similarly set up cron job to run script at boot by adding this line:

    @reboot /home/[username]/bin/rclone_batch.sh
    
  • Added an entry to my .bash_logout script to get a final synchronization at logout:

    /home/[username]/bin/rclone_batch.sh
    

4. Added my own Client ID and Client Secrect

  • Followed directions (at the end of the “Google drive” documentation page) to create a client_id and client_secret. (much of what I did here was unfamiliar to me)

  • Ran rclone config again, chose to edit my first (of the 3) remotes, and pasted in the id and secret at the appropriate prompts. This triggered another google login page in my browser, which came with a warning, and I had to click an "Advanced” option in order to proceed.

  • Tested/verified this setup.

  • Edited the /home/[username]/.config/rclone/rclone.conf file, copy/pasting the client_id and client_secret entries into appropriate places for the other remotes.

5. Things I chose not to do:

  • Encrypt the config file:
    I decided I’m too lazy to enter my credentials every time rclone runs. I will instead learn how to maintain good security on my Linux box in general.

  • Generate alerts when rclone encounters an error:
    Given that rclone can be made to send messages to syslog (the –syslog and –verbose options), I will instead set up a cron job to monitor syslog and make me aware of (any) new errors during my session.

  • Preclude long-running calls to rclone from colliding with one another:
    I stopped pursuing this when I learned that rclone already handles this issue, as documented in the “Lock File” section of the “Bisync” documentation page.

*rclone_batch.sh:

#!/bin/bash

path1=gdrive_[XXX]1:/
path2=/mnt/Disk2/gdrive_[XXX]1/
rclone bisync --syslog --verbose $path1 $path2 &

path1=gdrive_[XXX]2:/
path2=/mnt/Disk2/gdrive_[XXX]2/
rclone bisync --syslog --verbose $path1 $path2 &

path1=gdrive_[XXX]3:/
path2=/mnt/Disk2/gdrive_[XXX]3/
rclone bisync --syslog --verbose $path1 $path2 &

1 Like

You are welcome, very happy to hear you found a good solution and got it working well!

I like your very well structured guide and your paying attention to all the small details that make (even experienced) users stumble. Being mostly on Windows, I still now and then stumble in important details like “chmod u+x rclone_batch.sh

Thanks a lot for your contribution to the community! :slight_smile:

Nope...nope! Getting a whole mess of errors now. The guide above is flawed somehow. I'll revise it and update this post as soon as I figure out what's going on.

EDIT - As kapitainsky pointed out below, this is the wrong approach and the script below should not be used.
For me it works just fine after fixing the script by adding the --resync option.
So my new script looks like this:

#!/bin/bash

path1=gdrive_[XXX]1:/
path2=/mnt/Disk2/gdrive_[XXX]1/
rclone bisync --resync --syslog --verbose $path1 $path2 &

path1=gdrive_[XXX]2:/
path2=/mnt/Disk2/gdrive_[XXX]2/
rclone bisync --resync --syslog --verbose $path1 $path2 &

path1=gdrive_[XXX]3:/
path2=/mnt/Disk2/gdrive_[XXX]3/
rclone bisync --resync --syslog --verbose $path1 $path2 &

And before the automatic script worked once again I needed to run it manually once.

This option is not supposed to be used every time you run rclone bisync. If you run it every time you might lose data...

Only once, first time - see point 1 in bnc guide. He runs --resync manually as a part of the setup.

Oh, okay. Thank you for pointing it out. I will edit my answer above, not to have others use this wrong modification.

1 Like

@bnc maybe you could make it 100% clear that --rsync is needed for every path user bisync

e.g.:

rclone bisync --resync gdrive-[XXX]1:/ /media/[USERNAME]/Disk2/gdrive-[XXX]1
rclone bisync --resync gdrive-[XXX]2:/ /media/[USERNAME]/Disk2/gdrive-[XXX]2
rclone bisync --resync gdrive-[XXX]3:/ /media/[USERNAME]/Disk2/gdrive-[XXX]3

so this step is consistent with your rclone_batch.sh

Hey there! I'm glad to see this was of interest to someone. The script in the OP definitely had flaws (it never worked right for me), and life has dragged me away from that project, to the point I don't even remember how it all worked! I think some of the errors I was getting had to do with Linux drive mounting...I can't remember. In any case, please take whatever is useful and ditch the rest. Sorry for leaving half-finished work on the forum!

For any future reader here. I have written a guide google drive sync using systemd and rclone.
Hope you like it. I am using it for couple of months.

1 Like

Very nice guide :slight_smile:

By the way, I noticed this part:

Note that we are skipping Google docs . Because rclone has yet to resolve the issue.

and just wanted to let you know that I have submitted a fix for it. Hopefully it will be merged soon :slight_smile:

1 Like