How to setup RClone to sync from GCloud Virtual Machine

Nghia_Vu · November 21, 2021, 12:24pm

What is the problem you are having with rclone?

I have been recommended by a friend to try RClone for my task, but I have no experience with it yet. I don't understand the concepts mentioned in documents. I would be thankful for some pointers.

What is your rclone version (output from `rclone version`)

1.57.0

Which cloud storage system are you using?

Virtual machine on Google Cloud computing engine

Task description

I have a server running on a GCloud VM instance (Debian 11). It has a small data folder (~5MB) that needs to be backed up daily (or as close to daily as possible) either as a folder or zipped file. I intend to back it up on my local personal machine (MacBook Pro, Intel 64, Monterey), which I use almost daily.

Below the check list that I think I will need to cover:

Setup RClone on my local MacOS
Config RClone to manually pull from VM
Setup cron job to pull daily from VM, probably 10:0AM daily, and save it to local MacOS with a datetime-based name
[Optional extra] Setup cron job to clear obsolete copies (>7 days old)

The problem is:
a. I'm not sure whether my checklist is suitable for the task
b. If it is, I don't know which guide I should read to start: Which option should I use to install RClone, using homebrew, or binary, or docker? How to setup security to access the VM? How to setup cron jobs? ...

It would be very helpful if somebody could break the checklist down into smaller step-by-step subtask so I could understand exactly what I will need to do. Even more helpful if you can point towards relevant guides.

Thank you!

asdffdsa · November 21, 2021, 3:41pm

hello and welcome to the forum,

first step is to install rclone on macos.
for now use this. at some point, if you want to change the install method, you can do that.
https://rclone.org/downloads/#script-download-and-install

Nghia_Vu · November 22, 2021, 12:06am

Thank you. I have installed it on my MacOS

NOTE: The local-remote connection must be secure enough that only my specific devices can pull from this remote (right now, only my local Mac)

I couldn't find a way to edit the original post

asdffdsa · November 22, 2021, 12:11am

just as well, as re-editing old posts get confusing.

the next step is to get the rclone to access the files in the vm, correct?

Nghia_Vu · November 22, 2021, 2:07am

Yes
I have configured a remote which looks something like this

[remote_namp]
type = sftp
host = 11.22.33.44 # (not real)
user = mac19_rclone
key_file = ~/.ssh/id_rsa
pubkey_file = ~/.ssh/id_rsa.pub

But rclone sync threw error "public key file is not a certificate file"

A section in the doc says
If you have a certificate you may use it to sign your public key, creating a separate SSH user certificate that should be used instead of the plain public key extracted from the private key. Then you must provide the path to the user certificate public key file in pubkey_file.

And I have no idea how to create the certificate and sign my existing key.
If a new private-public pair must be generated (if the existing key cannot be used), then it must be acceptable to Google Cloud VM (I tried using a .PEM key but it apparently was the wrong format)

asdffdsa · November 22, 2021, 1:45pm

need to post the full debug log.
not an expert at sftp keys but here is an example of a working remote.

[rsync]
type = sftp
host = xxx.rsync.net
user = xxxx
key_file = key_file = C:\data\c\bitvisesshclient\rsync\id_rsa
md5sum_command = md5 -r
sha1sum_command = sha1 -r

and here is what the redacted authorized_keys looks like

Nghia_Vu · November 26, 2021, 12:06pm

Today I've successfully did my first cloning, using SFTP connection and authentication with ssh-agent.
The next task is cron job for daily backup, and optionally cronjob to clean up old backups.

Nghia_Vu · November 27, 2021, 1:26am

Before the cronjob, which I suspect wont be a problem. I'm looking for a way to save the downloaded folder as a file, such as zip, or any other format.

asdffdsa · November 27, 2021, 1:41am

might write a script, to use ssh to create a zip file on the vm and use rclone to download it.
there are a number of posts in the forum about using rclone cat/rcat and tar.
create a rclone mount of the sftp remote and use any zip program on the macos

system · December 27, 2021, 1:41am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

How to setup RClone to sync from GCloud Virtual Machine

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which cloud storage system are you using?

Task description

What is your rclone version (output from `rclone version`)