rclone+GDrive: Native "Backup and Sync" experience

[stdedos]

What is the problem you are having with rclone?

I want to know how can I get native "Backup and Sync" experience (i.e. latency-free, always-sync) on my Ubuntu.

I have a repository that I am syncing with GDrive (pls don't shame, I know what I do is wrong).
I can use WSL+git on top of the GDrive folder very comfortably (ofc, we assume that only one side is being edited at all times, so there are no sync-collisions).
A simple git status takes 10 seconds on Ubuntu+rclone (on --vfs-cache-mode full).

I tried both

curl https://rclone.org/install.sh | sudo bash
rclone config
rclone ls gdrive:
rclone mount gdrive: /media/usr/gdrive
rclone mount --daemon gdrive: /media/usr/gdrive
rclone mount --daemon --attr-timeout 1m --vfs-cache-mode full gdrive: /media/usr/gdrive

If what I see is expected, at least let me know so I don't worry.

What is your rclone version (output from rclone version)

$ rclone --version
rclone v1.53.2
- os/arch: linux/amd64
- go version: go1.15.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18.04

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

N/A

The rclone config contents with secrets removed.

(Followed the tutorial with my own API key)

A log from the command with the -vv flag

N/A

[ncw (Nick Craig-Wood)]

@stdedos

I want to know how can I get native "Backup and Sync" experience (i.e. latency-free, always-sync) on my Ubuntu.

Using --vfs-cache-mode full is the closest you will get to it at the moment...

@stdedos

A simple git status takes 10 seconds on Ubuntu+rclone (on --vfs-cache-mode full).

I tried this on a git repo I have on Google drive. The first git status took about 2 minutes and seemed to open and read every file in the checkout! A subsequent run took less than 1 second.

Other operations were similar (eg git log).

So I think it is about getting the cache hot...

[stdedos]

I tried git status consecutively multiple times, until I convinced myself that I will open a topic about it. git add was improbable that it would happen (.git/index.lock exists)

I know that it takes "10 seconds" because on the last (of n) invocation I did /usr/bin/time git status instead, and not because I counted it by a clock manually. I did it like 2-3 times, and then I lost interest - if something forces me to stop and busywait; basically I cannot make progress at all.

I know what I ask might be too time-consuming, no-benefit for you; however: Can there be way to contact to a rclone --daemon (or the non-daemon variant) that would show in an fancy-terminal, the total number of queue size, its progress, and the operations individually (files [local or remote path]/API URL (?)/size/percent-done/ETA)? Or something like progress -wM-equivalent (https://github.com/Xfennec/progress)?

I know for you a -vv output makes more sense, but for me it's a waste of deciphering time e.g. a list of 200 lines saying that (on an empty local directory) it would've created all the files, but it didn't do because --dry-run is active.
Instead, my proposal gives me much more input on "what is happening", even if I cannot do anything about the delays.

[ncw (Nick Craig-Wood)]

@stdedos

I know what I ask might be too time-consuming, no-benefit for you; however: Can there be way to contact to a rclone --daemon (or the non-daemon variant) that would show in an fancy-terminal, the total number of queue size, its progress, and the operations individually (files [local or remote path]/API URL (?)/size/percent-done/ETA)? Or something like progress -wM-equivalent (https://github.com/Xfennec/progress )?

If you run rclone with the --rc flag then you can use the remote control commands here: Remote Control / API

You probably want the core/stats call. That will give JSON output but could be used to build something nicer.