Background: see my previous question. While the challenge to accelerate rclone copy remains, I start a new thread as it is a distinct question. Test case: 10 k files on the server, 1 file modified (i.e. touched). The test case is approx 1/30 of my real use case.
I looked into three options. See outputs below for details.
- direct rclone copy from local to encrypted gdrive - 12 minutes
- rclone mount --vfs-cache-mode full plus rclone copy from local to mountpoint - 45 minutes
- script to identify outside of rclone the changed files via sqlite, then upload only those (rclone copy --files-from) - less than 30 seconds.
What is the problem you are having with rclone?
Do I miss flags? Is there a more elegant, built-in way to have a fast discovery and copy of the modified files?
What is your rclone version (output from rclone version
)
v1.55.0
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux: Ubuntu 20.04 LTS, 64bit
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
## Option 1 - copy to encryptgdrive - 12 minutes
$ touch /home/xxx/xxx/SourceC/Wein/jahrgaengemodel.cpp
$ date ; ./rclone --version ; ./rclone --progress --skip-links --filter-from ~/.config/rclone/buero-backup.filter copy / encryptgdrive:buero ; date
Sa 03 Apr 2021 13:22:57 CEST
rclone v1.55.0
- os/type: linux
- os/arch: amd64
- go/version: go1.16.2
- go/linking: static
- go/tags: cmount
Transferred: 8.624k / 8.624 kBytes, 100%, 6.442 kBytes/s, ETA 0s
Checks: 10186 / 10186, 100%
Transferred: 1 / 1, 100%
Elapsed time: 11m28.1s
Sa 03 Apr 2021 13:34:26 CEST
## Option 2 - mount --vfs-cache-mode full, then copy from local to mount - ~45 minutes
$ touch /home/xxx/xxx/SourceC/Wein/jahrgaengemodel.cpp
Terminal 1: $ ./rclone mount --vfs-cache-mode full encryptgdrive:buero /mnt/usbdisk/
Terminal 2: $ date ; ./rclone --version ; ./rclone --progress --skip-links --filter-from ~/.config/rclone/buero-backup.filter copy / /mnt/usbdisk/ ; date
Sa 03 Apr 2021 13:56:40 CEST
rclone v1.55.0
- os/type: linux
- os/arch: amd64
- go/version: go1.16.2
- go/linking: static
- go/tags: cmount
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 3270 / 10186, 32%
Elapsed time: 15m10.5s
>> then terminated by Ctrl-C
## Option 3 - home-made offline check for modified size or time - <30 seconds
$ sqlite3 ~/Heimnetzwerk/BackupConfig/rclone_testfilelist.db ".schema"
CREATE TABLE localfiles(filesize INTEGER NOT NULL, filetime TEXT NOT NULL, filepath TEXT NOT NULL UNIQUE);
CREATE TABLE remotefiles(filesize INTEGER NOT NULL, filetime TEXT NOT NULL, filepath TEXT NOT NULL UNIQUE);
$ touch /home/xxx/xxx/SourceC/Wein/jahrgaengemodel.cpp
$ date
Sa 03 Apr 2021 13:38:36 CEST
$ ./rclone --version
rclone v1.55.0
- os/type: linux
- os/arch: amd64
- go/version: go1.16.2
- go/linking: static
- go/tags: cmount
$ >&2 echo "clean the database"
clean the database
$ sqlite3 ~/Heimnetzwerk/BackupConfig/rclone_testfilelist.db "DELETE FROM remotefiles; DELETE FROM localfiles;"
$ >&2 echo "load list of remote files (with size and time)"
load list of remote files (with size and time)
$ ./rclone lsf --format "stp" --csv --recursive --files-only encryptgdrive:buero | sqlite3 -csv ~/Heimnetzwerk/BackupConfig/rclone_testfilelist.db ".import '|cat -' remotefiles"
$ >&2 echo "load list of local files (with size and time)"
load list of local files (with size and time)
$ ./rclone --skip-links --filter-from ~/.config/rclone/buero-backup.filter lsf --recursive --csv --files-only --format "stp" / | sqlite3 -csv ~/Heimnetzwerk/BackupConfig/rclone_testfilelist.db ".import '|cat -' localfiles"
$ >&2 echo "identify the files requiring update"
identify the files requiring update
$ sqlite3 ~/Heimnetzwerk/BackupConfig/rclone_testfilelist.db "SELECT l.filepath FROM localfiles l LEFT JOIN remotefiles r ON l.filepath=r.filepath WHERE l.filesize != r.filesize OR r.filesize IS NULL OR l.filetime > r.filetime" > /tmp/changedfiles
$ >&2 echo "upload these files:"
upload these files:
$ >&2 cat /tmp/changedfiles
home/xxx/xxx/SourceC/Wein/jahrgaengemodel.cpp
$ ./rclone copy --skip-links --progress --no-traverse --files-from /tmp/changedfiles / encryptgdrive:buero
Transferred: 8.624k / 8.624 kBytes, 100%, 4.140 kBytes/s, ETA 0s
Checks: 1 / 1, 100%
Transferred: 1 / 1, 100%
Elapsed time: 4.2s
$ date
Sa 03 Apr 2021 13:39:02 CEST
The rclone config contents with secrets removed.
[mygdrive]
type = drive
scope = drive
export_formats = odt,ods,odp,svg
token = {"access_token":"xxx","token_type":"Bearer","refresh_token":"xxx","expiry":"2021-04-03T14:20:08.76658435+02:00"}
root_folder_id = xxx
client_id = xxx.apps.googleusercontent.com
client_secret = xxx
[encryptgdrive]
type = crypt
remote = mygdrive:Backups
filename_encryption = standard
directory_name_encryption = true
password = xxx
password2 = xxx
A log from the command with the -vv
flag
$ ./rclone -vv --log-file /tmp/rclone-direct.log --progress --skip-links --filter-from ~/.config/rclone/buero-backup.filter copy / encryptgdrive:buero
$ cat /tmp/rclone-direct.log
2021/04/03 14:17:16 DEBUG : Using config file from "/home/xxx/.config/rclone/rclone.conf"
2021/04/03 14:17:16 DEBUG : rclone: Version "v1.55.0" starting with parameters ["./rclone" "-vv" "--log-file" "/tmp/rclone.log" "--progress" "--skip-links" "--filter-from" "/home/xxx/.config/rclone/buero-backup.filter" "copy" "/" "encryptgdrive:buero"]
2021/04/03 14:17:16 DEBUG : Creating backend with remote "/"
2021/04/03 14:17:16 DEBUG : local: detected overridden config - adding "{xxx}" suffix to name
2021/04/03 14:17:16 DEBUG : fs cache: renaming cache item "/" to be canonical "local{xxx}:/"
2021/04/03 14:17:16 DEBUG : Creating backend with remote "encryptgdrive:buero"
2021/04/03 14:17:16 DEBUG : Creating backend with remote "mygdrive:Backups/xxx"
2021/04/03 14:17:16 DEBUG : var: Excluded
2021/04/03 14:17:16 DEBUG : proc: Excluded
2021/04/03 14:17:16 DEBUG : cdrom: Excluded
2021/04/03 14:17:16 DEBUG : dev: Excluded
2021/04/03 14:17:16 DEBUG : lost+found: Excluded
2021/04/03 14:17:16 DEBUG : swapfile: Excluded
...
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/femtomail/femtomail.c: Size and modification time the same (differ by -286.486µs, within tolerance 1ms)
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/femtomail/femtomail.c: Unchanged skipping
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/femtomail/README.md: Size and modification time the same (differ by -286.486µs, within tolerance 1ms)
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/femtomail/README.md: Unchanged skipping
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/femtomail/read_local_mail.sh: Size and modification time the same (differ by -286.486µs, within tolerance 1ms)
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/femtomail/read_local_mail.sh: Unchanged skipping
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/InputSequence/.git/COMMIT_EDITMSG: Size and modification time the same (differ by -152.867µs, within tolerance 1ms)
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/InputSequence/.git/config: Size and modification time the same (differ by -319µs, within tolerance 1ms)
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/InputSequence/.git/config: Unchanged skipping
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/InputSequence/.git/index: Size and modification time the same (differ by -617.887µs, within tolerance 1ms)
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/InputSequence/.git/COMMIT_EDITMSG: Unchanged skipping
2021/04/03 14:17:22 DEBUG : home/xxx/xxx/SourceC/InputSequence/.git/index: Unchanged skipping
...