How to backup a crypt remote efficiently

kamen · February 2, 2022, 8:48pm

First of all cheers for the great tool you have implemented and shared. It is a great one!

What is the problem you are having with rclone?

I am trying to setup an efficient backup of a crypted remote. My setup works but it has certain drawbacks.

I read this: https://rclone.org/crypt/#backing-up-a-crypted-remote but I do not understand it. It seems to me that the sync command (as I use it) downloads all files from the remote instead of using the vfs full cache that I carefully set-up by mounting the crypt remote. These downloads are costly - in terms of time and money.

So my questions are:

How to make use of the vfs cache that I maintain in order to sync the crypt remote with another one?
What does a restore operation look like?

P.S. I will probably setup SFTP machine for the backup destination but I'm currently testing with local. Right now I do not have a preference if the backup remote should contain encrypted or non-encrypted files.

Run the command 'rclone version' and share the full output of the command.

rclone 1.57.0

os/version: nixos 21.11 (Porcupine) (64 bit)
os/kernel: 5.10.87 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.16.9
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

B2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone -v sync -i bb:bucket1/fa locl:/backup/fa

The rclone config contents with secrets removed.

[bb]
type = b2
account = ...
key = ...
hard_delete = true

[fa]
type = crypt
remote = bb:bucket1/fa
filename_encryption = obfuscate
password = ...
password2 = ...

[locl]
type = local

A log from the command with the `-vv` flag

rclone -vv sync -i bb:bucket1/fa locl:/backup/fa
2022/02/02 22:29:55 DEBUG : rclone: Version "1.57.0" starting with parameters ["/run/current-system/sw/bin/rclone" "-vv" "sync" "-i" "bb:bucket1/fa" "locl:/backup/fa"]
2022/02/02 22:29:55 DEBUG : Creating backend with remote "bb:bucket1/fa"
2022/02/02 22:29:55 DEBUG : Using config file from "/home/kamen/.config/rclone/rclone.conf"
2022/02/02 22:29:56 DEBUG : Couldn't decode error response: EOF
2022/02/02 22:29:56 DEBUG : Creating backend with remote "locl:/backup/fa"
2022/02/02 22:29:56 DEBUG : 23.DpEFA.wtDA: Size and modification time the same (differ by 0s, within tolerance 1ms)
2022/02/02 22:29:56 DEBUG : 23.DpEFA.wtDA: Unchanged skipping
2022/02/02 22:29:56 DEBUG : Local file system at /backup/fa: Waiting for checks to finish
2022/02/02 22:29:56 DEBUG : Local file system at /backup/fa: Waiting for transfers to finish
2022/02/02 22:29:56 DEBUG : Waiting for deletions to finish
2022/02/02 22:29:56 INFO  : There was nothing to transfer
2022/02/02 22:29:56 NOTICE: 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         1.5s

2022/02/02 22:29:56 DEBUG : 8 go routines active

asdffdsa · February 2, 2022, 9:07pm

hello and welcome to the forum,

rclone sync and rclone mount have nothing to do with each other.
i understand your idea but not going to be a reliable backup solution.
the vfs file cache is temporary, files get added, files get deleted,
but if most of the remote's files are already in the local vfs file cache.
then might rclone mount as the source.
imho, need to be careful when creating backups.

also, where did you get that rclone executable from, does not look like the official client?
can install it from https://rclone.org/downloads/#script-download-and-install

and a side question, what is the total size of all files in the crypt?

kamen · February 3, 2022, 8:26am

Thank you very much for your response and welcome message.
First of all the backup recommendation on the rclone crypt page is not clear to me:

What is remote:crypt and remote2:crypt in my case?
If it were recommended to use the crypted remote as a source (and not the original) the vfs cache would have been used implicitly from the source remote. This way we will keep the 3 advantages stated there and add one additional one or am I wrong?

On your questions:

I'm taking it from my OS and it is an official client
It will be hundreds of MB

asdffdsa · February 3, 2022, 2:23pm

i think your command is correct, tho no need for locl:
rclone -vv sync -i bb:bucket1/fa /backup/fa

the source is the files in their crypted state.
the dest is whatever you want to be, in your case, local.
and now that the crypt files are local, you can create a remote to acess them

[fa-local]
type = crypt
remote = /backup/fa
filename_encryption = obfuscate
password = ...
password2 = ...

rclone ls fa-local:

kamen · February 3, 2022, 3:58pm

Now that I read the documentation for thousand time I understand that the recommendation is to copy from one crypt remote to another one while both have the same passwords. I haven't looked at the code but I assume that this way rclone is able to sync without doing any encrypting or decrypting operations. What is not clear to me is:

Can rclone use the vfs cache from the source to do this or it has to download all the remote files?
What if the destination is not of type crypt - does it have to download all the remote files or it can use the vfs cache?

asdffdsa · February 3, 2022, 4:07pm

very small amount of data, why try to use the vfs file cache of a rclone mount?
why not just use rclone sync?

correct, and an added benefit is using md5 hash to verify file transfers.

imho, not realistic, the vfs file cache
--- is not mirror of the remote.
--- does not always contain full files. uses chunked reading and sparse files.

post your current rclone mount command?

kamen · February 3, 2022, 4:22pm

hundreds of GB - my mistake

rclone mount \
	--vfs-cache-mode full \
	--vfs-cache-poll-interval 0 \
	--use-server-modtime \
	fa: /home/kamen/stuff

There will be benefits of using the vfs cache for the sync even if not all files are in the cache and full. The question when is the cache used is still open.

asdffdsa · February 3, 2022, 4:47pm

if you want to use the mount as a souce, then
cp /home/kamen/stuff /path/to/backup/dir

not sure what you are asking?

kamen · February 3, 2022, 5:14pm

The question is if the backup destination is a rclone remote and I'm using a sync operation will the vfs cache be used? Will the vfs cache be used if the destination is a crypt remote? Will the vfs cache be used if the destination is a non-crypt remote? This is much more important to me than if decryption/encryption will happen during sync.

asdffdsa · February 3, 2022, 5:28pm

will not use the the vfs file cache.
rclone sync bb:bucket1/fa remote:

based on your rclone mount command, will use the vfs file cache.
rclone sync /home/kamen/stuff remote:
note: if a file is not in the vfs file cache,
then rclone will have to download it into the cache, decrypt it and copy it to remote:

asdffdsa · February 3, 2022, 11:07pm

keep in mind that, files in vfs file cache, are pruned out based on

--vfs-cache-max-age duration         Max age of objects in the cache (default 1h0m0s)

note: if a file is in-use/open, rclone will prune not it.

kamen · February 4, 2022, 4:06pm

Thanks for mentioning this.
The thing is that today's experience of mine shows that the vfs cache is not pruned neither after the default period nor after 8 hours. I suspect that this is because I use the parameter:

		--vfs-cache-poll-interval 0 \

However, this is not mentioned in the documentation so something is wrong somewhere.

Here is what I've done:

I mounted the directory
Copied files inside
Observed the contents of .cache/rcone/

I haven't touched the files for more than 8 hours yet all is there.

asdffdsa · February 4, 2022, 4:11pm

that is documented.
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)

Animosity022 · February 4, 2022, 4:11pm

A log file just makes life easier.

If you set it to 0, disable polling, then it doesn't poll as shown in the logfile.

2022/02/04 11:09:11 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/02/04 11:09:11 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "DB:" "/home/felix/test" "--vfs-cache-mode" "full" "--vfs-cache-poll-interval" "0" "-vvv"]
2022/02/04 11:09:11 DEBUG : Creating backend with remote "DB:"
2022/02/04 11:09:11 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/02/04 11:09:11 DEBUG : vfs cache: root is "/home/felix/.cache/rclone"
2022/02/04 11:09:11 DEBUG : vfs cache: data root is "/home/felix/.cache/rclone/vfs/DB"
2022/02/04 11:09:11 DEBUG : vfs cache: metadata root is "/home/felix/.cache/rclone/vfsMeta/DB"
2022/02/04 11:09:11 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfs/DB/"
2022/02/04 11:09:11 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfsMeta/DB/"
2022/02/04 11:09:11 DEBUG : Dropbox root '': Mounting on "/home/felix/test"
2022/02/04 11:09:11 DEBUG : vfs cache: cleaning thread disabled because poll interval <= 0

So if you set polling to 0 (turn if off) it won't poll and not remove anything so it's operating exactly as you've configured it.

system · March 6, 2022, 4:11pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.