Recommended approach for mounting an encrypted remote using rclone on Windows 10 to be used by another program for syncing, like syncthings

What is the problem you are having with rclone?

While I know this is possible and will work, I'm wondering what is the best approach for this. I want to:

  • Create a an S3 compatible remote on my Windows 10 machine
  • Wrap crypt around the remote
  • Mount the crypt as a drive letter
  • Use syncthings to do one-way sync from another drive on my machine to the crypt remote

I was thinking to use cache but apparently that is being retired.

The more I read, the more I questioned if this is even a good idea?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

BackBlaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

N/A

The rclone config contents with secrets removed.

N/A

A log from the command with the -vv flag

N/A

search the forum, as that has been discussed many times in the forum.
and i remember correctly, each time, the conclusion, it is not a good idea.

syncthing is block based and providers such as s3 as object/file based.
if syncthing has a file based mode, that might work.

tho the idea of using a crypt+mount will work.

i have used a simple file based copy program for over 10+ years, have gui.
it is called secondcopy.

can use rclone sync

I used syncthings as an example. I'm okay to use any other app like GoodSync or FreeFileSync. I was more curious how crypt + mount will work with a syncing program. I feel like there might be a lot of performance issue?

should work. tho you need to test it for yourself.

have you searched the forum for freefilesync as that has been discussed a number for times.

for example,
https://forum.rclone.org/t/copying-vfs-cache-on-windows-10/31467/14

https://forum.rclone.org/t/how-to-resume-upload/8354/14

should work. tho you need to test it for yourself.

I will but I wanted to check with the wider audience. The last thing you want are backups to not be usable for a restore. So while I will do my own test, I wanted to see if others have had any experience with this kind of configuration.

have you searched the forum for ...

Yes. But they mostly talk about a different configuration/needs. There was one post talking about ST but one is hardly enough to make a decision on. There are quite a few different backup/sync tools. I searched the ones I know but didn't find answers that quelled my curiosity. Hence why I posted to see if anyone is doing anything like what I want to do, and if they have had success with both the backups and restores, then I'll use whatever tool they use.

Most of these programs are quite complex, with a lot of built-in optimizations. It is unobvious how those optimizations might be affected when you put a crypt in the middle. Hence why I'm looking for folks with some 1st hand experience.

Copying VFS cache on Windows 10 - #14 by alfa404

Not using crypt.

How to resume upload? - #14 by AMaiden

Applies but it's just one person. I'm looking for more feedback from wider audience.

that should not affect a file copy program, as crypting is transparent.
freefilesync, has no knowledge of the crypt, as the mount presents a normal file system.

fyi, i do not use use rclone crypt for backups.

instead i use SSE-C - client side encryption provided by S3 providers, such as AWS, wasabi and B2.
it encrypts the contents of files but not the names of dirs/files.

that should not affect a file copy program, as crypting is transparent.

crypting/decrypting is done on the fly, right?

So sync program wants to compare if local file is the same as online file. Local file is not encrypted, online file is encrypted. So rclone crypt then has to encrypt the file before sync program can compare.

Right?

This may be fine for small files but if you have 1.5 TB of data, this process might get murky. Or so I assume?

tl;dr - rclone is very smart about all this.

right

not right.
when rclone first uploaded the crypted file, it saved the modtime of the local flie as metadata.

rclone will compare the modtimes, does not compare the contents.
so it takes the same amount of time to compare a 1Byte file as to compare a 1.5TiB file.

Crypt stores modification times using the underlying remote

I see.....

So then I just need to make sure my cache is big enough to keep all the metadata for all the crypted files. I assume that is what the VFS caching does?

well, if freefilesync only checks a single file, then only need metadata for that single file.
in addition, over time, rclone prunes the cache, and you can set hard limits on the max size.

I am not following. I'm backing up all of my files/folders -- 1.5 TB worth

Not sure what the "checks a single file" part is for/about?

ok, hopefully some folks will stop by soon.

1 Like