Parameter advice for backup program access

Help and Support
1

I have a laptop connected via ethernet to my router and then to FIOS. It should be a really decent connection.

What I am wanting to do is mount my pCloud account for two things. First I would like to use it just like any other fuse mounted file system (edit, copy, save files etc etc, standard stuff). The second thing, and where I am having issues, I would like to do is to be able to use BorgBackup or restic to backup pCloud via the mount. These might use two separate sets of rclone mount parameters.

I am currently mounting pCloud using the systemd automount. My fstab is:

rclonefs#pCloudCache: /cloud/pcloud fuse config=/home/username/.config/rclone/rclone.conf,allow-other,default-permissions,max-read-ahead=16M,noauto,x-systemd.automount,x-systemd.idle-timeout=300s,vfs-cache-mode=full,dir-cache-time=10s,cache-info-age=5s,_netdev 0 0

And my clone.conf for that section is:

[pCloudCache]
type = cache
remote = pCloud:
chunk_size = 3M
info_age = 1m0s
chunk_total_size = 500M

When I use this as a standard kind of fuse mounted fs things seem kind of decent. But when I try to do a backup of the mount it gets really, really slow. (I understand the first backup has to download all the data :slight_smile: ).

I want to backup data on the order of 60G and it is just standard kind of files (not large files like movies).

To do some tests, I tried doing just an rsync on a sub-directory to a local filesystem:

rsync -avz --stats /cloud/pcloud/Documents/computing delete/

I did this to rule out de-duplication, compression, encryption time that the backup software might do. But even the rsync was slow. It actually ran overnight and it is only about 300M of data in that sub-directory.

Rclone has a lot of parameters and I love parameters, but I would appreciate any advice on what I can change to see about getting better read-only performance. I guess there are even two parts to that, first is the initial download and for subsequent backups I suspect it just needs to check the stat on the file to see if it changed (let’s ignore inodes for now).

rclone v1.42

  • os/arch: linux/amd64
  • go version: go1.10.3
2

So, I played with the parameters a bit. I made a read only mount using:

rclonefs#Dropbox: /cloud/dropboxRO fuse config=/home/username/.config/rclone/rclone.conf,uid=1000,gid=1000,allow-other,default-permissions,noauto,x-systemd.automount,x-systemd.idle-timeout=300s,read-only,vfs-cache-mode=off,_netdev 0 0

That seems to work better. So, basically, I think, I turned off the caching completely.

I would still be curious to hear if there are other parameters that I should be using for this type of thing…

3

Yes, I think you’d be better off with --vfs-cache-mode writes rather than full unless you really want a local copy of everything.

I wouldn’t do my backup like this though, I would use rclone sync direct to or from pcloud. rclone sync is a lot more efficient than rsync to rclone mount as it can cut out lots of layers but most importantly it has a much better retry strategy.

You can then clear the mount directory cache to see any changes - see clearing the directory cache in the docs.

4

Got it. Really, though, I was just using rsync as a test. That does help knowing more about the vfs-cache-mode

What my plan is to use either restic or borg and I don’t really want to copy down all the data. The source directory for restic or borg would be the rclone mounted directory and the output directory would either be BackBlaze (for restic) or a local directory (for borg).

For example:

$ (cd /cloud/pcloudRO && restic -v -r b2:bucket:/pcloud backup Documents)

or

$ (cd /cloud/pcloudRO && borg create --ignore-inode --stats --compression lz4 --progress -e repokey /localdir::documents-{now} Documents )

5

That should work.

However you could use rclone to copy/sync direct from pcloud to backblaze if you wanted - rclone supports cloud to cloud copies. You can also build snapshots using --backup-dir. This isn’t as sophisticated as restic or borg, but may do for your purposes! I personally really like my backups 1:1 mapped file to file which is what rclone does, but that means that rclone doesn’t compress or de-dupe like restic which may be important to you. (Not that familiar with borg).

1 Like