[SNEBU] Backup to provider using snebu to rclone mount

Help and Support
1

What is the problem you are having with rclone?

Hi, first of all - thank you! Love your work!

I want to use rclone mount to mount my jottacloud drive locally in my debian system. The intent is then to use a 3rd party tool called snebu to create a backup of my NAS. I've simplified this using dd to remove snebu from the equation and concentrate on the rclone configuration for mount. Using dd works fine for a different set of block sizes and counts, I've transferred hundeds of GBs successfully to the mounted rclone.

When I run using snebu however I run into issues. I am able to transfer up to 50-60GB then at some point it fails, complaining that it was unable to write the file. When I check the mountpoint; this is what I see:

root@jottasync:/home/me# snebu-client backup --encryption-key /etc/snebu-backup.key -n spel /mnt/pool/Spel
Gathering full snapshot file manifest
Returning delta snapshot file manifest
Transfering files
Estimated backup size: 140.92 GB

Error creating directory /mnt/jotta/0E12%...............]  17.49G /  14.95 MB/s
Metadata recording error
root@jottasync:/home/me# mount
...
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
jotta: on /mnt/jotta type fuse.rclone (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
root@jottasync:/home/me# ls /mnt/jotta
ls: cannot access '/mnt/jotta': Transport endpoint is not connected

It seems fuse thinks everything is still mounted Ok, but the file system doesn't work of course. I can't find anything obvious in neither log nor output as to why the chain breaks down.

If I have missed to provide anything, please let me know. :slight_smile:

Thank you

What is your rclone version (output from rclone version)

rclone v1.53.3-DEV

  • os/arch: linux/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

debian, x64

Which cloud storage system are you using? (eg Google Drive)

Jottacloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --log-level "DEBUG" mount jotta: /mnt/jotta --vfs-cache-mode writes --log-file /mnt/tmp/log.txt --buffer-size=1G --vfs-read-chunk-size=64M --vfs-read-chunk-size-limit=2G --write-back-cache=1 --daemon --transfers=8 --cache-dir /mnt/tmp  --debug-fuse

The rclone config contents with secrets removed.

[jotta]
type = jottacloud
client_id = jottacli
client_secret = 
tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
token = {...}
configVersion = 1

A log from the command with the -vv flag

The actual log is 137MB so only pasting the start and end.
pastebin

2

Can you grep the log for ERROR and post those - that would be a good start.

Is it possible your disk fills up - you are using --vfs-cache-mode writes so things will be buffered to disk first before being uploaded?

I could do with seeing the last 1000 lines of the log up to the point when it stopped.

This is rather large and potentially uses 1GB of memory per open file! I suggest you leave it out - the default is 16M which is enough for most purposes.

3

Thank you @ncw!

The strange thing is, I never found an error in the log file. Now the file is unfortunately removed.

Not in this case, no. The /mnt/tmp is a folder on my NAS, with 6-7TB of free space. I did however run into the issue you mention with an exhausted cache when I used the local VM's drive as a cache. Can I in some way make these large backups inside my VM with only 10GB to spare for cache? Is it something I should do?

That might have caused the crash then, strange that it happened 40-50GB into the transfer though.

After posting, I tried a mount where I removed all the (seemingly) unnecessary options and ended up with:

rclone --log-level "DEBUG" mount jotta: /mnt/jotta --vfs-cache-mode full --log-file /mnt/tmp/log.txt --daemon --cache-dir /mnt/tmp  --debug-fuse

and I have now managed to run one successful backup of 140GB and am now at 84% on my second of 380GB.. So hopefully that was it.
(I have run with full cache mode before, without any success so that was probably not the culprit)

However, do you see if there are any options I should add, for speed, stability or any other reason?

Regards
Stefan

4

If you can make the backups with using --vfs-cache-mode off or use rclone copy or rclone rcat (to pipe things into) then rclone won't need to use extra disk space. However I think the Jottacloud backend might need to spool things to disk anyway to calculate an MD5SUM before uploading. This is stored in your systems temporary directory normally.

Great!

Turn off --debug-fuse - it has a small overhead.

Otherwise the minimal set of options looks good!

5

Super, thanks!

As I suspected, no worries. My NAS can act as a tmp-space as well. Regardless, there would not be any way for rclone to throttle snebu if I chose any other solution anyway I suppose? I.e. the data must be cached somewhere.

I am currently using the following command line to mount jottacloud immediactely on startup in /etc/rc.local:

today=$(date +"%Y-%m-%d")
logfile="/mnt/tmp/mount-${today}.txt"
rclone mount jotta: /mnt/jotta --vfs-cache-mode full --log-file ${logfile} --daemon --cache-dir /mnt/tmp

If I wanted my regular user to have access to the mount and the files, would --allow-other help?

I am experiencing a few hick ups concerning the mount which is troubling me. I wonder why they arise since the nfs mount is up. Do I need to worry about the integrity of the backup?

(base) [sri@7700k ~]$ tail -f /pool/tmp/backup/mount-2021-01-16.txt 
2021/01/16 20:00:01 ERROR : backup/71/711EB210C699D93A7E9A216731A9BBAE080F4A86F66F0D92A66546997E405F.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/71/711EB210C699D93A7E9A216731A9BBAE080F4A86F66F0D92A66546997E405F.enc: stale NFS file handle
2021/01/16 20:00:01 ERROR : backup/71/711EB210C699D93A7E9A216731A9BBAE080F4A86F66F0D92A66546997E405F.enc: Non-out-of-space error encountered during open
2021/01/16 20:00:01 ERROR : backup/71/711EB210C699D93A7E9A216731A9BBAE080F4A86F66F0D92A66546997E405F.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/71/711EB210C699D93A7E9A216731A9BBAE080F4A86F66F0D92A66546997E405F.enc: stale NFS file handle
2021/01/16 20:04:00 ERROR : backup/77/E83807E71C70A1B8394101A2E81CA807C26939B82661A376D629E08CC22866.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/77/E83807E71C70A1B8394101A2E81CA807C26939B82661A376D629E08CC22866.enc: stale NFS file handle
2021/01/16 20:04:00 ERROR : backup/77/E83807E71C70A1B8394101A2E81CA807C26939B82661A376D629E08CC22866.enc: Non-out-of-space error encountered during open
2021/01/16 20:04:00 ERROR : backup/77/E83807E71C70A1B8394101A2E81CA807C26939B82661A376D629E08CC22866.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/77/E83807E71C70A1B8394101A2E81CA807C26939B82661A376D629E08CC22866.enc: stale NFS file handle
2021/01/16 20:04:20 ERROR : backup/78/794716F1BCAC70EA6ACB0ECC96CDF4D5AA89BE44F812DE344B04DC97D6336E.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/78/794716F1BCAC70EA6ACB0ECC96CDF4D5AA89BE44F812DE344B04DC97D6336E.enc: stale NFS file handle
2021/01/16 20:04:20 ERROR : backup/78/794716F1BCAC70EA6ACB0ECC96CDF4D5AA89BE44F812DE344B04DC97D6336E.enc: Non-out-of-space error encountered during open
2021/01/16 20:04:20 ERROR : backup/78/794716F1BCAC70EA6ACB0ECC96CDF4D5AA89BE44F812DE344B04DC97D6336E.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/78/794716F1BCAC70EA6ACB0ECC96CDF4D5AA89BE44F812DE344B04DC97D6336E.enc: stale NFS file handle
2021/01/16 20:24:27 ERROR : backup/9A/C526E9DC0D499677D17A3A275307CA9BA5A80C0A327E521B60FDB1D48FD9C5.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/9A/C526E9DC0D499677D17A3A275307CA9BA5A80C0A327E521B60FDB1D48FD9C5.enc: stale NFS file handle
2021/01/16 20:24:27 ERROR : backup/9A/C526E9DC0D499677D17A3A275307CA9BA5A80C0A327E521B60FDB1D48FD9C5.enc: Non-out-of-space error encountered during open
2021/01/16 20:24:27 ERROR : backup/9A/C526E9DC0D499677D17A3A275307CA9BA5A80C0A327E521B60FDB1D48FD9C5.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/9A/C526E9DC0D499677D17A3A275307CA9BA5A80C0A327E521B60FDB1D48FD9C5.enc: stale NFS file handle
2021/01/16 20:44:22 ERROR : backup/BC/D7EE4EB3F15006A98EF8A87DA9B0C368083805C3B3B70D56ACD2DE6B10892E.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/BC/D7EE4EB3F15006A98EF8A87DA9B0C368083805C3B3B70D56ACD2DE6B10892E.enc: stale NFS file handle
2021/01/16 20:44:22 ERROR : backup/BC/D7EE4EB3F15006A98EF8A87DA9B0C368083805C3B3B70D56ACD2DE6B10892E.enc: Non-out-of-space error encountered during open
2021/01/16 20:44:22 ERROR : backup/BC/D7EE4EB3F15006A98EF8A87DA9B0C368083805C3B3B70D56ACD2DE6B10892E.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/BC/D7EE4EB3F15006A98EF8A87DA9B0C368083805C3B3B70D56ACD2DE6B10892E.enc: stale NFS file handle
2021/01/16 20:44:45 ERROR : backup/BD/808ABC2EE6EE7A1B42437CC89ABB528D22355B03C182545B43090AEC1E75B5.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/BD/808ABC2EE6EE7A1B42437CC89ABB528D22355B03C182545B43090AEC1E75B5.enc: stale NFS file handle
2021/01/16 20:44:45 ERROR : backup/BD/808ABC2EE6EE7A1B42437CC89ABB528D22355B03C182545B43090AEC1E75B5.enc: Non-out-of-space error encountered during open
2021/01/16 20:44:45 ERROR : backup/BD/808ABC2EE6EE7A1B42437CC89ABB528D22355B03C182545B43090AEC1E75B5.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/BD/808ABC2EE6EE7A1B42437CC89ABB528D22355B03C182545B43090AEC1E75B5.enc: stale NFS file handle
2021/01/16 20:44:51 ERROR : backup/BD/AD92228A0E0290419519161C757E145FC95F48483FFB718513D11491545F87.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/BD/AD92228A0E0290419519161C757E145FC95F48483FFB718513D11491545F87.enc: stale NFS file handle
2021/01/16 20:44:51 ERROR : backup/BD/AD92228A0E0290419519161C757E145FC95F48483FFB718513D11491545F87.enc: Non-out-of-space error encountered during open
2021/01/16 20:44:51 ERROR : backup/BD/AD92228A0E0290419519161C757E145FC95F48483FFB718513D11491545F87.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/BD/AD92228A0E0290419519161C757E145FC95F48483FFB718513D11491545F87.enc: stale NFS file handle
2021/01/16 20:51:36 ERROR : backup/C9/9D0F929E89963BFB84DE0653CBBA1B9519FBE377973BDAA5B718EE61326CA9.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/C9/9D0F929E89963BFB84DE0653CBBA1B9519FBE377973BDAA5B718EE61326CA9.enc: stale NFS file handle
2021/01/16 20:51:36 ERROR : backup/C9/9D0F929E89963BFB84DE0653CBBA1B9519FBE377973BDAA5B718EE61326CA9.enc: Non-out-of-space error encountered during open
2021/01/16 20:51:36 ERROR : backup/C9/9D0F929E89963BFB84DE0653CBBA1B9519FBE377973BDAA5B718EE61326CA9.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/C9/9D0F929E89963BFB84DE0653CBBA1B9519FBE377973BDAA5B718EE61326CA9.enc: stale NFS file handle
2021/01/16 20:52:19 ERROR : backup/CA/EE6BE8E6744ACC76A4A20268A8D0BA1BEFF7272A3A4A7DE538BA5D362F5CE3.enc: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/CA/EE6BE8E6744ACC76A4A20268A8D0BA1BEFF7272A3A4A7DE538BA5D362F5CE3.enc: stale NFS file handle
2021/01/16 20:52:19 ERROR : backup/CA/EE6BE8E6744ACC76A4A20268A8D0BA1BEFF7272A3A4A7DE538BA5D362F5CE3.enc: Non-out-of-space error encountered during open
2021/01/16 20:52:19 ERROR : backup/CA/EE6BE8E6744ACC76A4A20268A8D0BA1BEFF7272A3A4A7DE538BA5D362F5CE3.enc: vfs cache: failed to reload item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: truncate /mnt/tmp/vfs/jotta/backup/CA/EE6BE8E6744ACC76A4A20268A8D0BA1BEFF7272A3A4A7DE538BA5D362F5CE3.enc: stale NFS file handle
2021/01/17 18:49:26 ERROR : backup/4D/F5E7154B465AC8F2DD19561E4034432D37FC79DFA118DBF8AF785D53399AB0.enc: vfs cache: failed to write metadata file: vfs cache item: failed to encode metadata: write /mnt/tmp/vfsMeta/jotta/backup/4D/F5E7154B465AC8F2DD19561E4034432D37FC79DFA118DBF8AF785D53399AB0.enc: stale NFS file handle
2021/01/17 18:50:21 ERROR : backup/F8/EE5BFDB48503030EC54346FEADFDCFBE1F6B4116CC796896BABBF0F84C0FD7.enc: vfs cache: failed to write metadata file: vfs cache item: failed to encode metadata: write /mnt/tmp/vfsMeta/jotta/backup/F8/EE5BFDB48503030EC54346FEADFDCFBE1F6B4116CC796896BABBF0F84C0FD7.enc: stale NFS file handle
2021/01/17 19:29:53 ERROR : backup/EC/AFAB4D6FFC8BC8A70DEC7D24A07C4251F9709E02ED372B0425345A57290A8F.enc: Failed to copy: failed to calculate MD5: write /tmp/rclone-jcmd5-617123928: no space left on device
2021/01/17 19:29:53 ERROR : backup/DE/015EC8105662E20146D0A6D414A8B9115B636E302FD4AE9AC62AFF4AED7704.enc: Failed to copy: failed to calculate MD5: write /tmp/rclone-jcmd5-998794483: no space left on device
2021/01/17 19:29:53 ERROR : backup/07/6483185254DE1F43C912BF9DE5CDE6D0BC4EB8571CA292EAE5B9EA9920CE3B.enc: Failed to copy: failed to calculate MD5: write /tmp/rclone-jcmd5-596146077: no space left on device
2021/01/17 19:29:53 ERROR : backup/C3/7110C1D5749F1885DFC4D2B79435C6856731F17647D951AEA98098023D3B9A.enc: Failed to copy: failed to calculate MD5: write /tmp/rclone-jcmd5-358981558: no space left on device
2021/01/17 19:29:53 ERROR : backup/EC/AFAB4D6FFC8BC8A70DEC7D24A07C4251F9709E02ED372B0425345A57290A8F.enc: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: failed to calculate MD5: write /tmp/rclone-jcmd5-617123928: no space left on device
2021/01/17 19:29:53 ERROR : backup/DE/015EC8105662E20146D0A6D414A8B9115B636E302FD4AE9AC62AFF4AED7704.enc: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: failed to calculate MD5: write /tmp/rclone-jcmd5-998794483: no space left on device
2021/01/17 19:29:53 ERROR : backup/07/6483185254DE1F43C912BF9DE5CDE6D0BC4EB8571CA292EAE5B9EA9920CE3B.enc: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: failed to calculate MD5: write /tmp/rclone-jcmd5-596146077: no space left on device
2021/01/17 19:29:53 ERROR : backup/C3/7110C1D5749F1885DFC4D2B79435C6856731F17647D951AEA98098023D3B9A.enc: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: failed to calculate MD5: write /tmp/rclone-jcmd5-358981558: no space left on device
2021/01/17 22:44:34 ERROR : vfs cache: item close failed: vfs cache item: failed to encode metadata: write /mnt/tmp/vfsMeta/jotta/backup/tbjCcHXG: stale NFS file handle
2021/01/17 22:44:34 ERROR : backup/tbjCcHXG(0xc0060a48c0): RWFileHandle.Release error: vfs cache item: failed to encode metadata: write /mnt/tmp/vfsMeta/jotta/backup/tbjCcHXG: stale NFS file handle
6

They have continued a bit while uploading a huge chunk of data at 4.6TB:

2021/01/18 16:43:43 ERROR : backup/tb0bQNAF: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: open /mnt/tmp/vfs/jotta/backup/tb0bQNAF: no space left on device
2021/01/18 16:43:48 NOTICE: vfs cache: in KickCleaner, ready to kick cleaner
2021/01/18 21:45:08 ERROR : backup/2B/FA32C4CF0DF713EA97E8C546C275CAE57F384B8339F58A06C9C56B9F821F41.enc: item removed when it was writing/uploaded
2021/01/19 07:27:57 ERROR : backup/61/17E7991E4FD65B852FF8E3608EFCD54FC257B82A54ECB51D1A98F931EA7996.enc: item removed when it was writing/uploaded

Can I in some way test the integrity of the data, or in some other way find out whether there are perrmanent errors in the data?

7

Yes that should do it. Note that you shouldn't mount FUSE file systems as root. You can, but it isn't recommended or needed.

stale NFS file handle

Looks like the root cause of those errors.

This said no space left on device

Has snebu got a check option? That would be the best way.

8

Thank you! I have corrected that now. My current command line is

rclone --log-level "DEBUG" mount jotta: /mnt/jotta --vfs-cache-mode full --log-file /mnt/tmp/rclone-20200121.txt --daemon --cache-dir /mnt/tmp --allow-other

(Having enabled debug logs while validating the set up)

I agree. I also think that this might be the reason for the log message:

2021/01/18 16:43:43 ERROR : backup/tb0bQNAF: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: open /mnt/tmp/vfs/jotta/backup/tb0bQNAF: no space left on device

since as I said, the NAS has somewhere in the range of 6-8TB of space free. I think there were some logs which indicated that when there was an NFS hick-up, rclone seemed to switch to local /tmp instead and exhausting the available space there.

2021/01/17 18:49:26 ERROR : backup/4D/F5E7154B465AC8F2DD19561E4034432D37FC79DFA118DBF8AF785D53399AB0.enc: vfs cache: failed to write metadata file: vfs cache item: failed to encode metadata: write /mnt/tmp/vfsMeta/jotta/backup/4D/F5E7154B465AC8F2DD19561E4034432D37FC79DFA118DBF8AF785D53399AB0.enc: stale NFS file handle
2021/01/17 18:50:21 ERROR : backup/F8/EE5BFDB48503030EC54346FEADFDCFBE1F6B4116CC796896BABBF0F84C0FD7.enc: vfs cache: failed to write metadata file: vfs cache item: failed to encode metadata: write /mnt/tmp/vfsMeta/jotta/backup/F8/EE5BFDB48503030EC54346FEADFDCFBE1F6B4116CC796896BABBF0F84C0FD7.enc: stale NFS file handle
2021/01/17 19:29:53 ERROR : backup/EC/AFAB4D6FFC8BC8A70DEC7D24A07C4251F9709E02ED372B0425345A57290A8F.enc: Failed to copy: failed to calculate MD5: write /tmp/rclone-jcmd5-617123928: no space left on device
2021/01/17 19:29:53 ERROR : backup/DE/015EC8105662E20146D0A6D414A8B9115B636E302FD4AE9AC62AFF4AED7704.enc: Failed to copy: failed to calculate MD5: write /tmp/rclone-jcmd5-998794483: no space left on device
9

Ah, I the jottacloud backend uses /tmp to spool files to if it doesn't know the MD5SUM in advance. It shouldn't need to do that for a vfs cache mode full mount though.

You can set your own directory by setting the environment variable TMPDIR

10

I think it got a bit confused when the vfs cache suddenly disappeared. :slight_smile:

Perfect, thanks!

1 Like
11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.