Cryptcheck reports that a certain number of hashes could not be checked

[tjanez (Tadej Janež)]

I’m a little confused with rclone’s cryptcheck output.

For example, I have created a copy of a local directory on a crypt remote with:

rclone copy -v /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321 clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321

Afterwards, I run:

rclone cryptcheck /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321 clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321

which outputted the following:

2017/08/21 17:27:28 NOTICE: Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321': 0 files not in Local file system at /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321
2017/08/21 17:27:28 NOTICE: Local file system at /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321: 0 files not in Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321'
2017/08/21 17:27:33 NOTICE: Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321': 0 differences found
2017/08/21 17:27:33 NOTICE: Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321': 21 hashes could not be checked

My question is, why does cryptcheck report that 21 hashes could not be checked?

If it helps, there are 38 encrypted files on the remote:

$ rclone ls clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321 | wc -l
38

And running check instead of cryptcheck reports that 38 hashes could not be checked:

$ rclone check /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321 clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321
2017/08/21 17:36:24 NOTICE: Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321': 0 files not in Local file system at /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321
2017/08/21 17:36:24 NOTICE: Local file system at /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321: 0 files not in Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321'
2017/08/21 17:36:24 NOTICE: Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321': 0 differences found
2017/08/21 17:36:24 NOTICE: Encrypted drive 'clonezilla-notebook-backup-tadej-crypted:Fedora24-ZBook-20170321': 38 hashes could not be checked

[calisro (Rob)]

Are there permission problems on the local file system that prevent reading of those files with the user you are using?

[tjanez (Tadej Janež)]

@calisro

Are there permission problems on the local file system that prevent reading of those files with the user you are using?

I'm afraid there are no permission problems on the local file system:

$ whoami
tadej
$ find /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321 -type f -user tadej | wc -l
38
$ find /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321 -type f -not -user tadej | wc -l
0
$ find /run/media/tadej/wd-2tb-backup/clonezilla-notebook-backup/tadej/Fedora24-ZBook-20170321 -type f -not -perm /u+r | wc -l
0

The version of rclone I'm using:

$ rclone --version
rclone v1.37

[calisro (Rob)]

Check command outputs 'XX hashes could not be checked'

I’m getting the previous output message when trying to check my remote encrypted b2 backup. The number of files XX is pretty high, so I feel a bit uneasy about the integrity of my backup given I can’t seem to check it. Is there a workaround for this problem or maybe a different way to check the validity of an encrypted remote backup? Thanks

[tjanez (Tadej Janež)]

@calisro

I'm getting the previous output message when trying to check my remote encrypted b2 backup. The number of files XX is pretty high, so I feel a bit uneasy about the integrity of my backup given I can't seem to check it. Is there a workaround for this problem or maybe a different way to check the validity of an encrypted remote backup?

Thanks

@calisro, thanks for the pointer. However, I think this is not the same issue.

If I understand crypt's documentation correctly, hashes are indeed not stored for the crypt backend meaning rclone check would not work without the aforementioned option to calculate checksums if missing.

However, crypt's documentation says to use rclone cryptcheck command to check the integrity of a crypted remote instead of rclone check which can’t check the checksums properly.

This is also consistent with my experience:

• As I wrote in my initial post, running rclone check to check the integrity of a crypted remote reports that hashes for all files (38 in my example) could not be checked.

• On the other hand, running rclone cryptcheck reports that some hashes (21 in my example) could not be checked. I'm still puzzled why it reports that?

[ncw (Nick Craig-Wood)]

What is your remote? If it is s3 then s3 doesn’t store hashes for files which were uploaded with multipart upload which may explain it?

[tjanez (Tadej Janež)]

@ncw

What is your remote? If it is s3 then s3 doesn’t store hashes for files which were uploaded with multipart upload which may explain it?

@ncw, thanks for looking into this. My underlying remote for crypt is Backblaze B2.

[ncw (Nick Craig-Wood)]

Can you run rclone sha1sum on the underlying remote? I expect you will see 21 files without SHA1 checksums.

How big are those files?

[tjanez (Tadej Janež)]

@ncw

Can you run rclone sha1sum on the underlying remote? I expect you will see 21 files without SHA1 checksums.

Yes, you are right, there are 21 files without SHA1 checksums:

$ rclone lsd --crypt-show-mapping clonezilla-notebook-backup-tadej-crypted: |& grep Fedora24-ZBook-20170321
2017/08/31 18:17:50 NOTICE: Fedora24-ZBook-20170321: Encrypts to "an0mq4bq9i0rmet1lb8emitf01869h631m5pith8fr80nq4osfng"
          -1 2017-08-31 18:17:50        -1 Fedora24-ZBook-20170321

$ rclone sha1sum b2:clonezilla-notebook-backup-tadej/an0mq4bq9i0rmet1lb8emitf01869h631m5pith8fr80nq4osfng
cf4c282b873b2386586adf58f1bb23d0caea506b  0rsja303ka00reuiasjlmqj1cs
                                          13m888ncnr0ndehl50evbq1a8i02nqdh2uruu89115v8qaf3s94g
26cec103736a9f3a6202ccaadbae6482a161bfe7  1442b8nu7u27er3i3t8s58de437uvbg97bed818e2t2ulht6otcg
                                          17fbqn4sq2life7tqt14cicjqvqboqla6iahpnm3st7jhr5ftttg
                                          29snj4omp9svpuba1up5unqe2frvsp98l7eb1i28l33n35b10eq0
3462015e2795dd59e29b44b7eeb6c058174c223a  2j6snjon0am5poh2kgig04kqtk
                                          2noe0ifb8nfp64dkm14f64cu5tq7kaib1o05ro4qdi7u3qflfn40
                                          4rajf1q7dsd7m8lu14nkvlpcou1pqpqaj3gj8eqeoodik8njm5s0
                                          97k43ni51402ovmdl74e8l44rk9qrbcj35284mn3e4tv7217mud0
                                          98l8klpfivd992cvl8cc20lfua0k4tinsrfbq33cavprm0heud90
                                          apacq9mr8bfotgs777bir0ppvbm05cntlqec2ejd7s7e7ig67ri0
a2dd99e0407131b5d0bd3eaa54940de24ecc9fa7  bf5afac9hmljfr9b307d5ecuo8
428cbabb500d75a538730cd170e1fa85b0ebb811  bm7l9tlj4o86fistu0niav6638
                                          cgs5njm3uca1ot844qu7gmdc7li4sh6ajehv942pdp20qlgebf70
1583a38bf05e6b6c8e8354374e54fb73777e9509  dfriiqu43ndhfhdgvsfc4oui4o64ft7mrsnn03hh4bhuofh80070
                                          f9f6ld2b0t85ct6oqhnkvvcgch05sd8eb6cr0h6b4p1219i55ek0
ecdd9a122f7c316ee3d9ac239f96e1ecb98f302d  gf2k3r4qi5p5nh96ku43b0edfo
6922c7f2ded74ac27e5f0c4f9316a1ba59e3c644  giq1g4ujfeufap3oqr24gbqp18
9c92fab6d3afee5385990333c166ce4ddca625dd  h4a4son85uto1pmfhrifco8ukb4h03af4vmmhf366lefb3p864fg
                                          hegh1mq1jguta77gvuhqs53a4vmbq9lcvjivijc9q3phk7ck5k50
                                          i38c71ki952foiai6eot94e1oifnfqbfdeh9muru2mvefoemas9g
b0f0ec0f7f7fe69eac72f254f0b0671bfb32e61a  i5a34dmknpjkrg0keb7cbrn020
a052f63d52b23e4dd6f6d40cfbd31da87a16d5e7  k08dopphags54dkofnv23stt9417qg48m73oceuj56catgoo5lv0
                                          keg0l33eg8ihte8mv9n0c88upnn5s3442gn4n3190t6ftkssdol0
                                          m7b7b8866f5s01nb69cbri2lhtn159q5alqceepj7l5skadun3n0
                                          mg9uobqflf0p9ptsj1ll6m9hir1nvuohj2u4sfr3gcj21eqgsgd0
                                          n47e663psn24v4l5ajsok2utgjve934uroip7cimbuvciml3dd00
                                          pcj1m4lbboo4vccfpkvvaee0hm6b7mr83ivufjo0arm19oo0gmq0
                                          pjfkesfffu4ggprbj816da7f2d9ukr905di643tg1lrc66789npg
d651c3684fb72eb4c8a2a9c95a3c747e89888511  ptjci1du4p9h0tfb6omna79lh0
36bfa42b48b459ebc96c70a12f2e3490c10ef3b8  qc4ohs058lev5cppqaguqfm17o
fe57aa03b97a07a397313df24c6c5f1ec47f37ed  rujl0obhmmqskq87l8laq78ah4
0b87b56dba806349132b821d5c22c1063fc1fad3  s1driq7hsj78a65n415p86i3oc
e429175d6d7737b608619b84d487f77dac4fcfbe  thdng8iebppg8o02bu1f4mcsj3jtr7nqpa4cthbiisdu33t5gilg
                                          tirird2b0ifgdlt3n0c0789emro7r78i8iv9p800repv6jno7oi0
                                          tuvb7a5pr9us0vc591i80datpl522cs699rgrdpdsdk2uovvpff0
                                          uc5dnup7a3n770goqm5i3acnv86qa8se65v8qtq1dh09qnlhb6a0
f188b855f717b5c572f165f8c45426bdb2d6ff75  ug739u650k96h910ntag7k28sc

$ rclone sha1sum b2:clonezilla-notebook-backup-tadej/an0mq4bq9i0rmet1lb8emitf01869h631m5pith8fr80nq4osfng | cut -f 1 -d " "| grep -cvP '\S'
21

@ncw

How big are those files?

Those files without a SHA1 checksum are all of size 4 GiBs and one is of size a little under 2 GiB:

$ rclone ls b2:clonezilla-notebook-backup-tadej/an0mq4bq9i0rmet1lb8emitf01869h631m5pith8fr80nq4osfng
       85 0rsja303ka00reuiasjlmqj1cs
4077278107 13m888ncnr0ndehl50evbq1a8i02nqdh2uruu89115v8qaf3s94g
174903015 1442b8nu7u27er3i3t8s58de437uvbg97bed818e2t2ulht6otcg
4097000032 17fbqn4sq2life7tqt14cicjqvqboqla6iahpnm3st7jhr5ftttg
4097000032 29snj4omp9svpuba1up5unqe2frvsp98l7eb1i28l33n35b10eq0
      560 2j6snjon0am5poh2kgig04kqtk
4097000032 2noe0ifb8nfp64dkm14f64cu5tq7kaib1o05ro4qdi7u3qflfn40
4097000032 4rajf1q7dsd7m8lu14nkvlpcou1pqpqaj3gj8eqeoodik8njm5s0
4097000032 97k43ni51402ovmdl74e8l44rk9qrbcj35284mn3e4tv7217mud0
4097000032 98l8klpfivd992cvl8cc20lfua0k4tinsrfbq33cavprm0heud90
4097000032 apacq9mr8bfotgs777bir0ppvbm05cntlqec2ejd7s7e7ig67ri0
      529 bf5afac9hmljfr9b307d5ecuo8
     2870 bm7l9tlj4o86fistu0niav6638
4097000032 cgs5njm3uca1ot844qu7gmdc7li4sh6ajehv942pdp20qlgebf70
      458 dfriiqu43ndhfhdgvsfc4oui4o64ft7mrsnn03hh4bhuofh80070
4097000032 f9f6ld2b0t85ct6oqhnkvvcgch05sd8eb6cr0h6b4p1219i55ek0
    12338 gf2k3r4qi5p5nh96ku43b0edfo
      358 giq1g4ujfeufap3oqr24gbqp18
  1048352 h4a4son85uto1pmfhrifco8ukb4h03af4vmmhf366lefb3p864fg
4097000032 hegh1mq1jguta77gvuhqs53a4vmbq9lcvjivijc9q3phk7ck5k50
4097000032 i38c71ki952foiai6eot94e1oifnfqbfdeh9muru2mvefoemas9g
     7427 i5a34dmknpjkrg0keb7cbrn020
      160 k08dopphags54dkofnv23stt9417qg48m73oceuj56catgoo5lv0
4097000032 keg0l33eg8ihte8mv9n0c88upnn5s3442gn4n3190t6ftkssdol0
4097000032 m7b7b8866f5s01nb69cbri2lhtn159q5alqceepj7l5skadun3n0
4097000032 mg9uobqflf0p9ptsj1ll6m9hir1nvuohj2u4sfr3gcj21eqgsgd0
4097000032 n47e663psn24v4l5ajsok2utgjve934uroip7cimbuvciml3dd00
4097000032 pcj1m4lbboo4vccfpkvvaee0hm6b7mr83ivufjo0arm19oo0gmq0
4097000032 pjfkesfffu4ggprbj816da7f2d9ukr905di643tg1lrc66789npg
    27347 ptjci1du4p9h0tfb6omna79lh0
       63 qc4ohs058lev5cppqaguqfm17o
     1101 rujl0obhmmqskq87l8laq78ah4
      716 s1driq7hsj78a65n415p86i3oc
      272 thdng8iebppg8o02bu1f4mcsj3jtr7nqpa4cthbiisdu33t5gilg
1988065794 tirird2b0ifgdlt3n0c0789emro7r78i8iv9p800repv6jno7oi0
4097000032 tuvb7a5pr9us0vc591i80datpl522cs699rgrdpdsdk2uovvpff0
4097000032 uc5dnup7a3n770goqm5i3acnv86qa8se65v8qtq1dh09qnlhb6a0
      222 ug739u650k96h910ntag7k28sc

[ncw (Nick Craig-Wood)]

I see what is gong on. Until recently rclone didn’t upload sha1 hashes for large files uploaded with b2 as it would have meant making a copy of the data.

However if you try the latest beta you should find that even large files have sha1hashes. This has been enabled by a new feature in the b2 API.

I hope this explains the problem!

[tjanez (Tadej Janež)]

@ncw

I see what is gong on. Until recently rclone didn’t upload sha1 hashes for large files uploaded with b2 as it would have meant making a copy of the data.

Oh, I see, thanks for the explanation.

@ncw

However if you try the latest beta you should find that even large files have sha1hashes. This has been enabled by a new feature in the b2 API.

That's great news. I'll try with the latest beta.

One more thing. Do I have to reupload all the large files or is there a way to avoid that?

@ncw

I hope this explains the problem!

Yes, thanks a lot!

[ncw (Nick Craig-Wood)]

@tjanez

Do I have to reupload all the large files or is there a way to avoid that?

Unfortunately I don't think the B2 API lets you set metadata without uploading the whole file (that is what rclone has to do to update the modified time).

[tjanez (Tadej Janež)]

@ncw

Unfortunately I don’t think the B2 API lets you set metadata without uploading the whole file (that is what rclone has to do to update the modified time).

Ok, I see. Thanks for the info!