Can't list or copy some individual files using --b2-version-at with crypt

Suspected Bug
1

What is the problem you are having with rclone?

Can't list or copy some individual files using --b2-version-at with crypt

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2
- os/version: darwin 14.2.1 (64 bit)
- os/kernel: 23.2.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.21.6
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -q --b2-version-at=10d lsjson --max-depth=1 b2-crypt:/Dustin/Personal/Brushes/dW.abr

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[b2]
type = b2
account = XXX
key = XXX

[b2-crypt]
type = crypt
remote = b2:bucket-name
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

2024/02/01 17:02:06 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "-vv" "--b2-version-at=10d" "lsjson" "b2-crypt:Dustin/Personal/Brushes/dW.abr"]
2024/02/01 17:02:06 DEBUG : Creating backend with remote "b2-crypt:Dustin/Personal/Brushes/dW.abr"
2024/02/01 17:02:06 DEBUG : Using config file from "/Users/dw/.config/rclone/rclone.conf"
2024/02/01 17:02:06 DEBUG : Creating backend with remote "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs"
2024/02/01 17:02:06 DEBUG : b2: detected overridden config - adding "{SEVCy}" suffix to name
2024/02/01 17:02:07 DEBUG : fs cache: renaming cache item "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs" to be canonical "b2{SEVCy}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs"
2024/02/01 17:02:07 DEBUG : fs cache: switching user supplied name "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs" for canonical name "b2{SEVCy}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs"
[
]

I could just be doing something wrong, but the backstory to this is that I'm just testing a bucket that I deleted a bunch of files and folders from to see if I can list and/or copy the deleted files to my local drive. Most of the time it works just fine, but I've encountered some instances where the files don't list. I seem to have narrowed it down to files whose parent directory has also been deleted.

I can prove there are files in the folder when going back 10 days:

$ rclone -vv --b2-version-at=10d lsjson b2-crypt:"Dustin/Personal/Brushes"
2024/02/01 17:10:08 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "-vv" "--b2-version-at=10d" "lsjson" "b2-crypt:Dustin/Personal/Brushes"]
2024/02/01 17:10:08 DEBUG : Creating backend with remote "b2-crypt:Dustin/Personal/Brushes"
2024/02/01 17:10:08 DEBUG : Using config file from "/Users/dw/.config/rclone/rclone.conf"
2024/02/01 17:10:08 DEBUG : Creating backend with remote "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18"
2024/02/01 17:10:08 DEBUG : b2: detected overridden config - adding "{AjCHr}" suffix to name
2024/02/01 17:10:09 DEBUG : fs cache: renaming cache item "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18" to be canonical "b2{AjCHr}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18"
2024/02/01 17:10:09 DEBUG : fs cache: switching user supplied name "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18" for canonical name "b2{AjCHr}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18"
[
{"Path":"dW.abr","Name":"dW.abr","Size":3093864,"MimeType":"application/octet-stream","ModTime":"2023-08-05T23:16:12.456Z","IsDir":false,"ID":"4_zaaa12738c901ecfc889b0f10_f105f6e62c89b7714_d20230813_m182328_c000_v0001410_t0050_u01691951007978"},
{"Path":"flat bristle.png","Name":"flat bristle.png","Size":9376,"MimeType":"image/png","ModTime":"2021-12-04T16:03:47.504Z","IsDir":false,"ID":"4_zaaa12738c901ecfc889b0f10_f1000d3912ab9ec76_d20230813_m182328_c000_v0001071_t0010_u01691951008141"},
{"Path":"gesso.png","Name":"gesso.png","Size":312550,"MimeType":"image/png","ModTime":"2023-08-05T23:16:22.924Z","IsDir":false,"ID":"4_zaaa12738c901ecfc889b0f10_f118ede77fae14433_d20230813_m182328_c000_v0001400_t0021_u01691951008153"},
{"Path":"splat.jpg","Name":"splat.jpg","Size":4095950,"MimeType":"image/jpeg","ModTime":"2023-08-05T23:16:18.043Z","IsDir":false,"ID":"4_zaaa12738c901ecfc889b0f10_f109a77d850204ad9_d20230813_m182328_c000_v0001060_t0005_u01691951008103"},
{"Path":"watercolor paper.png","Name":"watercolor paper.png","Size":577793,"MimeType":"image/png","ModTime":"2023-08-05T23:16:16.830Z","IsDir":false,"ID":"4_zaaa12738c901ecfc889b0f10_f11026f2d78b3f838_d20230813_m182328_c000_v0001410_t0026_u01691951008111"},
{"Path":"willow charcoal.jpg","Name":"willow charcoal.jpg","Size":3202895,"MimeType":"image/jpeg","ModTime":"2023-08-05T23:16:15.117Z","IsDir":false,"ID":"4_zaaa12738c901ecfc889b0f10_f1099902b854295a5_d20230813_m182328_c000_v0001410_t0010_u01691951008417"}

When I try to copy the folder over to my local computer it works:

$ rclone -vv --b2-version-at=10d copy b2-crypt:Dustin/Personal/Brushes Brushes
2024/02/01 17:14:47 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "-vv" "--b2-version-at=10d" "copy" "b2-crypt:Dustin/Personal/Brushes" "Brushes"]
2024/02/01 17:14:47 DEBUG : Creating backend with remote "b2-crypt:Dustin/Personal/Brushes"
2024/02/01 17:14:47 DEBUG : Using config file from "/Users/dw/.config/rclone/rclone.conf"
2024/02/01 17:14:47 DEBUG : Creating backend with remote "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18"
2024/02/01 17:14:47 DEBUG : b2: detected overridden config - adding "{AFibL}" suffix to name
2024/02/01 17:14:48 DEBUG : fs cache: renaming cache item "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18" to be canonical "b2{AFibL}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18"
2024/02/01 17:14:48 DEBUG : fs cache: switching user supplied name "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18" for canonical name "b2{AFibL}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18"
2024/02/01 17:14:48 DEBUG : Creating backend with remote "Brushes"
2024/02/01 17:14:48 DEBUG : fs cache: renaming cache item "Brushes" to be canonical "/Users/dw/Desktop/Brushes"
2024/02/01 17:14:48 DEBUG : dW.abr: Need to transfer - File not found at Destination
2024/02/01 17:14:48 DEBUG : flat bristle.png: Need to transfer - File not found at Destination
2024/02/01 17:14:48 DEBUG : gesso.png: Need to transfer - File not found at Destination
2024/02/01 17:14:48 DEBUG : splat.jpg: Need to transfer - File not found at Destination
2024/02/01 17:14:48 DEBUG : watercolor paper.png: Need to transfer - File not found at Destination
2024/02/01 17:14:48 DEBUG : willow charcoal.jpg: Need to transfer - File not found at Destination
2024/02/01 17:14:48 DEBUG : Local file system at /Users/dw/Desktop/Brushes: Waiting for checks to finish
2024/02/01 17:14:48 DEBUG : Local file system at /Users/dw/Desktop/Brushes: Waiting for transfers to finish
2024/02/01 17:14:49 DEBUG : flat bristle.png.janeyuf3.partial: renamed to: flat bristle.png
2024/02/01 17:14:49 INFO  : flat bristle.png: Copied (new)
2024/02/01 17:14:49 DEBUG : dW.abr.lutevoc2.partial: renamed to: dW.abr
2024/02/01 17:14:49 INFO  : dW.abr: Copied (new)
2024/02/01 17:14:49 DEBUG : gesso.png.mivuboy1.partial: renamed to: gesso.png
2024/02/01 17:14:49 INFO  : gesso.png: Copied (new)
2024/02/01 17:14:50 DEBUG : splat.jpg.nidayov6.partial: renamed to: splat.jpg
2024/02/01 17:14:50 INFO  : splat.jpg: Copied (new)
2024/02/01 17:14:50 DEBUG : watercolor paper.png.cisayib2.partial: renamed to: watercolor paper.png
2024/02/01 17:14:50 INFO  : watercolor paper.png: Copied (new)
2024/02/01 17:14:50 DEBUG : willow charcoal.jpg.dinazun0.partial: renamed to: willow charcoal.jpg
2024/02/01 17:14:50 INFO  : willow charcoal.jpg: Copied (new)
2024/02/01 17:14:50 INFO  : 
Transferred:   	   10.769 MiB / 10.769 MiB, 100%, 1.005 MiB/s, ETA 0s
Transferred:            6 / 6, 100%
Elapsed time:         2.8s

2024/02/01 17:14:50 DEBUG : 19 go routines active

But it doesn't when I try to copy any one of the files (I emptied the local folder before attempting it):

$ rclone -vv --b2-version-at=10d copy b2-crypt:Dustin/Personal/Brushes/dW.abr Brushes
2024/02/01 17:19:14 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "-vv" "--b2-version-at=10d" "copy" "b2-crypt:Dustin/Personal/Brushes/dW.abr" "Brushes"]
2024/02/01 17:19:14 DEBUG : Creating backend with remote "b2-crypt:Dustin/Personal/Brushes/dW.abr"
2024/02/01 17:19:14 DEBUG : Using config file from "/Users/dw/.config/rclone/rclone.conf"
2024/02/01 17:19:14 DEBUG : Creating backend with remote "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs"
2024/02/01 17:19:14 DEBUG : b2: detected overridden config - adding "{IskLP}" suffix to name
2024/02/01 17:19:14 DEBUG : fs cache: renaming cache item "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs" to be canonical "b2{IskLP}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs"
2024/02/01 17:19:14 DEBUG : fs cache: switching user supplied name "b2:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs" for canonical name "b2{IskLP}:bucket-name/vp0t0k8bk8rfoah48mkm769tck/2nanaovkckkh918mkhhf2o8m48/69h9b2q2im5os2t68virhl3b18/9ar3hsie52kk400g6rdak00eqs"
2024/02/01 17:19:14 DEBUG : Creating backend with remote "Brushes"
2024/02/01 17:19:14 DEBUG : fs cache: renaming cache item "Brushes" to be canonical "/Users/dw/Desktop/Brushes"
2024/02/01 17:19:15 DEBUG : Local file system at /Users/dw/Desktop/Brushes: Waiting for checks to finish
2024/02/01 17:19:15 DEBUG : Local file system at /Users/dw/Desktop/Brushes: Waiting for transfers to finish
2024/02/01 17:19:15 INFO  : There was nothing to transfer
2024/02/01 17:19:15 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.0s

2024/02/01 17:19:15 DEBUG : 11 go routines active
2

FYI - I have noticed similar problems with crypt over S3 bucket with versioning. But I never investigated it further.

I think it is a bug.

It would help if you could create steps to reproduce this issue.

3

Sure.

  1. I created a new bucket and set it to keep files around for a few days.
  2. I set up a separate configuration to use crypt on top of b2 for the new bucket
  3. I created a new folder on my local computer with the following directory structure:
    keep/this/folder/test.txt
delete/this/folder/test.txt
  4. I synced this new folder to the bucket using the following command: rclone sync --fast-list --delete-excluded --filter='- .DS_Store' test b2-crypt-test:
  5. I deleted the delete folder in my local copy
  6. I synced the local copy to the bucket using the same command as #4 which hid the delete folder and everything in it.

Here's what is output when I try to list delete/this/folder:

$ rclone -vv --b2-version-at="2024-02-02T07:47:00-06:00" lsjson b2-crypt-test:delete/this/folder
2024/02/02 08:05:11 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "-vv" "--b2-version-at=2024-02-02T07:47:00-06:00" "lsjson" "b2-crypt-test:delete/this/folder"]
2024/02/02 08:05:11 DEBUG : Creating backend with remote "b2-crypt-test:delete/this/folder"
2024/02/02 08:05:11 DEBUG : Using config file from "/Users/dw/.config/rclone/rclone.conf"
2024/02/02 08:05:11 DEBUG : Creating backend with remote "b2:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk"
2024/02/02 08:05:11 DEBUG : b2: detected overridden config - adding "{Fyd_P}" suffix to name
2024/02/02 08:05:12 DEBUG : fs cache: renaming cache item "b2:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk" to be canonical "b2{Fyd_P}:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk"
2024/02/02 08:05:12 DEBUG : fs cache: switching user supplied name "b2:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk" for canonical name "b2{Fyd_P}:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk"
[
{"Path":"test.txt","Name":"test.txt","Size":3,"MimeType":"text/plain; charset=utf-8","ModTime":"2024-02-02T13:42:42.805Z","IsDir":false,"ID":"4_zfa71b7c809c16cac88db0f10_f1123231cf78ee4a5_d20240202_m134601_c000_v0001066_t0011_u01706881561039"}
]
2024/02/02 08:05:12 DEBUG : 11 go routines active

Here's what is output when I try to list the file itself:

$ rclone -vv --b2-version-at="2024-02-02T07:47:00-06:00" lsjson b2-crypt-test:delete/this/folder/test.txt
2024/02/02 08:06:59 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "-vv" "--b2-version-at=2024-02-02T07:47:00-06:00" "lsjson" "b2-crypt-test:delete/this/folder/test.txt"]
2024/02/02 08:06:59 DEBUG : Creating backend with remote "b2-crypt-test:delete/this/folder/test.txt"
2024/02/02 08:06:59 DEBUG : Using config file from "/Users/dw/.config/rclone/rclone.conf"
2024/02/02 08:06:59 DEBUG : Creating backend with remote "b2:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk/bg246k60m2ies61s7nf41fvqo8"
2024/02/02 08:06:59 DEBUG : b2: detected overridden config - adding "{Fyd_P}" suffix to name
2024/02/02 08:07:00 DEBUG : fs cache: renaming cache item "b2:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk/bg246k60m2ies61s7nf41fvqo8" to be canonical "b2{Fyd_P}:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk/bg246k60m2ies61s7nf41fvqo8"
2024/02/02 08:07:00 DEBUG : fs cache: switching user supplied name "b2:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk/bg246k60m2ies61s7nf41fvqo8" for canonical name "b2{Fyd_P}:dw-test/0f496h30upkc0mmb925nt4ptqc/q4lukkiicfvsnrq2k651vlb1fk/rlmjj44m1p9hcpti7lkgfvn7gk/bg246k60m2ies61s7nf41fvqo8"
[
]
2024/02/02 08:07:00 DEBUG : 11 go routines active
4

This does appear to be a bug

Here is a simple replication

echo "hello" | rclone rcat b2:cleanup-test/delete/file.txt
sleep 60
rclone delete --b2-hard-delete=false b2:cleanup-test/delete/file.txt

OK

$ rclone lsl --b2-version-at "2024-02-03 11:40:02" b2:cleanup-test/delete/
        6 2024-02-03 11:39:59.886000000 file.txt

Nothing

$ rclone lsl --b2-version-at "2024-02-03 11:40:02" b2:cleanup-test/delete/file.txt

I tried this with s3 too and it worked, so this is a b2 specific bug.

Could you open a new issue on Github about this please with a link to this forum post. This may be hard to fix!

In the mean time you can use the --include trick as a workaround, so instead of

rclone -q --b2-version-at=10d lsjson --max-depth=1 b2-crypt:/Dustin/Personal/Brushes/dW.abr

Do

rclone -q --b2-version-at=10d lsjson --max-depth=1 b2-crypt:/Dustin/Personal/Brushes/ --include "/dW.abr"

That should work for copy also.

1 Like
5

Done, and thanks for the workaround!

2 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.