--create-empty-src-dirs issue with B2

Help and Support
1

What is the problem you are having with rclone?

I am trying to sync a test Backblaze B2 bucket with empty folders to a local folder. When using --create-empty-src-dirs combined with the --fast-list flag, the empty source folders are not created in destination. However, the same command without the --fast-lit flags creates the empty folders as expected.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
  • os/kernel: 10.0.19044.1706 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync b2:grupoipi D:\B2 --exclude ".bzEmpty" --tpslimit 10 --fast-list -vv --create-empty-src-dirs

The rclone config contents with secrets removed.

[b2]
type = b2
account = 003b****0007
key = K00****e4A

A log from the command with the -vv flag

With --fast-list flag

rclone sync b2:grupoipi D:\B2 --exclude ".bzEmpty" --tpslimit 10 --fast-list -vv --create-empty-src-dirs
2022/05/18 22:39:56 INFO  : Starting transaction limiter: max 10 transactions/s with burst 1
2022/05/18 22:39:56 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "sync" "b2:grupoipi" "D:\\B2" "--exclude" ".bzEmpty" "--tpslimit" "10" "--fast-list" "-vv" "--create-empty-src-dirs"]
2022/05/18 22:39:56 DEBUG : Creating backend with remote "b2:grupoipi"
2022/05/18 22:39:56 DEBUG : Using config file from "C:\\Users\\Jordi\\AppData\\Roaming\\rclone\\rclone.conf"
2022/05/18 22:39:57 DEBUG : Creating backend with remote "D:\\B2"
2022/05/18 22:39:57 DEBUG : fs cache: renaming cache item "D:\\B2" to be canonical "//?/D:/B2"
2022/05/18 22:39:57 DEBUG : Folder 1/.bzEmpty: Excluded from sync (and deletion)
2022/05/18 22:39:57 DEBUG : Folder 1/Subfolder 1/.bzEmpty: Excluded from sync (and deletion)
2022/05/18 22:39:57 DEBUG : Folder 1/Subfolder 2/.bzEmpty: Excluded from sync (and deletion)
2022/05/18 22:39:57 DEBUG : Local file system at //?/D:/B2: Waiting for checks to finish
2022/05/18 22:39:57 DEBUG : Local file system at //?/D:/B2: Waiting for transfers to finish
2022/05/18 22:39:57 DEBUG : Waiting for deletions to finish
2022/05/18 22:39:57 INFO  : There was nothing to transfer
2022/05/18 22:39:57 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.0s

2022/05/18 22:39:57 DEBUG : 6 go routines active

Without --fast-list flag

rclone sync b2:grupoipi D:\B2 --exclude ".bzEmpty" --tpslimit 10 -vv --create-empty-src-dirs
2022/05/18 22:46:07 INFO  : Starting transaction limiter: max 10 transactions/s with burst 1
2022/05/18 22:46:07 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "sync" "b2:grupoipi" "D:\\B2" "--exclude" ".bzEmpty" "--tpslimit" "10" "-vv" "--create-empty-src-dirs"]
2022/05/18 22:46:07 DEBUG : Creating backend with remote "b2:grupoipi"
2022/05/18 22:46:07 DEBUG : Using config file from "C:\\Users\\Jordi\\AppData\\Roaming\\rclone\\rclone.conf"
2022/05/18 22:46:07 DEBUG : Creating backend with remote "D:\\B2"
2022/05/18 22:46:07 DEBUG : fs cache: renaming cache item "D:\\B2" to be canonical "//?/D:/B2"
2022/05/18 22:46:08 DEBUG : Folder 1/.bzEmpty: Excluded
2022/05/18 22:46:08 DEBUG : Folder 1/Subfolder 2/.bzEmpty: Excluded
2022/05/18 22:46:08 DEBUG : Folder 1/Subfolder 1/.bzEmpty: Excluded
2022/05/18 22:46:08 DEBUG : Local file system at //?/D:/B2: Waiting for checks to finish
2022/05/18 22:46:08 DEBUG : Local file system at //?/D:/B2: Waiting for transfers to finish
2022/05/18 22:46:08 DEBUG : Folder 1/Subfolder 1: Making directory
2022/05/18 22:46:08 DEBUG : Folder 1/Subfolder 2: Making directory
2022/05/18 22:46:08 DEBUG : Local file system at //?/D:/B2: copied 2 directories
2022/05/18 22:46:08 DEBUG : Waiting for deletions to finish
2022/05/18 22:46:08 INFO  : There was nothing to transfer
2022/05/18 22:46:08 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.2s

2022/05/18 22:46:08 DEBUG : 6 go routines active
2

How have you got empty folders? As far as I'm aware B2 doesn't support empty folders?

Can you do

rclone lsr -R --disable ListR b2:grupoipi D:\B2

And this (which uses --fast-list internally)

rclone lsr -R b2:grupoipi D:\B2

And post the output? I'd like to see if the intermediate directories are missing there.

3

Backblazes does not allow empty folders, but simulates empty folders by creating dummy files called .bzEmpty.

I've tried both commands but the output is unkown command "lsr"

rclone lsr -R --disable ListR b2:grupoipi D:\B2
Error: unknown command "lsr" for "rclone"

Did you mean this?
        ls
        lsd
        lsf
        lsl

Run 'rclone --help' for usage.
You could use 'rclone selfupdate' to get latest features.

2022/05/19 11:50:11 Fatal error: unknown command "lsr" for "rclone"

Did you mean this?
        ls
        lsd
        lsf
        lsl

rclone lsr -R b2:grupoipi D:\B2
Error: unknown command "lsr" for "rclone"

Did you mean this?
        ls
        lsd
        lsf
        lsl

Run 'rclone --help' for usage.
You could use 'rclone selfupdate' to get latest features.

2022/05/19 11:55:56 Fatal error: unknown command "lsr" for "rclone"

Did you mean this?
        ls
        lsd
        lsf
        lsl

The bucket has only one folder named Folder 1 and two subfolders named Subfolder 1 and Subfolder 2 and three dummy files .bzEmpty (one for each folder).

rclone lsf b2:grupoipi -R
Folder 1/.bzEmpty
Folder 1/Subfolder 1/.bzEmpty
Folder 1/Subfolder 2/.bzEmpty
Folder 1/
Folder 1/Subfolder 1/
Folder 1/Subfolder 2/
4

Sorry I meant lsf

Can you run the lsf commands above with the --exclude so

rclone lsf -R --exclude ".bzEmpty" --disable ListR b2:grupoipi D:\B2

rclone lsf -R --exclude ".bzEmpty" b2:grupoipi D:\B2

I suspect this is caused by the exclude.

5

Both commands have the same otuput:

rclone lsf -R --exclude ".bzEmpty" --disable ListR b2:grupoipi
Folder 1/
Folder 1/Subfolder 1/
Folder 1/Subfolder 2/

rclone lsf -R --exclude ".bzEmpty" b2:grupoipi
Folder 1/
Folder 1/Subfolder 1/
Folder 1/Subfolder 2/
6

This turned out to be a subtle bug in the tree making code of --fast-list.

Have a go with this:

v1.59.0-beta.6156.0dfca41a9.fix-fast-list on branch fix-fast-list (uploaded in 15-30 mins)

7

It certainly seemed a subtle bug. Can you upload the Windows version to try it out?

8

I've given the builder a kick, it should hopefully produce a windows binary in 10 mins or so.

9

The v1.59.0-beta.6156.0dfca41a9.fix-fast-list solves the issue. I've tested it also with a bucket with 200.000 files and tens of thousands of empty folders and worked without any issue.

10

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.59

11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.