Sync empty directories to B2

Help and Support
1

I followed the rules and used the template below, but the answer is likely straightforward. I found posts about this issue but they are 2.5 to over 5 years old. I upgraded to the latest binary and tried removing --fast-list and adding --create-empty-src-dirs in various combinations, but it never syncs the empties to B2. I understand B2 doesn't store empty directories, but does rclone use some workaround? Or do I have to put something in them so they aren't empty? It's not clear to me. This post --create-empty-src-dirs issue with B2 matches my issue sort of, but I think they are syncing to local, while I'm trying to sync to B2.

What is the problem you are having with rclone?

Rclone is not syncing empty directories to Backblaze B2

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2

  • os/version: ubuntu 16.04 (64 bit)
  • os/kernel: 4.4.0-83-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2 (and AWS S3 but only testing B2 now)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/sbin/rclone sync --transfers=3 --fast-list --bwlimit "06:00,2000K 22:00,2500K" --create-empty-src-dirs -v /NetworkShare/backup/ B2:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

This doesn't seem relevant to this question.

A log from the command that you were trying to run with the -vv flag

2024/11/30 10:57:15 INFO : Starting bandwidth limiter at 1.953Mi Byte/s
2024/11/30 10:57:48 INFO : There was nothing to transfer
2024/11/30 10:57:48 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 59305 / 59305, 100%
Elapsed time: 32.8s

2

AFAIK empty dirs are not supported by B2. Nothing rclone can do about it.

If you have to store there data with empty dirs you have to encapsulate it within objects. Look at some backup software like restic or kopia.

3

b2 offers a s3 compatible api and rclone supports empty dir on s3.
https://rclone.org/s3/#s3-directory-markers
would have to test that to know for sure.

4

I tried adding "--s3-directory-markers" but still didn't upload empty dirs. Hmmm.. Anybody know how this works? The link asdffdsa provided looked promising.

5

it is relevant, please answer that question?

6

Alright here it is:

[B2]
type = crypt
remote = B2_BASE:redacted
filename_encryption = standard
password = XXX
password2 =

[B2_BASE]
type = b2
account = XXX
key = XXX
endpoint =

[S3]
type = crypt
remote = S3_BASE:redacted
filename_encryption = standard
password = XXX
password2 =

[S3_BASE]
type = s3
env_auth = false
access_key_id = XXX
secret_access_key = XXX
region = us-east-1
endpoint =
location_constraint =
acl = private
server_side_encryption =
storage_class = ONEZONE_IA

[S3_DEEPGLACIER]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-east-1
acl = private
storage_class = DEEP_ARCHIVE

[local]
type = crypt
remote = local_base:/NetworkShare/temp
filename_encryption = standard
directory_name_encryption = true
password = XXX

[local_base]
type = local

7

it does work

# list source
+ rclone tree /home/user01/zork
/
├── empty.dir
└── non.empty.dir
    └── file.ext

# list dest
+ rclone tree b2:zorkzork
NOTICE: Failed to tree: directory not found

# copy source to dest
+ rclone copy /home/user01/zork b2:zorkzork --create-empty-src-dirs --s3-directory-markers -vv
DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "copy" "/home/user01/zork" "b2:zorkzork" "--create-empty-src-dirs" "--s3-directory-markers" "-vv"]
DEBUG : Creating backend with remote "/home/user01/zork"
DEBUG : Using config file from "/home/user01/.config/rclone/rclone.conf"
DEBUG : Creating backend with remote "b2:zorkzork"
DEBUG : b2: detected overridden config - adding "{juk_h}" suffix to name
DEBUG : fs cache: renaming cache item "b2:zorkzork" to be canonical "b2{juk_h}:zorkzork"
DEBUG : empty.dir: Making directory
INFO  : S3 bucket zorkzork: Bucket "zorkzork" created with ACL ""
DEBUG : empty.dir/: Creating directory marker
DEBUG : non.empty.dir: Making directory
DEBUG : non.empty.dir/: Creating directory marker
DEBUG : non.empty.dir/file.ext: Need to transfer - File not found at Destination
DEBUG : S3 bucket zorkzork: Waiting for checks to finish
DEBUG : S3 bucket zorkzork: Waiting for transfers to finish
DEBUG : non.empty.dir/file.ext: md5 = 093557a3a0cef26d4c9da0f40aeb0d12 OK
INFO  : non.empty.dir/file.ext: Copied (new)
INFO  : 
Transferred:   	        308 B / 308 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         1.3s

# list dest
+ rclone tree b2:zorkzork
/
├── empty.dir
└── non.empty.dir
    └── file.ext
8

It works now! On my AWS S3 bucket anyway. I'll experiment with B2 but as you said I'd have to switch to S3 API which seems easy.

I'm not sure if I was missing it because rclone sync kept telling me nothing was transferred, or because I took out --create-empty-src-dirs when I added in --s3-directory-markers.