Not able to transfer empty directories from azure file shares to S3 bucket

Help and Support
1

I am trying to transfer my azure file shares data to AWS S3 bucket using RCLONE.

All the directories and subdirectories got transferred to S3 bucket which contains the files. But empty directories were missing in the destination(S3 bucket)

Using below command to perform this copy actions --
cd
rclone.exe -P sync ./ s3:<my_bucket>

I also tries to add flag --create-empty-src-dirs and use below command, but it also not transferring the empty directories to S3 bucket.
rclone.exe -P sync ./ s3:<my_bucket> --create-empty-src-dirs

I also tried to use flag --s3-directory-markers and run the below command but its also not works
rclone.exe -P sync ./ s3:<my_bucket> --s3-directory-markers

I also tried to add directory_markers = true in RLCONE config file, but no result

Below is rclone.conif file that we are using
[s3]
type = s3
provider = AWS
env_auth = true
region = eu-central-1
location_constraint = EU
acl = private
server_side_encryption = AES256
storage_class = STANDARD
profile = GED
directory_markers = true

Please suggest.

2

Leave directory_markers = true in your S3 config

Create some test data directory on Azure containing empty dir.

Run:

rclone sync azure:test_dir s3:bucket/test_dir --create-empty-src-dirs -vv

and post output here

3

I need to transfer all files and directories from azure file shares to AWS S3 bucket.

I have mounted my file share to server then using files as locally to transfer to S3.

Using this below command for now --
cd
rclone.exe -P sync ./ s3:<my_bucket>

4

It is fine - does no matter if your source is local or azure directly.

PS. Can you format commands and stuff you post? It is not easy to read when you do not do this. Enclose things in three ticks markers:

image

So it is will look like this:

my command
5

Create test directory and use the below command --
'''
rclone sync ./ s3:awsd-ged-database-dump/test_dir --create-empty-src-dirs -vv
'''
Below is the ouput --
X:\BOOST-PATCH\test>rclone sync ./ s3:awsd-ged-database-dump/test_dir --create-empty-src-dirs -vv
2023/11/08 08:51:21 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "sync" "./" "s3:awsd-ged-database-dump/test_dir" "--create-empty-src-dirs" "-vv"]
2023/11/08 08:51:21 DEBUG : Creating backend with remote "./"
2023/11/08 08:51:21 DEBUG : Using config file from "C:\Users\AL1103137\AppData\Roaming\rclone\rclone.conf"
2023/11/08 08:51:21 DEBUG : fs cache: renaming cache item "./" to be canonical "//?/X:/BOOST-PATCH/test"
2023/11/08 08:51:21 DEBUG : Creating backend with remote "s3:awsd-ged-database-dump/test_dir"
2023/11/08 08:51:22 DEBUG : rclone: Need to transfer - File not found at Destination
2023/11/08 08:51:22 DEBUG : S3 bucket awsd-ged-database-dump path test_dir: Waiting for checks to finish
2023/11/08 08:51:22 DEBUG : S3 bucket awsd-ged-database-dump path test_dir: Waiting for transfers to finish
2023/11/08 08:51:22 DEBUG : rclone: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2023/11/08 08:51:22 INFO : rclone: Copied (new)
2023/11/08 08:51:22 DEBUG : test1: Making directory
2023/11/08 08:51:22 DEBUG : test2: Making directory
2023/11/08 08:51:22 DEBUG : S3 bucket awsd-ged-database-dump path test_dir: copied 2 directories
2023/11/08 08:51:22 DEBUG : Waiting for deletions to finish
2023/11/08 08:51:22 INFO : testdir-2.txt: Deleted
2023/11/08 08:51:22 INFO : testdir.txt: Deleted
2023/11/08 08:51:22 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 2 / 2, 100%
Deleted: 2 (files), 0 (dirs)
Transferred: 1 / 1, 100%
Elapsed time: 0.7s

2023/11/08 08:51:22 DEBUG : 7 go routines active

6

For markers use ``` not ''' - you can see result in preview. So you do not have to post gibberish until it looks right.

So all works now?

7

Its not working. I am not able to see directories in S3 bucket

8

Please let me know if we can setup one call.

9

No we can not. If you need special support please consider becoming sponsor which might include support contract.

Now back to your problem.

Please provide some evidence - not only say that it does not work.

can you run in the same folder/mount you used for previous test:

rclone tree .

and

rclone tree s3:awsd-ged-database-dump/test_dir
10

Here my test with S3 config including directory_markers = true:

Two empty local directories and some file

$ rclone tree .
/
├── A
│   └── B
└── test.txt

2 directories, 1 files

rclone sync with and without --create-empty-src-dirs

$ rclone sync . S3:test/test
$ rclone tree S3:test/test
/
└── test.txt

0 directories, 1 files

$ rclone sync . S3:test/test --create-empty-src-dirs
$ rclone tree S3:test/test
/
├── A
│   └── B
└── test.txt

2 directories, 1 files
11

X:\BOOST-PATCH\test>rclone tree .
/
├── rclone
├── test1
└── test2

2 directories, 1 files

X:\BOOST-PATCH\test>rclone tree s3:awsd-ged-database-dump/test_dir
/
└── rclone

0 directories, 1 files

12

and what is your rclone config?

run:

rclone config redacted

and post output here