How to sync local dirs with files to s3 with the same structure

I'm trying to sync 10+ TB of Jfrog Artifactory's data from local disk to s3, but seems that rclone doesn't create directories for me.
On my local disk I have the following structure:

filestore/0a/0a0024aa67e06e2ed037ddfd57ff7f74ed029a42

If I do

rclone sync filestore/ s3:my_bucket/

then rclone sync all the data without the directory structure under the my_bucket, how can I sync my local data so that on the s3 side is excatly the same data structure?

I'm using v1.57.0 rclone

rclone v1.57.0
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

hi,
rclone sync will mirror the source to dest, including the directory structure.

does the source dir have subdirs?
what is the output of

  • tree -d -L 1 filestore
  • rclone lsd filestore -vv

Yes, source have subdirs

tree -d -L 1 filestore
filestore
├── 00
├── 01
├── 02
├── 03
├── 04
├── 05
├── 06
├── 07
├── 08
├── 09
├── 0a
ls -lR filestore/
filestore/:
total 4
drwx------ 2 artifactory artifactory 4096 Nov 29 14:53 0a

filestore/0a:
total 860184
-rw-r----- 1 artifactory artifactory        62 Sep 12  2018 0a10a4df9d2b8cb7db55ea06df8d796679bc39e4
2021/11/29 15:01:54 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsd" "filestore" "-vv"]
2021/11/29 15:01:54 DEBUG : Creating backend with remote "filestore"
2021/11/29 15:01:54 DEBUG : Using config file from "/user/.config/rclone/rclone.conf"
2021/11/29 15:01:54 DEBUG : fs cache: renaming cache item "filestore" to be canonical "/filestore"
          -1 2021-11-29 14:43:58        -1 00
          -1 2021-11-29 14:36:19        -1 01
          -1 2021-11-29 12:47:13        -1 02
          -1 2021-11-29 12:55:48        -1 03
          -1 2021-11-29 14:41:08        -1 04

now rclone tree s3:my_bucket -d --max-depth=1

Seems that adding filestore in the end of bucket name did the magic :smiley:

rclone sync filestore s3:my_bucket/filestore

And the result is:

rclone tree s3:my_bucket/
/
└── filestore
    └── 0a
        ├── 0a04c20af868290194bc42a7892e72ac85b4bc7a

2 directories, 27 files

yes, that is documented here
"It is always the contents of the directory that is synced, not the directory so when source:path is a directory, it's the contents of source:path that are copied, not the directory name and contents. See extended explanation in the copy command above if unsure."

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.