This is kind of a general question that I didnt manage to find it reply reading some of the PR and comments in github.
I am running:
]# rclone version rclone v1.55.0-DEV - os/type: linux - os/arch: amd64 - go/version: go1.14.12 - go/linking: dynamic - go/tags: none ]# lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 8.3.2011 Release: 8.3.2011 Codename: n/a
To copy content in a file system to Ceph S3 backend. My rclone.conf looks like:
[s3prevessin] type = s3 provider = Ceph env_auth = false access_key_id = XXXXX secret_access_key = XXXX endpoint = host acl = public-read
And the command is simply:
rclone --checksum -P --s3-upload-cutoff=1G --s3-chunk-size=256M --transfers=2 --checkers=4 --log-level DEBUG copy /mnt/media_share/media_data s3prevessin:publicqa/mnt/media_share/media_data
The content under
/mnt/media_share/media_data is about a million of files of different sizes (it'd a CMS system). It's organized on directories e.g.
I cant provide a log as I dont have it. But we have found that rclone compresses directories under this first directory level in a tar file, and then copies that to destination in S3 bucket.
I kind of understand if rclone compresses on the fly and places a tar in the S3 backend but I was kind of surprised to find my source compressed e.g.
Is this expected behavior when there is a huge number of files in the source? it's because the backend is S3?