Rclone is creating a tar on the source directory

[Ruben_Gaspar]

Hello,

This is kind of a general question that I didnt manage to find it reply reading some of the PR and comments in github.

I am running:

]# rclone version
rclone v1.55.0-DEV
- os/type: linux
- os/arch: amd64
- go/version: go1.14.12
- go/linking: dynamic
- go/tags: none

]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 8.3.2011
Release:        8.3.2011
Codename:       n/a

To copy content in a file system to Ceph S3 backend. My rclone.conf looks like:

[s3prevessin]
type = s3
provider = Ceph
env_auth = false
access_key_id = XXXXX
secret_access_key = XXXX
endpoint = host
acl = public-read

And the command is simply:

 rclone --checksum  -P  --s3-upload-cutoff=1G --s3-chunk-size=256M --transfers=2 --checkers=4 --log-level DEBUG copy /mnt/media_share/media_data s3prevessin:publicqa/mnt/media_share/media_data 

The content under /mnt/media_share/media_data is about a million of files of different sizes (it'd a CMS system). It's organized on directories e.g. /mnt/media_share/media_data/2000, /mnt/media_share/media_data/2001, etc.

I cant provide a log as I dont have it. But we have found that rclone compresses directories under this first directory level in a tar file, and then copies that to destination in S3 bucket.
I kind of understand if rclone compresses on the fly and places a tar in the S3 backend but I was kind of surprised to find my source compressed e.g. /mnt/media_share/media_data/2000/directoryA.tar, /mnt/media_share/media_data/2000/directoryB.tar, etc

Is this expected behavior when there is a huge number of files in the source? it's because the backend is S3?

Thank you,
Ruben

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

never heard of rclone having the ability to create .tar files.
and
never heard of rclone copy modifying the source files

without the debug log, how do you know, as a fact, that it is rclone doing that?
can you post it or the relevant parts that show rclone is modifying the source and creating .tar files?

and here on the forum, i do remember a post i contributed to about .tar files
Can rclone tar files off a directory for a copy in flight

[Ruben_Gaspar]

Thanks a lot for the reply and welcome greetings!

In fact it looked strange to me too. I have just tried to reproduce the problem running the command manually e.g.:

rclone --checksum  -P  --s3-upload-cutoff=1G --s3-chunk-size=256M --transfers=2 --checkers=4 --log-level DEBUG --log-file=/tmp/copying.log copy /mnt/master_share/master_data/WebLectures s3prevessin:master/mnt/master_share/master_data/WebLectures

On a big directory structure and I dont manage. So it may come from something else. It's simply very weird. It's may be related to rundeck as we dispatch this jobs from a central node.

Anyway, apologies. Please close the case. And thanks again for your support.

[asdffdsa (jojothehumanmonkey)]

for sure, it is coming from something else.

the case will close itself.