What is the problem you are having with rclone?
I'm trying to copy bucket from one s3 storage host to another. But if I try to use tree command on source bucket or copy/sync command, they can not be finished and cause a large memory consumption.
Run the command 'rclone version' and share the full output of the command.
os/version: darwin 12.4 (64 bit)
os/kernel: 21.5.0 (arm64)
Which cloud storage system are you using? (eg Google Drive)
Local company's AWS S3 implementation.
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
rclone -vv tree s3:tvisual_accounts
The rclone config contents with secrets removed.
type = s3
provider = AWS
access_key_id = *
secret_access_key = *
region = msk
endpoint = https://s3.*.ru
A log from the command with the
2022/07/14 19:18:40 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "-vv" "tree" "s3:tvisual_accounts"]
2022/07/14 19:18:40 DEBUG : Creating backend with remote "s3:tvisual_accounts"
2022/07/14 19:18:40 DEBUG : Using config file from "/Users/*/.config/rclone/rclone.conf"
July 14, 2022, 4:31pm
what does that mean?
are you using AWS or what?
Yes. It is AWS S3 tuned for a company needs. Installed on company server, not AWS cloud
July 14, 2022, 4:45pm
guess i still do not undertand what
AWS software is run on the local server?
AWS created the software, using minio or what?
and what does
infinite loop mean?
I know only that it is s3 and as sre team say it implements 90% of AWS s3 interfaces.
Infinite loop I mean that this tree command execute infinitely. It doesn’t finish. But only for this bucket. Another bucket with only one directory inside it copying without problem
It's easier to have numbers rather than words that could mean different things to different people.
What does large memory consumption mean? 1GB / 4GB / 8GB / 64GB?
How many objects are in the bucket you are trying to list? How long are you waiting? Did you try --fast-list?
July 14, 2022, 5:20pm
if the cloud provider is not offical
AWS S3, then need to change
provider = AWS
provider = Other
Thank you very much:) It really helps with my problem. After changing provider to "Other" it started to work as expected. I've been thinking all the time that if it implements s3 api I should use AWS provider...
July 17, 2022, 5:25pm
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.