Read and write on folder with too many files are very slow

Diego_Rosa · July 19, 2021, 9:15pm

I'm using a IAM user with full acess to everything.

2021/07/19 18:16:07 DEBUG : Using config file from "c:\rclone\rclone.conf"
2021/07/19 18:16:07 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "ls" "jmfcloud:" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG" "--log-file=rclone.log"]
2021/07/19 18:16:07 DEBUG : Creating backend with remote "jmfcloud:"
2021/07/19 18:16:07 Failed to create file system for "jmfcloud:": didn't find section in config file

asdffdsa · July 19, 2021, 9:21pm

you need to use the name of the remote from the config file

to get that list of remotes use,
rclone listremotes

Diego_Rosa · July 19, 2021, 9:23pm

JMF:
jmf-cloud:

I changed before the name of the wasabi config to JMF and my bucket name on wasabi is jmfcloud.
The jmf-cloud is the of AWS S3.

asdffdsa · July 19, 2021, 9:24pm

i am getting confused,
you need to run the commands on an existing remote name and bucket, whatever it is...

Diego_Rosa · July 19, 2021, 9:59pm

I did, I have two rclone configurations, one for AWS S3 and one for Wasabi
AWS S3 bucket name is jmf-cloud and is working good.
The wasabi bucket name is jmfcloud and this is what Im not getting right with rclone mount.

[JMFAWS]
type = s3
provider = AWS
env_auth = false
access_key_id = xxx
secret_access_key = xxx
region = sa-east-1
location_constraint = sa-east-1
storage_class = INTELLIGENT_TIERING

[JMF]
type = s3
provider = Wasabi
access_key_id = xxx
secret_access_key = xxx
region = us-east-1
endpoint = s3.wasabisys.com
acl = bucket-owner-full-control

asdffdsa · July 19, 2021, 10:02pm

as i wrote, you need to run command on an existing remote and bucket name, whatever it is.

if you want to run a command using a remote named JMF
then rclone ls JMF: -vv

Diego_Rosa · July 19, 2021, 10:04pm

I just did, still getting the "Failed to ls: AccessDenied: Access Denied
status code: 403, request id: 79F91FE955E01AAF, host id: /5rL5baQRwopUfqpie6dv6kPE3ke5kGtt2lJMyJiM1jDRIzeXorr6qh4mZAZp3oU77dj11zJhPms"

asdffdsa · July 19, 2021, 10:05pm

you need to post the entire command and output, not just a snippet

Diego_Rosa · July 19, 2021, 10:05pm

c:\rclone>rclone ls JMF: -vv
2021/07/19 19:03:20 DEBUG : Using config file from "c:\rclone\rclone.conf"
2021/07/19 19:03:20 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "ls" "JMF:" "-vv"]
2021/07/19 19:03:20 DEBUG : Creating backend with remote "JMF:"
2021/07/19 19:03:22 DEBUG : 6 go routines active
2021/07/19 19:03:22 Failed to ls: AccessDenied: Access Denied
status code: 403, request id: 79F91FE955E01AAF, host id: /5rL5baQRwopUfqpie6dv6kPE3ke5kGtt2lJMyJiM1jDRIzeXorr6qh4mZAZp3oU77dj11zJhPms

asdffdsa · July 19, 2021, 10:08pm

run
rclone ls JMF: --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=rclone.log
and post the file rclone.log

Diego_Rosa · July 19, 2021, 10:11pm

rclone.log (5.6 KB)

asdffdsa · July 19, 2021, 10:16pm

try this in the config file and run the command again...

[JMF]
type = s3
provider = Wasabi
access_key_id = xxx
secret_access_key = xxx
endpoint = s3.us-east-2.wasabisys.com
acl = bucket-owner-full-control

Diego_Rosa · July 19, 2021, 11:01pm

It worked now!
The problem was with the endpoint indeed.
Why is that? This endpoint wasn't showing on the console configuration.

asdffdsa · July 19, 2021, 11:04pm

good, as per that debug log,

Please send all future requests to this endpoint.</Message><Bucket>jmf-logs</Bucket><Endpoint>jmf-logs.s3.us-east-2.wasabisys.com<

as to why...
did you create the bucket using rclone or the wasabi website?
i guess via the website, correct?

Diego_Rosa · July 19, 2021, 11:14pm

Yes, I created the bucket using the website

asdffdsa · July 19, 2021, 11:16pm

that make sense,

you can check the endpoint at the website....

Diego_Rosa · July 19, 2021, 11:40pm

I see, thanks for the help.
I need one last help with the rclone.

All the files that I put on the mount driver, ether if is on aws or wasabi, it set the Creating Date or Motification Date the same, it didnt get the original creation of the file. This is really bad, because we need the latest files to be on top when acessing.

It is in portuguese, but is the Creation Date and Modification Date

asdffdsa · July 19, 2021, 11:55pm

rclone also has no concept of creation time.

as far as i know, with aws s3 and its many clones like wasabi have no concept of creation time.

and for what it is worth, aws s3 is FLAT object storage, no real concept of folders.
folders are emulated, just an abstraction as documented at
https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html

you might try to use
https://rclone.org/docs/#use-server-modtime
i think but not 100% sure, in effect, the upload time is the creation time....

https://forum.rclone.org/t/keep-the-origin-creation-date-rclone/8373/8

Diego_Rosa · July 20, 2021, 12:52pm

Looks like that --use-server-modtime is working for me now.
When I put a object, it preserve the creation time.

I'm getting a issue with wasabi, I deleting a folder that have another folders inside (but just empy folders, no files) and the folders apear again. Weird.

asdffdsa · July 20, 2021, 12:54pm

you need to post a full debug log.....