Configuring swift with aws_access_key_id

I’m trying to set up RCLONE with a Swift OpenStack endpoint on NSF’s Jetstream cloud.

The method I’ve been using to authenticate for writing is to use amazon-like credentials with a bucket_endpoint:


ubuntu@rsignell-api-u-1:~$ more ~/.aws/config


In case it’s useful, these credentials were created with:

openstack ec2 credentials create

Does anyone know where these credentials would go in either the S3 or Swift sections of rclone.conf?

Those looks very much like s3 credentials I’d say, so I suggest you try making an S3 remote. There is a place to put an endpoint. Note that a swift endpoint would likely end with /v2 or /v3.

There is S3 middleware for openstack swift or the provider may be using CEPH which has an S3 endpoint.

Note that if you set env_auth to true in the config then rclone should read your .aws/config file.

A bit of searching reveals that they are using CEPH.

@ncw, thanks for the suggestion to try s3 AWS for my OpenStack Swift endpoint!

The command

rclone sync s3:rsignell/nwm/test_week5c swift:rsignell/nwm/test_week5c

is now working with this rclone.conf:

$ more ~/.config/rclone/rclone.conf
type = s3
provider = AWS
env_auth = true
access_key_id = AKIAJWVSISGxxxxxxxxxxxx
secret_access_key = qY/Q3XkVMBegKf4uxxxxxxxxxxxxxxxxxxxxxxxxxxx
region = us-west-2
location_constraint = us-west-2
acl = public-read
storage_class =

type = s3
provider = AWS
env_auth = true
access_key_id = 22e2ce9b70454xxxxxxxxxxxxxxxxxx
secret_access_key = 4ea8f872c4cxxxxxxxxxxxxxxxxxx
region =
location_constraint =
acl = public-read
endpoint =
storage_class =

but it’s way slower than the copy I did of the same dataset between s3 and gce.

The size of the dataset I’m transferring from S3 to Swift is:

(IOOS3) ubuntu@rsignell-api-u-1:~/data$ rclone size s3:rsignell/nwm/test_week5c
Total objects: 1307
Total size: 94.490 GBytes 

And it’s copying at a rate of 3 objects/5 minutes, which means it will take 36 hours to transfer 95GB. !!

Does that mean something is likely wrong with the transfer or I just need different settings?

I suppose transfer rates might depend on netwoking speed, though I calculate 95GB is about 0.75 MByte/s which is quite slow…

Things to try: Use --checksum this will avoid the read of the modtime on S3 which will save a transaction - that should speed things up. --fast-list should also speed things up. And you can increase transfers - say --transfers 16.

Wow! I tried the 95GB/1300object AWS S3 to SWIFT S3 transfer again with the settings

 rclone sync s3:rsignell/nwm/test_week5c swift:rsignell/nwm/test_week5d --checksum --fast-list --transfers 16

and it finished in just one hour!

Thank you! I hope your wife enjoys the flowers! :grinning:


Thank you very much!