Rclone on aws S3 amazing slow in docker containers

The problem happens on several aws ec2 instances.
The same command rclone lsd ":s3,provider=AWS,env_auth:":

  • with rclone 1.55.1
    • takes 0.13s on the host, but 4.4s in two of the docker containers
  • with rclone 1.58.1
    • takes 0.1s on the host and several minutes in the containers

Any idea of what could possibly be going on ?!

N.B:

  • I checked with aws cli, and the times are a bit slower in the containers but not to that extent

What is the problem you are having with rclone?

rclone runs 30 to thousands time slower in docker containers

Which cloud storage system are you using? (eg Google Drive)

aws S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsd ":s3,provider=AWS,env_auth:"

hi,

have you tested using rclone version?

can you post a rclone debug log, and the dockers commands/files you use to run rclone?

there's nothing particular in the logs. Dockerfiles are very long, but the rclone executable was just copied inside.
Anyway I did another test using network=host, i.e using the host network stack, and that is a fast as on the host.
So the slowness happens in the docker network bridge. Just wondering what kind of network calls would slow down rclone more than aws cli.

Anyway here's the log:

2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 DEBUG : Setting --vfs-cache-mode "writes" from environment variable RCLONE_VFS_CACHE_MODE="writes"
2022/05/24 12:12:41 NOTICE: --dump-headers is obsolete - please use --dump headers instead
2022/05/24 12:12:41 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vvv" "--dump-headers" "lsd" ":s3,env_auth:"]
2022/05/24 12:12:41 DEBUG : Creating backend with remote ":s3,env_auth:"
2022/05/24 12:12:41 DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
2022/05/24 12:12:41 NOTICE: Config file "/home/docker/.config/rclone/rclone.conf" not found - using defaults
2022/05/24 12:12:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/05/24 12:12:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/05/24 12:12:41 DEBUG : fs cache: renaming cache item ":s3,env_auth:" to be canonical ":s3{YTu53}:"
2022/05/24 12:12:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/05/24 12:12:41 DEBUG : HTTP REQUEST (req 0xc0006a5a00)
2022/05/24 12:12:41 DEBUG : PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
User-Agent: rclone/v1.57.0
Content-Length: 0
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
Accept-Encoding: gzip

2022/05/24 12:12:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

It hangs on that for minutes

I have the same behaviour using the official rclone docker image:

$ docker run --rm -ti rclone/rclone -vvv --dump-headers lsd ":s3,region=us-east-1,provider=AWS,env_auth:" 
2022/05/24 12:47:24 NOTICE: --dump-headers is obsolete - please use --dump headers instead
2022/05/24 12:47:24 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "-vvv" "--dump-headers" "lsd" ":s3,region=us-east-1,provider=AWS,env_auth:"]
2022/05/24 12:47:24 DEBUG : Creating backend with remote ":s3,region=us-east-1,provider=AWS,env_auth:"
2022/05/24 12:47:24 DEBUG : :s3: detected overridden config - adding "{sxY6V}" suffix to name
2022/05/24 12:47:24 NOTICE: Config file "/config/rclone/rclone.conf" not found - using defaults
2022/05/24 12:47:24 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/05/24 12:47:24 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/05/24 12:47:24 DEBUG : fs cache: renaming cache item ":s3,region=us-east-1,provider=AWS,env_auth:" to be canonical ":s3{sxY6V}:"
2022/05/24 12:47:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/05/24 12:47:24 DEBUG : HTTP REQUEST (req 0xc000862d00)
2022/05/24 12:47:24 DEBUG : PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
User-Agent: rclone/v1.58.1
Content-Length: 0
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
Accept-Encoding: gzip

2022/05/24 12:47:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

switching to network host mode make it instantaneous

last piece of information: on my local computer, using the official rclone docker image, it is fast both on bridge and host mode. So there's something related to my ec2 instances...

I progressed a bit: it seems to be related to the IAM role that is used to give access to our S3 buckets:

  • the slowdown happens on docker network bridge mode when using the role
  • giving explicit AWS credentials fix the problem

For the record, the problem was the ec2 metadata http-put-response-hop-limit that must be increased in docker network bridge mode: e.g.

aws ec2 modify-instance-metadata-options --instance-id $ID --http-put-response-hop-limit 3  --http-endpoint enabled

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.