Rclone Failed to rc: connection failed: Post "http://10.0.32.119:5572/sync/copy": EOF

What is the problem you are having with rclone?

while copying files from AWS to GCS, I get this EOF error. Running in GKE cluster.

Run the command 'rclone version' and share the full output of the command.

1.58.1

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

// running as a rclone webserver on a GKE cluster as kubernetes deployment:

     rclone
      rcd
      --rc-addr
      :5572

The rclone config contents with secrets removed.

// gcs config in the rclone.conf file
    [gcs]
    type = google cloud storage
    project_number = xxxxxxxx
    object_acl = bucketOwnerFullControl
    bucket_acl = private
    bucket_policy_only = true
    location = europe-west2
    storage_class = STANDARD

// on webserver pod running as environment vars for the remote
      RCLONE_RC_ENABLE_METRICS:                      true
      RCLONE_CONFIG_FEATURESPACE_ACCESS_KEY_ID:      <set to the key 'access_key_id' in secret 'stg-rclone-webserver-featurespace-oci-creds'>  Optional: false
      RCLONE_CONFIG_FEATURESPACE_ENDPOINT:
      RCLONE_CONFIG_FEATURESPACE_SECRET_ACCESS_KEY:  <set to the key 'secret_access_key' in secret 'stg-rclone-webserver-featurespace-oci-creds'>  Optional: false
      RCLONE_CONFIG_FEATURESPACE_TYPE:               s3
      RCLONE_CONFIG_FEATURESPACE_REGION:             eu-west-2
      RCLONE_RC_USER:                                <set to the key 'rclone_user' in secret 'stg-rclone-webserver-creds'>      Optional: false
      RCLONE_RC_PASS:                                <set to the key 'rclone_password' in secret 'stg-rclone-webserver-creds'>  Optional: false
      RCLONE_WEB_GUI:                                true
      RCLONE_WEB_GUI_NO_OPEN_BROWSER:                true

A log from the command with the -vv flag

2022/09/22 17:45:41 DEBUG : --max-age 1d to 2022-09-21 17:45:41.39214167 +0000 UTC m=-86399.934854412
2022/09/22 17:45:41 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "rc" "--rc-addr=10.0.32.119:5572" "--rc-user=xxx" "--rc-pass=xxxxxxx" "sync/copy" "srcFs=featurespace:dev-dojo-featurespace-data/" "dstFs=gcs:ps-data-mdm-dev-featurespace/" "--max-age" "1d" "--timeout" "20m" "--no-traverse" "--update" "--use-server-modtime" "-vvv"]
2022/09/22 17:52:33 DEBUG : 2 go routines active
2022/09/22 17:52:33 Failed to rc: connection failed: Post "http://10.0.32.119:5572/sync/copy": EOF

I'm running the rclone as webserver on a GKE cluster and running a cronjob within the same cluster, to copy data from a AWS S3 bucket to GCS bucket, getting this EOF error. Sometimes it working and sometimes I get this error. There are lots of files that gets checked and then copies only the new files.

The rc command is timing out by the look of it.

You can increase --timeout, but a better approach is to start an async job

i will try, thank you ncw.

1 Like