What is the problem you are having with rclone?
while copying files from AWS to GCS, I get this EOF error. Running in GKE cluster.
Run the command 'rclone version' and share the full output of the command.
1.58.1
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp)
// running as a rclone webserver on a GKE cluster as kubernetes deployment:
rclone
rcd
--rc-addr
:5572
The rclone config contents with secrets removed.
// gcs config in the rclone.conf file
[gcs]
type = google cloud storage
project_number = xxxxxxxx
object_acl = bucketOwnerFullControl
bucket_acl = private
bucket_policy_only = true
location = europe-west2
storage_class = STANDARD
// on webserver pod running as environment vars for the remote
RCLONE_RC_ENABLE_METRICS: true
RCLONE_CONFIG_FEATURESPACE_ACCESS_KEY_ID: <set to the key 'access_key_id' in secret 'stg-rclone-webserver-featurespace-oci-creds'> Optional: false
RCLONE_CONFIG_FEATURESPACE_ENDPOINT:
RCLONE_CONFIG_FEATURESPACE_SECRET_ACCESS_KEY: <set to the key 'secret_access_key' in secret 'stg-rclone-webserver-featurespace-oci-creds'> Optional: false
RCLONE_CONFIG_FEATURESPACE_TYPE: s3
RCLONE_CONFIG_FEATURESPACE_REGION: eu-west-2
RCLONE_RC_USER: <set to the key 'rclone_user' in secret 'stg-rclone-webserver-creds'> Optional: false
RCLONE_RC_PASS: <set to the key 'rclone_password' in secret 'stg-rclone-webserver-creds'> Optional: false
RCLONE_WEB_GUI: true
RCLONE_WEB_GUI_NO_OPEN_BROWSER: true
A log from the command with the -vv flag
2022/09/22 17:45:41 DEBUG : --max-age 1d to 2022-09-21 17:45:41.39214167 +0000 UTC m=-86399.934854412
2022/09/22 17:45:41 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "rc" "--rc-addr=10.0.32.119:5572" "--rc-user=xxx" "--rc-pass=xxxxxxx" "sync/copy" "srcFs=featurespace:dev-dojo-featurespace-data/" "dstFs=gcs:ps-data-mdm-dev-featurespace/" "--max-age" "1d" "--timeout" "20m" "--no-traverse" "--update" "--use-server-modtime" "-vvv"]
2022/09/22 17:52:33 DEBUG : 2 go routines active
2022/09/22 17:52:33 Failed to rc: connection failed: Post "http://10.0.32.119:5572/sync/copy": EOF
I'm running the rclone as webserver on a GKE cluster and running a cronjob within the same cluster, to copy data from a AWS S3 bucket to GCS bucket, getting this EOF error. Sometimes it working and sometimes I get this error. There are lots of files that gets checked and then copies only the new files.