What is the problem you are having with rclone?
Im runing rclone in Cloud in kubernetes in Jenkins agent container. I have a job that sync many files for different products. This jenkins agent pod had 2GB memory limit. One of the products has grown to about 870k files in COS S3. and all of a sudden sync for this product is failing. I`m syncing between S3 buckets.
I have increase the memory from 2GB to 5GB on the Pod/container and it still crashes. It even look like it crashes during the checking of file differences.
Here is output from the Jenkins log:
16:41:54 2022-10-28 14:41:54 [INFO] Publishing **** to staging
16:42:55 2022/10/28 14:42:55 INFO :
16:42:55 Transferred: 0 B / 0 B, -, 0 B/s, ETA -
16:42:55 Checks: 299425 / 309432, 97%
16:42:55 Elapsed time: 1m0.5s
16:42:55
16:43:07 /tmp/jenkins1505612189605231137.sh: line 42: 177 Killed
rclone sync ${OVERWRITE}--no-update-modtime --metadata --checksum --checkers=6 --transfers=3 COS_US_SOUTH:ibm-docs-dev/"$PRODUCT_KEY" COS_US_SOUTH:ibm-docs-stage/"$PRODUCT_KEY" -v
16:43:07 [INFO] Status: 137
and the jenkins job continues as if nothing happened. it means the container does not run OOM.
I have red several good forum posts here on the forum so you can see I tried to tune the checkers and transfers down from defaults.
I would like to hear anyones opinion on what can go wrong or what I should try to pinpont the problem.
Run the command 'rclone version' and share the full output of the command.
rclone v1.60.0-beta.6355.67fd60275
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 4.15.0-175-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.3
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Cloud Object Storage S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone sync ${OVERWRITE}--no-update-modtime --metadata --checksum --checkers=6 --transfers=3 COS_US_SOUTH:ibm-docs-dev/"$PRODUCT_KEY" COS_US_SOUTH:ibm-docs-stage/"$PRODUCT_KEY" -v
another command I tried was with a DEBUG and dry run, and it failed too
DEBUG : rclone: Version "v1.60.0-beta.6355.67fd60275" starting with parameters ["rclone" "sync" "--no-update-modtime" "--metadata" "--checksum" "--checkers=6" "--transfers=3" "--dry-run" "--log-file=rclone_log.txt" "--log-level" "DEBUG" "COS_US_SOUTH:ibm-docs-dev/XYZ" "COS_US_SOUTH:ibm-docs-stage/XYZ"]
The rclone config contents with secrets removed.
[COS_US_SOUTH]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = ****
secret_access_key = ****
endpoint = s3.us-south.cloud-object-storage.appdomain.cloud
A log from the command with the -vv
flag
I have used lgo level DEBUG and output to file.
After abut1,5 minute, 90MB of log output (about milion log lines) it end with no errors.
Same output for every other file it was checking.
...
2022/10/31 09:00:39 DEBUG : p10eai/BC552409.htm: md5 = 7e43768d1a969854ec3c7fe063d2352d OK
2022/10/31 09:00:39 DEBUG : p10eai/BC552409.htm: Size and md5 of src and dst objects identical
2022/10/31 09:00:39 DEBUG : p10eai/BC552409.htm: Unchanged skipping
2022/10/31 09:00:39 DEBUG : p10eai/BC55240A.htm: md5 = 58f9375c20d0b77cc6c59f27022da0c1 OK
2022/10/31 09:00:39 DEBUG : p10eai/BC55240A.htm: Size and md5 of src and dst objects identical
2022/10/31 09:00:39 DEBUG : p10eai/BC55240A.htm: Unchanged skipping
This image shows top about 3 seconds before the rclone process was closed so it reached about 31%MEM