Unable to sync data between two containers in different environments(from env1 to env2). Received the following error:
Failed to sync: not deleting files as there were IO errors
The demographics of container is as follows:
74TB of Object storage.
150 Million Objects.
What is your rclone version (output from rclone version)
rclone v1.47.0
Which OS you are using and how many bits (eg Windows 7, 64 bit)
RHEL 7
Which cloud storage system are you using? (eg Google Drive)
Openstack Swift
The command you were trying to run (eg rclone copy /tmp remote:tmp)
[env1]
type = swift
env_auth = false
user = robot1
key = UUUUUUUUUUUUUUUUUUUUUUU
auth = https://xyz.com:5000/v2.0
tenant = machine1-reports
tenant_id = XXXXXXXXXXXXXXXXXXXXXXXXXXX
region = machine1.region
[env2]
type = swift
env_auth = true
user = robot2
key = YYYYYYYYYYYYYYYYYYYYYYYYYYY
auth = https://abc.com:5000/v3
domain = abc.xyz.com
tenant = machine2-reports
tenant_domain = abc.xyz.com
region = machine2.region
A log from the command with the -vv flag
2020/09/28 17:22:13 DEBUG : rclone: Version "v1.47.0" starting with parameters ["/usr/bin/rclone" "-vvv" "-P" "sync" "env1:test" "env2:test" "--retries=5" "--transfers=4" "--checkers=4" "--tpslimit=2" "--drive-pacer-min-sleep=100ms" "--bwlimit=9M" "--drive-chunk-size=512M"]
2020/09/28 17:22:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/09/28 17:22:13 INFO : Starting bandwidth limiter at 9MBytes/s
2020/09/28 17:22:13 INFO : Starting HTTP transaction limiter: max 2 transactions/s with burst 1
2020-09-28 17:28:37 ERROR : : error reading source directory: object not found
2020-09-28 17:28:37 INFO : Swift container test: Waiting for checks to finish
2020-09-28 17:28:37 INFO : Swift container test: Waiting for transfers to finish
2020-09-28 17:28:37 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:28:37 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:28:37 ERROR : Attempt 1/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:34:53 ERROR : : error reading source directory: object not found
2020-09-28 17:34:53 INFO : Swift container test: Waiting for checks to finish
2020-09-28 17:34:53 INFO : Swift container test: Waiting for transfers to finish
2020-09-28 17:34:53 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:34:53 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:34:53 ERROR : Attempt 2/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:41:08 ERROR : : error reading source directory: object not found
2020-09-28 17:41:08 INFO : Swift container test: Waiting for checks to finish
2020-09-28 17:41:08 INFO : Swift container test: Waiting for transfers to finish
2020-09-28 17:41:08 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:41:08 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:41:08 ERROR : Attempt 3/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:47:24 ERROR : : error reading source directory: object not found
2020-09-28 17:47:24 INFO : Swift container test: Waiting for checks to finish
2020-09-28 17:47:24 INFO : Swift container test: Waiting for transfers to finish
2020-09-28 17:47:24 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:47:24 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:47:24 ERROR : Attempt 4/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:53:43 ERROR : : error reading source directory: object not found
2020-09-28 17:53:43 INFO : Swift container test: Waiting for checks to finish
2020-09-28 17:53:43 INFO : Swift container test: Waiting for transfers to finish
2020-09-28 17:53:43 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:53:43 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:53:43 ERROR : Attempt 5/5 failed with 2 errors and: not deleting files as there were IO errors
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 2 (retrying may help)
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 31m30.2s
2020/09/28 17:53:43 Failed to sync: not deleting files as there were IO errors
Thank you for your reply. But Still having the issues with rclone after installing v1.53.1.
2020/09/28 19:00:17 DEBUG : rclone: Version "v1.53.1" starting with parameters ["/usr/bin/rclone" "-vvv" "-P" "sync" "env1:test" "env2:test" "--retries=5" "--transfers=4" "--checkers=4" "--tpslimit=2" "--drive-pacer-min-sleep=100ms" "--bwlimit=9M" "--drive-chunk-size=512M"]
2020/09/28 19:00:17 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/09/28 19:00:17 INFO : Starting bandwidth limiter at 9MBytes/s
2020/09/28 19:00:17 INFO : Starting HTTP transaction limiter: max 2 transactions/s with burst 1
2020/09/28 19:00:17 DEBUG : Creating backend with remote "env1:test"
2020/09/28 19:00:17 DEBUG : Creating backend with remote "env2:test"
2020-09-28 19:06:34 ERROR : 20130728060014.test.57606696.tar: dangling large object with no contents
2020-09-28 19:32:25 ERROR : : error reading source directory: directory not found
2020-09-28 19:32:25 DEBUG : Swift container test Waiting for checks to finish
2020-09-28 19:32:25 DEBUG : Swift container test Waiting for transfers to finish
2020-09-28 19:32:25 ERROR : Swift container test not deleting files as there were IO errors
2020-09-28 19:32:25 ERROR : Swift container test not deleting directories as there were IO errors
2020-09-28 19:32:25 INFO : There was nothing to transfer
2020-09-28 19:32:25 ERROR : Attempt 1/5 failed with 1 errors and: directory not found
2020-09-28 19:38:50 ERROR : 20130728060014.test.57606696.tar: dangling large object with no contents
2020-09-28 20:04:32 ERROR : 000: Failed to copy: Too Large Object
2020-09-28 20:04:33 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/1014": EOF)
2020-09-28 20:04:33 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020-09-28 20:04:33 DEBUG : 1014: Received error: Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/1014": EOF - low level retry 1/10
2020-09-28 20:04:35 DEBUG : pacer: Reducing sleep to 0s
2020-09-28 20:04:35 ERROR : 20090405030535.test.22786080.tar: Failed to copy: Too Large Object
2020-09-28 20:04:37 ERROR : 1014: Failed to copy: Too Large Object
2020-09-28 20:04:38 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405035643.test.22786082.tar": EOF)
2020-09-28 20:04:38 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020-09-28 20:04:38 DEBUG : 20090405035643.test.22786082.tar: Received error: Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405035643.test.22786082.tar": EOF - low level retry 1/10
2020-09-28 20:04:40 DEBUG : pacer: Reducing sleep to 0s
2020-09-28 20:04:40 ERROR : 20090405035643.test.22786082.tar.sum: Failed to copy: Too Large Object
2020-09-28 20:04:41 ERROR : 20090405035643.test.22786082.tar: Failed to copy: Bad Request
2020-09-28 20:04:41 ERROR : 20090405040116.test.22786084.tar: Failed to copy: Too Large Object
2020-09-28 20:04:45 ERROR : 20090405040549.test.22786086.tar: Failed to copy: Too Large Object
2020-09-28 20:04:47 ERROR : 20090405041023.test.22786088.tar: Failed to copy: Too Large Object
2020-09-28 20:09:37 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405030535.test.22786080.tar.sum": net/http: timeout awaiting response headers)
2020-09-28 20:09:37 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020-09-28 20:09:37 DEBUG : 20090405030535.test.22786080.tar.sum: Received error: Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405030535.test.22786080.tar.sum": net/http: timeout awaiting response headers - low level retry 1/10
2020-09-28 20:09:38 DEBUG : pacer: Reducing sleep to 0s```
rclone -vvv -P sync env1:test env2:test --retries=5 --transfers=4 --filter "- *tar*" --fast-list. Currently, I am running the above command removed most of the arguments.. It is in initial stages.. let see if it works.