Rclone - Failed to sync: not deleting files as there were IO errors

What is the problem you are having with rclone?

Unable to sync data between two containers in different environments(from env1 to env2). Received the following error:
Failed to sync: not deleting files as there were IO errors
The demographics of container is as follows:

  1. 74TB of Object storage.
  2. 150 Million Objects.

What is your rclone version (output from rclone version)

rclone v1.47.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

RHEL 7

Which cloud storage system are you using? (eg Google Drive)

Openstack Swift

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vvv -P sync env1:test env2:test --retries=5 --transfers=4 --checkers=4 --tpslimit=2 --drive-pacer-min-sleep=100ms --bwlimit=9M --drive-chunk-size=512M

The rclone config contents with secrets removed.

[env1]
type = swift
env_auth = false
user = robot1
key = UUUUUUUUUUUUUUUUUUUUUUU
auth = https://xyz.com:5000/v2.0
tenant = machine1-reports
tenant_id = XXXXXXXXXXXXXXXXXXXXXXXXXXX
region = machine1.region

[env2]
type = swift
env_auth = true
user = robot2
key = YYYYYYYYYYYYYYYYYYYYYYYYYYY
auth = https://abc.com:5000/v3
domain = abc.xyz.com
tenant =  machine2-reports
tenant_domain = abc.xyz.com
region = machine2.region

A log from the command with the -vv flag

2020/09/28 17:22:13 DEBUG : rclone: Version "v1.47.0" starting with parameters ["/usr/bin/rclone" "-vvv" "-P" "sync" "env1:test" "env2:test" "--retries=5" "--transfers=4" "--checkers=4" "--tpslimit=2" "--drive-pacer-min-sleep=100ms" "--bwlimit=9M" "--drive-chunk-size=512M"]
2020/09/28 17:22:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/09/28 17:22:13 INFO  : Starting bandwidth limiter at 9MBytes/s
2020/09/28 17:22:13 INFO  : Starting HTTP transaction limiter: max 2 transactions/s with burst 1
2020-09-28 17:28:37 ERROR : : error reading source directory: object not found
2020-09-28 17:28:37 INFO  : Swift container test: Waiting for checks to finish
2020-09-28 17:28:37 INFO  : Swift container test: Waiting for transfers to finish
2020-09-28 17:28:37 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:28:37 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:28:37 ERROR : Attempt 1/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:34:53 ERROR : : error reading source directory: object not found
2020-09-28 17:34:53 INFO  : Swift container test: Waiting for checks to finish
2020-09-28 17:34:53 INFO  : Swift container test: Waiting for transfers to finish
2020-09-28 17:34:53 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:34:53 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:34:53 ERROR : Attempt 2/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:41:08 ERROR : : error reading source directory: object not found
2020-09-28 17:41:08 INFO  : Swift container test: Waiting for checks to finish
2020-09-28 17:41:08 INFO  : Swift container test: Waiting for transfers to finish
2020-09-28 17:41:08 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:41:08 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:41:08 ERROR : Attempt 3/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:47:24 ERROR : : error reading source directory: object not found
2020-09-28 17:47:24 INFO  : Swift container test: Waiting for checks to finish
2020-09-28 17:47:24 INFO  : Swift container test: Waiting for transfers to finish
2020-09-28 17:47:24 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:47:24 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:47:24 ERROR : Attempt 4/5 failed with 2 errors and: not deleting files as there were IO errors
2020-09-28 17:53:43 ERROR : : error reading source directory: object not found
2020-09-28 17:53:43 INFO  : Swift container test: Waiting for checks to finish
2020-09-28 17:53:43 INFO  : Swift container test: Waiting for transfers to finish
2020-09-28 17:53:43 ERROR : Swift container test: not deleting files as there were IO errors
2020-09-28 17:53:43 ERROR : Swift container test: not deleting directories as there were IO errors
2020-09-28 17:53:43 ERROR : Attempt 5/5 failed with 2 errors and: not deleting files as there were IO errors
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 2 (retrying may help)
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:    31m30.2s
2020/09/28 17:53:43 Failed to sync: not deleting files as there were IO errors

hello and welcome to the forum,

that version of rclone is ancient.
update using this
https://rclone.org/downloads/#script-download-and-install

Thank you for your reply. But Still having the issues with rclone after installing v1.53.1.

2020/09/28 19:00:17 DEBUG : rclone: Version "v1.53.1" starting with parameters ["/usr/bin/rclone" "-vvv" "-P" "sync" "env1:test" "env2:test" "--retries=5" "--transfers=4" "--checkers=4" "--tpslimit=2" "--drive-pacer-min-sleep=100ms" "--bwlimit=9M" "--drive-chunk-size=512M"]
2020/09/28 19:00:17 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/09/28 19:00:17 INFO  : Starting bandwidth limiter at 9MBytes/s
2020/09/28 19:00:17 INFO  : Starting HTTP transaction limiter: max 2 transactions/s with burst 1
2020/09/28 19:00:17 DEBUG : Creating backend with remote "env1:test"
2020/09/28 19:00:17 DEBUG : Creating backend with remote "env2:test"
2020-09-28 19:06:34 ERROR : 20130728060014.test.57606696.tar: dangling large object with no contents
2020-09-28 19:32:25 ERROR : : error reading source directory: directory not found
2020-09-28 19:32:25 DEBUG : Swift container test Waiting for checks to finish
2020-09-28 19:32:25 DEBUG : Swift container test Waiting for transfers to finish
2020-09-28 19:32:25 ERROR : Swift container test not deleting files as there were IO errors
2020-09-28 19:32:25 ERROR : Swift container test not deleting directories as there were IO errors
2020-09-28 19:32:25 INFO  : There was nothing to transfer
2020-09-28 19:32:25 ERROR : Attempt 1/5 failed with 1 errors and: directory not found
2020-09-28 19:38:50 ERROR : 20130728060014.test.57606696.tar: dangling large object with no contents
2020-09-28 20:04:32 ERROR : 000: Failed to copy: Too Large Object
2020-09-28 20:04:33 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/1014": EOF)
2020-09-28 20:04:33 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020-09-28 20:04:33 DEBUG : 1014: Received error: Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/1014": EOF - low level retry 1/10
2020-09-28 20:04:35 DEBUG : pacer: Reducing sleep to 0s
2020-09-28 20:04:35 ERROR : 20090405030535.test.22786080.tar: Failed to copy: Too Large Object
2020-09-28 20:04:37 ERROR : 1014: Failed to copy: Too Large Object
2020-09-28 20:04:38 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405035643.test.22786082.tar": EOF)
2020-09-28 20:04:38 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020-09-28 20:04:38 DEBUG : 20090405035643.test.22786082.tar: Received error: Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405035643.test.22786082.tar": EOF - low level retry 1/10
2020-09-28 20:04:40 DEBUG : pacer: Reducing sleep to 0s
2020-09-28 20:04:40 ERROR : 20090405035643.test.22786082.tar.sum: Failed to copy: Too Large Object
2020-09-28 20:04:41 ERROR : 20090405035643.test.22786082.tar: Failed to copy: Bad Request
2020-09-28 20:04:41 ERROR : 20090405040116.test.22786084.tar: Failed to copy: Too Large Object
2020-09-28 20:04:45 ERROR : 20090405040549.test.22786086.tar: Failed to copy: Too Large Object
2020-09-28 20:04:47 ERROR : 20090405041023.test.22786088.tar: Failed to copy: Too Large Object
2020-09-28 20:09:37 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405030535.test.22786080.tar.sum": net/http: timeout awaiting response headers)
2020-09-28 20:09:37 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020-09-28 20:09:37 DEBUG : 20090405030535.test.22786080.tar.sum: Received error: Put "https://abc.com:8080/v1/AUTH_ABC12344555/test/20090405030535.test.22786080.tar.sum": net/http: timeout awaiting response headers - low level retry 1/10
2020-09-28 20:09:38 DEBUG : pacer: Reducing sleep to 0s```

that is only for google drive, so you can remove it.
tho should not solve the problem.

Yes I removed it still same problem.

what is the size of that file?

Greater than 5G and it is segmented

rclone -vvv -P sync env1:test env2:test --retries=5 --transfers=4 --filter "- *tar*" --fast-list. Currently, I am running the above command removed most of the arguments.. It is in initial stages.. let see if it works.

your original command, did not see filters?

Sorry I meant arguments.

these are confusing errors:
on the one hand, rclone cannot read the source directory.
on the second hand, rclone complains that a file being too large

if I do... rclone ls env1:test it does give me the list.

no worries, we have many experts. i am sure they will stop by soon.

This is the problem that needs fixing.

What rclone is saying there is that it read a multpart object manifest from swift which should have had contents within it, but it found no contents.

I think if you delete that one object then the sync should continue.

This happens when the segments container gets deleted.

I was able to delete the "tar" file.

I changed the command to following and when I ran it I did not see the "dangling large object with no contents".

/usr/bin/rclone -vvv -P copy --files-from 200610.txt env1:test env2:test --retries=1 --low-level-retries 1 --transfers=4 --timeout 600s --contimeout 10m --log-file 200610_rclone.log

Contents of 200610.txt. Only contains meta data

AFADFAD/test/20061031.quarterly.test.txt.gz
AFADFAD/test/20061031.transaction.quarterly.test.txt.gz
AFADFAD/test/20061031.transfer.quarterly.test.txt.gz
asdasdasd/AFADFAD/test/20061031.quarterly.test.txt.gz
asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz
asdasdasd/AFADFAD/test/20061031.transfer.quarterly.test.txt.gz
asdasdasd/asdasdasd/AFADFAD/test/20061031.quarterly.test.txt.gz
asdasdasd/asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz
asdasdasd/asdasdasd/AFADFAD/test/20061031.transfer.quarterly.test.txt.gz

The Object sizes are really low. They sizes are about 100KB. But still I see "TOO Large Object"

2020/09/29 15:53:46 DEBUG : Swift container test: Waiting for checks to finish
2020/09/29 15:53:46 DEBUG : Swift container test: Waiting for transfers to finish
2020/09/29 15:53:46 ERROR : asdasdasd/asdasdasd/AFADFAD/test/20061031.quarterly.test.txt.gz: Failed to copy: Too Large Object
2020/09/29 15:53:47 ERROR : asdasdasd/AFADFAD/test/20061031.quarterly.test.txt.gz: Failed to copy: Bad Request
2020/09/29 15:53:47 ERROR : asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz: Failed to copy: Bad Request
2020/09/29 15:53:47 ERROR : asdasdasd/asdasdasd/AFADFAD/test/20061031.transfer.quarterly.test.txt.gz: Failed to copy: Too Large Object
2020/09/29 16:03:47 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.xyz.com:8080/v1/AUTH_sdasdasdasdiuiuhadjsbasd/test/asdasdasd/AFADFAD/test/20061031.transfer.quarterly.test.txt.gz": net/http: timeout awaiting respons
e headers)
2020/09/29 16:03:47 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020/09/29 16:03:47 ERROR : asdasdasd/AFADFAD/test/20061031.transfer.quarterly.test.txt.gz: Failed to copy: Put "https://abc.xyz.com:8080/v1/AUTH_sdasdasdasdiuiuhadjsbasd/test/asdasdasd/AFADFAD/test/20061031.transfer.quarterly
.test.txt.gz": net/http: timeout awaiting response headers
2020/09/29 16:03:47 DEBUG : pacer: low level retry 1/1 (error Put "https://abc.xyz.com:8080/v1/AUTH_sdasdasdasdiuiuhadjsbasd/test/asdasdasd/asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz": net/http: timeout awaiting response headers)
2020/09/29 16:03:47 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/09/29 16:03:47 ERROR : asdasdasd/asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz: Failed to copy: Put "https://abc.xyz.com:8080/v1/AUTH_sdasdasdasdiuiuhadjsbasd/test/asdasdasd/asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz": net/http: timeout awaiting response headers
2020/09/29 16:03:47 INFO  : There was nothing to transfer
2020/09/29 16:03:47 ERROR : Attempt 1/1 failed with 9 errors and: Put "https://abc.xyz.com:8080/v1/AUTH_sdasdasdasdiuiuhadjsbasd/test/asdasdasd/asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz": net/http: timeout awaiting response headers
2020/09/29 16:03:47 INFO  : 
Transferred:      118.498k / 118.498 kBytes, 100%, 201 Bytes/s, ETA 0s
Errors:                 9 (retrying may help)
Elapsed time:      15m7.9s

2020/09/29 16:03:47 DEBUG : 2 go routines active
2020/09/29 16:03:47 Failed to copy with 9 errors: last error was: Put "https://abc.xyz.com:8080/v1/AUTH_sdasdasdasdiuiuhadjsbasd/test/asdasdasd/asdasdasd/AFADFAD/test/20061031.transaction.quarterly.test.txt.gz": net/http: timeout awaiting response headers

Good

That error "Too Large Object" is coming from the object storage I think... The remote end returned a 413 error.

If you run with -vv --dump headers that might give a clue.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.