I've been using some automation scripting for migrating data between two swift clusters. Until today, my rclone copy commands have worked fine, but now it apprears that rclone is requiring that the destination server needs a copy of the file being copied or it will fail outright.
What is your rclone version (output from rclone version)
rclone v1.52.2-276-gc2f3949d-beta
os/arch: linux/amd64
go version: go1.14.6
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Cent 7.7
Which cloud storage system are you using? (eg Google Drive)
Swift
The command you were trying to run (eg rclone copy /tmp remote:tmp)
# rclone copy "newuat":"1-content-1" "newqatuat":"1-content-1" --no-check-certificate --no-check-dest --use-mmap --buffer-size 299M --files-from "1-content-1~~~aa" --stats 1m --stats-one-line-date --size-only --checkers=1 --transfers=20 --cache-tmp-upload-path /mnt/shared/rclone_tmp --no-traverse -P
2020-07-31 00:47:07 ERROR : 0000fsqf1.content: Failed to copy: Object Not Found
2020-07-31 00:47:07 ERROR : Attempt 1/3 failed with 1 errors and: Object Not Found
2020-07-31 00:47:07 ERROR : 0000fsqf1.content: Failed to copy: Object Not Found
2020-07-31 00:47:07 ERROR : Attempt 2/3 failed with 1 errors and: Object Not Found
2020-07-31 00:47:08 ERROR : 0000fsqf1.content: Failed to copy: Object Not Found
2020-07-31 00:47:08 ERROR : Attempt 3/3 failed with 1 errors and: Object Not Found
2020/07/31 00:47:08 - 57 / 57 Bytes, 100%, 41 Bytes/s, ETA 0s
2020/07/31 00:47:08 Failed to copy: Object Not Found
The load balancer for the destination server newqatuat is showing the following 404s in logs:
[newuat]
type = swift
env_auth = false
user = uat
key = REDACTED
auth = http://newprodswiftlb-vip.localatlas.com/auth/v1.0
storage_url = http://newprodswiftlb-vip.localatlas.com/v1/AUTH_uat
endpoint_type = internal
[newqatuat]
type = swift
env_auth = false
user = uat
key = REDACTED
auth = http://newqatswiftlb-vip.localatlas.com/auth/v1.0
storage_url = http://newqatswiftlb-vip.localatlas.com/v1/AUTH_uat
endpoint_type = internal
A log from the command with the -vv flag
# rclone copy -vv "newuat":"1-content-1" "newqatuat":"1-content-1" --no-check-certificate --no-check-dest --use-mmap --buffer-size 299M --files-from "1-content-1~~~aa" --stats 1m --stats-one-line-date --size-only --checkers=1 --transfers=20 --cache-tmp-upload-path /mnt/shared/rclone_tmp --no-traverse -P
2020/07/31 00:51:42 DEBUG : rclone: Version "v1.52.2-276-gc2f3949d-beta" starting with parameters ["rclone" "copy" "-vv" "newuat:1-content-1" "newqatuat:1-content-1" "--no-check-certificate" "--no-check-dest" "--use-mmap" "--buffer-size" "299M" "--files-from" "1-content-1~~~aa" "--stats" "1m" "--stats-one-line-date" "--size-only" "--checkers=1" "--transfers=20" "--cache-tmp-upload-path" "/mnt/shared/rclone_tmp" "--no-traverse" "-P"]
2020/07/31 00:51:42 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020-07-31 00:51:42 DEBUG : Swift container 1-content-1: Waiting for checks to finish
2020-07-31 00:51:42 DEBUG : Swift container 1-content-1: Waiting for transfers to finish
2020-07-31 00:51:43 ERROR : 0000fsqf1.content: Failed to copy: Object Not Found
2020-07-31 00:51:43 INFO : There was nothing to transfer
2020-07-31 00:51:43 ERROR : Attempt 1/3 failed with 1 errors and: Object Not Found
2020-07-31 00:51:43 DEBUG : Swift container 1-content-1: Waiting for checks to finish
2020-07-31 00:51:43 DEBUG : Swift container 1-content-1: Waiting for transfers to finish
2020-07-31 00:51:43 ERROR : 0000fsqf1.content: Failed to copy: Object Not Found
2020-07-31 00:51:43 INFO : There was nothing to transfer
2020-07-31 00:51:43 ERROR : Attempt 2/3 failed with 1 errors and: Object Not Found
2020-07-31 00:51:43 DEBUG : Swift container 1-content-1: Waiting for checks to finish
2020-07-31 00:51:43 DEBUG : Swift container 1-content-1: Waiting for transfers to finish
2020-07-31 00:51:43 ERROR : 0000fsqf1.content: Failed to copy: Object Not Found
2020-07-31 00:51:43 INFO : There was nothing to transfer
2020-07-31 00:51:43 ERROR : Attempt 3/3 failed with 1 errors and: Object Not Found
2020/07/31 00:51:43 - 57 / 57 Bytes, 100%, 41 Bytes/s, ETA 0s
2020/07/31 00:51:43 INFO : 2020/07/31 00:51:43 - 57 / 57 Bytes, 100%, 41 Bytes/s, ETA 0s
2020/07/31 00:51:43 DEBUG : 10 go routines active
2020/07/31 00:51:43 Failed to copy: Object Not Found
Rclone is also failing if the destination container doesnt exist:
# rclone copy qat:10514-pdf newqat:somerando-10514-pdf --no-check-certificate --no-check-dest --use-mmap --buffer-size 299M --stats 1m --stats-one-line-date --size-only --checkers=1 --transfers=20 --cache-tmp-upload-path /mnt/shared/rclone_tmp -P
2020-07-31 01:04:48 ERROR : 0000j43in.pdf: Failed to copy: Container Not Found
2020-07-31 01:04:49 ERROR : 0000j43ip.pdf: Failed to copy: Container Not Found
2020-07-31 01:04:50 ERROR : 0000j43jo.pdf: Failed to copy: Container Not Found
2020-07-31 01:04:50 ERROR : 0000j43iq.pdf: Failed to copy: Container Not Found
2020-07-31 01:04:51 ERROR : 0000j43io.pdf: Failed to copy: Container Not Found
2020-07-31 01:04:51 ERROR : 0000j43jw.pdf: Failed to copy: Container Not Found
2020-07-31 01:04:52 ERROR : 0000j43js.pdf: Failed to copy: Container Not Found
2020-07-31 01:04:52 ERROR : 0000j43jq.pdf: Failed to copy: Container Not Found
The issue is not on the source side. Rclone is failing when trying to copy to the destination. If you look at the loadbalancer logs I provided, the rclone does a head check on the destination for the object or container, if it returns 404 on the destination rclone fails outright. I'm pretty sure this isn't the intended behavior from a migration perspective.
[root@cxvlhgdj022]~# rclone copy qat:10514-pdf newqatuat:10514-pdf
2020/07/31 15:52:15 ERROR : 0000j43in.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:15 ERROR : 0000j43ip.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:16 ERROR : 0000j43io.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:16 ERROR : 0000j43iq.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:17 ERROR : 0000j43jh.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:18 ERROR : 0000j43ji.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:18 ERROR : 0000j43jj.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:19 ERROR : 0000j43jk.pdf: Failed to copy: Container Not Found
2020/07/31 15:52:20 ERROR : 0000j43jl.pdf: Failed to copy: Container Not Found
It is specifically failing because the container isn't present on the destination, not the source.
[root@cxvlhgdj022]~# rclone copy uat_uat:13770-native-j newqatuat:13770-native-j -P
2020-07-31 15:55:53 ERROR : 0000jtpbj.pdf: Failed to copy: Container Not Found
2020-07-31 15:55:54 ERROR : 0000kcmoj.docx: Failed to copy: Container Not Found
2020-07-31 15:55:54 ERROR : 0000kckhj.sxterm: Failed to copy: Container Not Found
2020-07-31 15:55:54 ERROR : Attempt 1/3 failed with 3 errors and: Container Not Found
2020-07-31 15:55:55 ERROR : 0000kckhj.sxterm: Failed to copy: Container Not Found
2020-07-31 15:55:56 ERROR : 0000jtpbj.pdf: Failed to copy: Container Not Found
2020-07-31 15:55:56 ERROR : 0000kcmoj.docx: Failed to copy: Container Not Found
2020-07-31 15:55:56 ERROR : Attempt 2/3 failed with 3 errors and: Container Not Found
2020-07-31 15:55:57 ERROR : 0000jtpbj.pdf: Failed to copy: Container Not Found
2020-07-31 15:55:58 ERROR : 0000kckhj.sxterm: Failed to copy: Container Not Found
Transferred: 0 / 22.842 kBytes, 0%, 0 Bytes/s, ETA -
Errors: 2 (retrying may help)
Transferred: 0 / 1, 0%
Elapsed time: 6.2s
Transferring:
* 0000kcmoj.docx: 0% /22.842k, 0/s, -^C^
yea appears to only be an issue using this endpoint as dest:
[root@cxvlhgdj022]~# rclone copy newqat:20059-content-2 qat:20059-content-2-test -P
Transferred: 14.994k / 14.994 kBytes, 100%, 52.985 kBytes/s, ETA 0s
Transferred: 2 / 2, 100%
Elapsed time: 0.7s
[root@cxvlhgdj022]~# rclone copy qat:20059-content-2-test newqat:20059-content-2-test-back -P
2020-07-31 16:26:14 ERROR : 0000lhp92.content: Failed to copy: Container Not Found
2020-07-31 16:26:15 ERROR : 0000lj9j2.content: Failed to copy: Container Not Found
2020-07-31 16:26:15 ERROR : Attempt 1/3 failed with 2 errors and: Container Not Found
2020-07-31 16:26:16 ERROR : 0000lhp92.content: Failed to copy: Container Not Found
2020-07-31 16:26:17 ERROR : 0000lj9j2.content: Failed to copy: Container Not Found
2020-07-31 16:26:17 ERROR : Attempt 2/3 failed with 2 errors and: Container Not Found
2020-07-31 16:26:18 ERROR : 0000lhp92.content: Failed to copy: Container Not Found
2020-07-31 16:26:18 ERROR : 0000lj9j2.content: Failed to copy: Container Not Found
2020-07-31 16:26:18 ERROR : Attempt 3/3 failed with 2 errors and: Container Not Found
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 2 (retrying may help)
Elapsed time: 4.9s
2020/07/31 16:26:18 Failed to copy with 2 errors: last error was: Container Not Found
[root@cxvlhgdj022]~# rclone ls qat:20059-content-2-test
14771 0000lhp92.content
583 0000lj9j2.content
any ideas / suggestions why rclone would only be working unidirectionally here?
Rclone does a HEAD request on an object to see if it exists before copying it - maybe it is already there and doesn't need copying.
Can you do the failing test with -vv --dump bodies --retries 1 --low-level-retries 1 and post the result? You might need to redact some sensitive info.