And --metadata has no effect (despite the errors in my other thread. Not sure if it is the same problem or not.
The rclone config contents with secrets removed.
N/A -- local to locl
A log from the command with the -vv flag
2024/09/10 13:13:20 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "copy" "src" "dst" "-vv" "--copy-links"]
2024/09/10 13:13:20 DEBUG : Creating backend with remote "src"
2024/09/10 13:13:20 DEBUG : Using config file from "/<snip>/rclone.conf"
2024/09/10 13:13:20 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2024/09/10 13:13:20 DEBUG : fs cache: renaming cache item "src" to be canonical "local{12rtk}:/<snip>/src"
2024/09/10 13:13:20 DEBUG : Creating backend with remote "dst"
2024/09/10 13:13:20 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2024/09/10 13:13:20 DEBUG : fs cache: renaming cache item "dst" to be canonical "local{12rtk}:/<snip>/dst"
2024/09/10 13:13:20 DEBUG : file.txt: Need to transfer - File not found at Destination
2024/09/10 13:13:20 DEBUG : link.txt: Need to transfer - File not found at Destination
2024/09/10 13:13:20 DEBUG : Local file system at /<snip>/dst: Waiting for checks to finish
2024/09/10 13:13:20 DEBUG : Local file system at /<snip>/dst: Waiting for transfers to finish
2024/09/10 13:13:20 DEBUG : /<snip>/dst/link.txt.a28710be.partial: isCloned: true, error: <nil>
2024/09/10 13:13:20 DEBUG : link.txt.a28710be.partial: server-side cloned!
2024/09/10 13:13:20 DEBUG : /<snip>/dst/file.txt.a28710be.partial: isCloned: true, error: <nil>
2024/09/10 13:13:20 DEBUG : file.txt.a28710be.partial: server-side cloned!
2024/09/10 13:13:20 ERROR : link.txt: Failed to copy: object not found
2024/09/10 13:13:20 DEBUG : file.txt: md5 = d8e8fca2dc0f896fd7cb4cb0031ba249 OK
2024/09/10 13:13:20 DEBUG : file.txt.a28710be.partial: renamed to: file.txt
2024/09/10 13:13:20 INFO : file.txt: Copied (server-side copy)
2024/09/10 13:13:20 ERROR : Attempt 1/3 failed with 1 errors and: object not found
2024/09/10 13:13:20 DEBUG : link.txt: Need to transfer - File not found at Destination
2024/09/10 13:13:20 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/09/10 13:13:20 DEBUG : file.txt: Unchanged skipping
2024/09/10 13:13:20 DEBUG : Local file system at /<snip>/dst: Waiting for checks to finish
2024/09/10 13:13:20 DEBUG : Local file system at /<snip>/dst: Waiting for transfers to finish
2024/09/10 13:13:20 DEBUG : /<snip>/dst/link.txt.a28710be.partial: isCloned: false, error: couldn't copy from /<snip>/src/link.txt to /<snip>/dst/link.txt.a28710be.partial: errno -1
2024/09/10 13:13:20 ERROR : link.txt: Failed to copy: couldn't copy from /<snip>/src/link.txt to /<snip>/dst/link.txt.a28710be.partial: errno -1
2024/09/10 13:13:20 INFO : link.txt.a28710be.partial: Removing failed copy
2024/09/10 13:13:20 ERROR : Attempt 2/3 failed with 1 errors and: couldn't copy from /<snip>/src/link.txt to /<snip>/dst/link.txt.a28710be.partial: errno -1
2024/09/10 13:13:20 DEBUG : link.txt: Need to transfer - File not found at Destination
2024/09/10 13:13:20 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/09/10 13:13:20 DEBUG : file.txt: Unchanged skipping
2024/09/10 13:13:20 DEBUG : Local file system at /<snip>/dst: Waiting for checks to finish
2024/09/10 13:13:20 DEBUG : Local file system at /<snip>/dst: Waiting for transfers to finish
2024/09/10 13:13:20 DEBUG : /<snip>/dst/link.txt.a28710be.partial: isCloned: true, error: <nil>
2024/09/10 13:13:20 DEBUG : link.txt.a28710be.partial: server-side cloned!
2024/09/10 13:13:20 DEBUG : link.txt: md5 = d8e8fca2dc0f896fd7cb4cb0031ba249 OK
2024/09/10 13:13:20 DEBUG : link.txt.a28710be.partial: renamed to: link.txt
2024/09/10 13:13:20 INFO : link.txt: Copied (server-side copy)
2024/09/10 13:13:20 ERROR : Attempt 3/3 succeeded
2024/09/10 13:13:20 INFO :
Transferred: 10 B / 10 B, 100%, 0 B/s, ETA -
Checks: 2 / 2, 100%
Transferred: 2 / 2, 100%
Server Side Copies: 2 @ 10 B
Elapsed time: 0.0s
2024/09/10 13:13:20 DEBUG : 6 go routines active
Hmm, so far I am not able to reproduce this... but I am still on 14.5. I can try updating (been meaning to anyway) and see if that makes any difference.
Is your link.txt just a normal symlink like created with ln -s? Anything special about it?
Thank you both! I think I have figured out how to reliably reproduce it now. I think there are actually two different bugs:
--copy-links is erroneously behaving like --links, unless --local-no-clone is used.
In other words, the dst ends up with a link instead of a regular file. (Can you confirm whether you are seeing this too?)
Creating a link to a relative path in the dst fails if the file it's pointing to doesn't exist yet.
I think I wasn't seeing this at first because I tried with an absolute path instead of relative.
I was also confused at first because whether a link is created before its target is subject to the whims of concurrency and ordering. So, in the above example where file comes before link alphabetically, I found that to make it deterministically fail I had to add:
--check-first --order-by name,desc --transfers 1
and to make it deterministically succeed I had to change desc to asc.
This would also explain why it fails but then succeeds on retry (which also confused me.)
Now that I think I know why it's happening I think I should be able to fix it. One thing I guess I'm not sure about is how we normally handle the scenario where we want to create a relative link in the dest to another dest file that doesn't exist yet -- I guess we probably just don't check whether it exists, and create it regardless?
Thanks for all of the hard work! I am glad you were able to reproduce it. Honestly, I am still not 100% this is the bug I am hitting in my 3rd party tool but it's a bug either way.
Would you like me to put it on github?
I am not sure what the current practice is, but I would say that by Principle of Least Surprise, it should copy whether or not it's a broken link. Same thing with --links and .rclonelink files. It should happen regardless
Is there a link to an executable or a build? I didn't see it on https://beta.rclone.org/. (and I don't have Go installed on this computer).
If not, I will test later this week when I can build it at home. Either way, thanks for looking at this. I am hoping it also addresses the issue in my 3rd party tool.
I also made a couple of improvements over the version @kapitainsky tried earlier, as I realized that in that version, while the bug was fixed, it was only copying rather than cloning. This version should fix that.
I hope so too! Let me know if you have a chance to try it out.
(If you got email notifications that it didn't work, I have since deleted those replies. I had messed up hard-coding the path to the beta in my test code)