Rclone --links gets tripped up by existing .rclonelink files

Suspected Bug
1

What is the problem you are having with rclone?

(This is not actually causing me any harm but it is worth noting. And it is extremely esoteric)

rclone can be tricked into not recreating a link when using --links by the existing files.

See the full steps below but the expect behavior is the the .rclonelink file gets overwritten by a link

What is your rclone version (output from rclone version)

rclone v1.53.0
- os/arch: darwin/amd64
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS. I could test on linux if needed. I have no easy access to windows.

Which cloud storage system are you using? (eg Google Drive)

crypt and local but I suspect any non-local + local would exhibit it.

I use crypt to mimic a normal non-local remote.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Step 0:

Set up a tree like this:

.
├── A
│   ├── file1.txt
│   └── file2.txt -> file1.txt
└── cfg

Step 1: Sync from A to cB: with --links (see config below)

$ rclone --config cfg -vv --links sync A cB:

Not including the output as this does what you expect. The file tree:

├── A
│   ├── file1.txt
│   └── file2.txt -> file1.txt
├── B
│   ├── file1.txt.bin
│   └── file2.txt.rclonelink.bin
└── cfg

Step 2: Sync from cB: to C withOUT --links

$ rclone --config cfg -vv sync cB: C

.
├── A
│   ├── file1.txt
│   └── file2.txt -> file1.txt
├── B
│   ├── file1.txt.bin
│   └── file2.txt.rclonelink.bin
├── C
│   ├── file1.txt
│   └── file2.txt.rclonelink
└── cfg

Again, this is what you'd expect (no need for output)

Step 3: Sync again with --links

$ rclone --config cfg -vv --links sync cB: C

output:

2020/09/07 14:35:19 DEBUG : rclone: Version "v1.53.0" starting with parameters ["rclone" "--config" "cfg" "-vv" "--links" "sync" "cB:" "C"]
2020/09/07 14:35:19 DEBUG : Using config file from "/Users/<USERNAME>/Desktop/tmp/cfg"
2020/09/07 14:35:19 DEBUG : Creating backend with remote "cB:"
2020/09/07 14:35:20 DEBUG : Creating backend with remote "B"
2020/09/07 14:35:20 DEBUG : fs cache: renaming cache item "B" to be canonical "/Users/<USERNAME>/Desktop/tmp/B"
2020/09/07 14:35:20 DEBUG : Creating backend with remote "C"
2020/09/07 14:35:20 DEBUG : fs cache: renaming cache item "C" to be canonical "/Users/<USERNAME>/Desktop/tmp/C"
2020/09/07 14:35:20 DEBUG : .DS_Store: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/09/07 14:35:20 DEBUG : .DS_Store: Unchanged skipping
2020/09/07 14:35:20 DEBUG : file1.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/09/07 14:35:20 DEBUG : file1.txt: Unchanged skipping
2020/09/07 14:35:20 DEBUG : file2.txt.rclonelink: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/09/07 14:35:20 DEBUG : file2.txt.rclonelink: Unchanged skipping
2020/09/07 14:35:20 DEBUG : Local file system at /Users/<USERNAME>/Desktop/tmp/C: Waiting for checks to finish
2020/09/07 14:35:20 DEBUG : Local file system at /Users/<USERNAME>/Desktop/tmp/C: Waiting for transfers to finish
2020/09/07 14:35:20 DEBUG : Waiting for deletions to finish
2020/09/07 14:35:20 INFO  : There was nothing to transfer
2020/09/07 14:35:20 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 3 / 3, 100%
Elapsed time:         0.0s

And

.
├── A
│   ├── file1.txt
│   └── file2.txt -> file1.txt
├── B
│   ├── file1.txt.bin
│   └── file2.txt.rclonelink.bin
├── C
│   ├── file1.txt
│   └── file2.txt.rclonelink
└── cfg

This is the bug: You'd expect to see file2.txt -> file1.txt and file2.txt.rclonelink removed.

Extra Comparison: Remove C and do it again

$ rm -r C
$ rclone --config cfg -vv --links sync cB: C

With:

2020/09/07 14:37:44 DEBUG : rclone: Version "v1.53.0" starting with parameters ["rclone" "--config" "cfg" "-vv" "--links" "sync" "cB:" "C"]
2020/09/07 14:37:44 DEBUG : Using config file from "/Users/<USERNAME>/Desktop/tmp/cfg"
2020/09/07 14:37:44 DEBUG : Creating backend with remote "cB:"
2020/09/07 14:37:44 DEBUG : Creating backend with remote "B"
2020/09/07 14:37:44 DEBUG : fs cache: renaming cache item "B" to be canonical "/Users/<USERNAME>/Desktop/tmp/B"
2020/09/07 14:37:44 DEBUG : Creating backend with remote "C"
2020/09/07 14:37:44 DEBUG : fs cache: renaming cache item "C" to be canonical "/Users/<USERNAME>/Desktop/tmp/C"
2020/09/07 14:37:44 DEBUG : Local file system at /Users/<USERNAME>/Desktop/tmp/C: Waiting for checks to finish
2020/09/07 14:37:44 DEBUG : Local file system at /Users/<USERNAME>/Desktop/tmp/C: Waiting for transfers to finish
2020/09/07 14:37:45 INFO  : file2.txt.rclonelink: Copied (new)
2020/09/07 14:37:45 INFO  : .DS_Store: Copied (new)
2020/09/07 14:37:45 INFO  : file1.txt: Copied (new)
2020/09/07 14:37:45 DEBUG : Waiting for deletions to finish
2020/09/07 14:37:45 INFO  :
Transferred:   	   44.170k / 44.170 kBytes, 100%, 72.629 MBytes/s, ETA 0s
Transferred:            3 / 3, 100%
Elapsed time:         0.0s

and

.
├── A
│   ├── file1.txt
│   └── file2.txt -> file1.txt
├── B
│   ├── file1.txt.bin
│   └── file2.txt.rclonelink.bin
├── C
│   ├── file1.txt
│   └── file2.txt -> file1.txt
└── cfg

The rclone config contents with secrets removed.

[cB]
type = crypt
remote = B
filename_encryption = off
directory_name_encryption = false
password = F20ZSHcZcYercN4Tf8iFmB7eWrz21yeZhQ
password2 = 8Vyto3fD96u0tlLozJUCx3vw4WtqQDpCPw

A log from the command with the -vv flag

See it inline

Summary

I absolutely think this is a bug, Evidenced by the fact that the result of a sync should be the same regardless of existing files and you can see from "Step 3" and "Extra Comparison" it is not the case.

However, I would also classify this as very low priority and probably not even one likely to be worth fixing. Though it may be worth noting the behavior as it can matter if someone does a sync without --links, realizes the error, and then tries to do it again.

2

I think this is because rclone is refusing to change the real file into a link.

I think this issue is related

3

Interesting. And this leads me to two question:

First: What if you reverse it:

Do a --links sync and then without. What I would expect is to get the link created and then the link deleted (it is a sync) and the .rclonelink file there.

What I end up with is:

.
├── A
│   ├── file1.txt
│   └── file2.txt -> file1.txt
├── B
│   ├── file1.txt.bin
│   └── file2.txt.rclonelink.bin
├── C
│   ├── file1.txt
│   ├── file2.txt -> file1.txt
│   └── file2.txt.rclonelink
└── cfg

This again is surprising...I think. You could argue that without any kind of link setting, you'd expect rclone to ignore links. Or you could argue that sync is supposed to mirror the source to the dest and they do not match.

To test if this is universal or just with file2 in particular, I made:

$ ln -s BROKEN C/broken.txt

So now it is a broken link there. I did the sync again and the broken link is still there.

So my question (of part 1) is whether it is expected behavior that errant links at the destination are not deleted? I would have expected that they would be.

First part II:

Still thinking on these lines, will rclone delete extra .rclonelink files?

$ rclone --config cfg -vv --links sync C cB:

The broken.txt.rclonelink is there

$ rm C/broken.txt
$ rclone --config cfg -vv --links sync C cB:

Now it is gone.

This feels inconsistent.

Question 2:

You say that rclone will not change a file to a link. Is this a bug though? Or a feature?

I'd argue a bug as it goes against principle of least surprise as it is inconsistent. But I could see the alternative argument

4

I think that is probably what happened. Rclone ignored the link in C without the --links parameter.

You should have seen a log about this.

I'd call it a corner case :slight_smile:

I did this beta a while ago - it might be worth you having a go with it. This will let a file overwrite a symlink.

v1.54.0-beta.4756.4d97a6d2c.fix-3400-symlink on branch fix-3400-symlink (uploaded in 15-30 mins)

5

Thanks, I'll take a look. As I noted, this is more from me exploring rclone and the different behaviors than an actual need at this point. I thought I would at least document my findings.

It may be worth the discussion to decide how to handle this or, since it is likely low priority, let this thread be the documentation and leave it at that!

6

If I can't decide how rclone should work then I copy what rsync does...

1 Like
7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.