--copy-links not following symlinks on 1.60.0

cberni · November 14, 2022, 1:50pm

What is the problem you are having with rclone?

--copy-links is not following symlinks on 1.60.0 using rclone sync

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.0

os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.4.0-131-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.19.2
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

google cloud storage

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /test remote:bucket/test --copy-links --fast-list

The rclone config contents with secrets removed.

[remote]
type = google cloud storage
service_account_file =

A log from the command with the `-vv` flag

normal, just ignore the symlink and not follow anymore. (version 1.60.0)
just revert to 1.59.2 and it start to follow symlinks again with --copy-links

Animosity022 · November 14, 2022, 1:58pm

It's super helpful to include an actual debug log with the output.

felix@gemini:~$ rclone copy /home/felix/test /home/felix/test2 -vv --copy-links
2022/11/14 08:57:14 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/14 08:57:14 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "copy" "/home/felix/test" "/home/felix/test2" "-vv" "--copy-links"]
2022/11/14 08:57:14 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/14 08:57:14 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/14 08:57:14 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 08:57:14 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
2022/11/14 08:57:14 DEBUG : Creating backend with remote "/home/felix/test2"
2022/11/14 08:57:14 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 08:57:14 DEBUG : fs cache: renaming cache item "/home/felix/test2" to be canonical "local{12rtk}:/home/felix/test2"
2022/11/14 08:57:14 DEBUG : Local file system at /home/felix/test2: Waiting for checks to finish
2022/11/14 08:57:14 DEBUG : hosts: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/11/14 08:57:14 DEBUG : hosts: Unchanged skipping
2022/11/14 08:57:14 DEBUG : Local file system at /home/felix/test2: Waiting for transfers to finish
2022/11/14 08:57:14 DEBUG : blah: md5 = de97f73ff3488b742fe04ed0af1dc50d OK
2022/11/14 08:57:14 INFO  : blah: Copied (new)
2022/11/14 08:57:14 INFO  :
Transferred:   	        511 B / 511 B, 100%, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2022/11/14 08:57:14 DEBUG : 5 go routines active
felix@gemini:~$ cd test
felix@gemini:~/test$ ls
blah  hosts
felix@gemini:~/test$ ls -al
total 8
drwxrwxr-x 1 felix felix  18 Nov 14 08:56 .
drwxr-x--- 1 felix felix 266 Nov 14 08:57 ..
lrwxrwxrwx 1 felix felix  10 Nov 14 08:56 blah -> /etc/hosts
-rw-r--r-- 1 felix felix 511 Nov 14 08:56 hosts
felix@gemini:~/test$ cd ..

cberni · November 14, 2022, 3:41pm

ok. the problem is using --include-from.....

--include-from=/etc//rclone-include.txt

/test/test-link/**

command

rclone sync / remote:BUCKET/test --copy-links --fast-list --include-from=/etc//rclone-include.txt --config=/etc//rclone.conf -vv

the vv log...



2022/11/14 12:35:29 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "sync" "/" "remote:BUCKET/test" "--copy-links" "--fast-list" "--include-from=/etc/rclone-include.txt" "--config=/etc/rclone.conf" "-vv"]
2022/11/14 12:35:29 DEBUG : Creating backend with remote "/"
2022/11/14 12:35:29 DEBUG : Using config file from "/etc/rclone.conf"
2022/11/14 12:35:29 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 12:35:29 DEBUG : fs cache: renaming cache item "/" to be canonical "local{12rtk}:/"
2022/11/14 12:35:29 DEBUG : Creating backend with remote "remote:BUCKET/test"
2022/11/14 12:35:29 DEBUG : root: Excluded
2022/11/14 12:35:29 DEBUG : mnt: Excluded
2022/11/14 12:35:29 DEBUG : test-link: Excluded
2022/11/14 12:35:29 DEBUG : srv: Excluded
2022/11/14 12:35:29 DEBUG : opt: Excluded
2022/11/14 12:35:29 DEBUG : media: Excluded
2022/11/14 12:35:29 DEBUG : usr: Excluded
2022/11/14 12:35:29 DEBUG : restic-backup: Excluded
2022/11/14 12:35:29 DEBUG : tmp: Excluded
2022/11/14 12:35:29 DEBUG : union: Excluded
2022/11/14 12:35:29 DEBUG : snap: Excluded
2022/11/14 12:35:29 DEBUG : etc: Excluded
2022/11/14 12:35:29 DEBUG : sys: Excluded
2022/11/14 12:35:29 DEBUG : var: Excluded
2022/11/14 12:35:29 DEBUG : cdrom: Excluded
2022/11/14 12:35:29 DEBUG : local: Excluded
2022/11/14 12:35:29 DEBUG : boot: Excluded
2022/11/14 12:35:29 DEBUG : home: Excluded
2022/11/14 12:35:29 DEBUG : proc: Excluded
2022/11/14 12:35:29 DEBUG : dev: Excluded
2022/11/14 12:35:29 DEBUG : run: Excluded
2022/11/14 12:35:29 DEBUG : lost+found: Excluded
2022/11/14 12:35:30 DEBUG : GCS bucket BUCKET path test: Waiting for checks to finish
2022/11/14 12:35:30 DEBUG : GCS bucket BUCKET path test: Waiting for transfers to finish
2022/11/14 12:35:30 DEBUG : Waiting for deletions to finish
2022/11/14 12:35:30 INFO  : There was nothing to transfer
2022/11/14 12:35:30 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.6s

2022/11/14 12:35:30 DEBUG : 5 go routines active

cberni · November 14, 2022, 3:43pm

Animosity022 · November 14, 2022, 4:14pm

Not sure I'm positive yet the bottom scenario should work or not as I have to think about it a bit more.

I recreated something similar.

felix@gemini:~/test$ rclone ls /home/felix/test -vv -L --include 'test2**'
2022/11/14 11:12:07 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/14 11:12:07 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "ls" "/home/felix/test" "-vv" "-L" "--include" "test2**"]
2022/11/14 11:12:07 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/14 11:12:07 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/14 11:12:07 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 11:12:07 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
2022/11/14 11:12:07 DEBUG : test3: Excluded
        0 test2/test2file
2022/11/14 11:12:07 DEBUG : 2 go routines active
felix@gemini:~/test$ ls -al
total 12
drwxrwxr-x 1 felix felix  38 Nov 14 10:58 .
drwxr-x--- 1 felix felix 266 Nov 14 08:57 ..
lrwxrwxrwx 1 felix felix  10 Nov 14 08:56 blah -> /etc/hosts
-rw-r--r-- 1 felix felix 511 Nov 14 08:56 hosts
lrwxrwxrwx 1 felix felix  17 Nov 14 10:58 test2 -> /home/felix/test2
drwxrwxr-x 1 felix felix  18 Nov 14 10:54 test3
felix@gemini:~/test$ rclone ls /home/felix/test -vv -L --include 'test2/**'
2022/11/14 11:12:58 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/14 11:12:58 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "ls" "/home/felix/test" "-vv" "-L" "--include" "test2/**"]
2022/11/14 11:12:58 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/14 11:12:58 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/14 11:12:58 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 11:12:58 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
2022/11/14 11:12:58 DEBUG : test3: Excluded
2022/11/14 11:12:58 DEBUG : 2 go routines active

The top works but the bottom doesn't as test3 does as that's a real directory.

felix@gemini:~/test$ rclone ls /home/felix/test -vv -L --include 'test3/**'
2022/11/14 11:14:46 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/14 11:14:46 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "ls" "/home/felix/test" "-vv" "-L" "--include" "test3/**"]
2022/11/14 11:14:46 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/14 11:14:46 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/14 11:14:46 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 11:14:46 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
        0 test3/test3file
2022/11/14 11:14:46 DEBUG : 2 go routines active

cberni · November 14, 2022, 4:26pm

the symlink: /test/test-link

--include="/test/**" works ok
--include="/test/test-link/**" does not work.

on version 1.59.2 both works fine.

Animosity022 · November 14, 2022, 4:31pm

I'm seeing the same in 1.60 vs 1.59.2:

felix@gemini:~/test$ rclone ls /home/felix/test -vv -L --include 'test2/**'
2022/11/14 11:30:39 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/14 11:30:39 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "ls" "/home/felix/test" "-vv" "-L" "--include" "test2/**"]
2022/11/14 11:30:39 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/14 11:30:39 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/14 11:30:39 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 11:30:39 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
2022/11/14 11:30:39 DEBUG : test3: Excluded
2022/11/14 11:30:39 DEBUG : 2 go routines active
felix@gemini:~/test$ ./rclone ls /home/felix/test -vv -L --include 'test2/**'
2022/11/14 11:30:47 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/14 11:30:47 DEBUG : rclone: Version "v1.59.2" starting with parameters ["./rclone" "ls" "/home/felix/test" "-vv" "-L" "--include" "test2/**"]
2022/11/14 11:30:47 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/14 11:30:47 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 11:30:47 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/14 11:30:47 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
2022/11/14 11:30:47 DEBUG : blah: Excluded
2022/11/14 11:30:47 DEBUG : hosts: Excluded
2022/11/14 11:30:47 DEBUG : test3: Excluded
2022/11/14 11:30:47 DEBUG : rclone: Excluded
        0 test2/test2file
2022/11/14 11:30:47 DEBUG : 2 go routines active

Animosity022 · November 14, 2022, 4:32pm

There was a few filter fixes in 1.60 so @ncw should be able to take a peek as we narrowed down the issue very specifically.

ncw · November 14, 2022, 6:05pm

It looks like that this is the first bug caused by pushing the filters down into the local backend.

A bit of git bisect confirms that

commit 8d1fff9a8237c64f73cc02e273921efde1553487
Author: Nick Craig-Wood <nick@craig-wood.com>
Date:   Thu Aug 11 12:20:34 2022 +0100

    local: obey file filters in listing to fix errors on excluded files
    
    Fixes #6376

 backend/local/local.go               | 15 +++++++++++++++
 backend/local/local_internal_test.go | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+)

This looks like an easy fix though, we just need to move the filter test to after we've resolved the symlink to be a file or a directory.

Please give this a go

v1.61.0-beta.6565.a61d219bc.fix-local-filter on branch fix-local-filter (uploaded in 15-30 mins)

Animosity022 · November 14, 2022, 6:28pm

Looks good to me as it fixed my test case.

./rclone ls /home/felix/test -vv -L --include 'test2/**'
2022/11/14 13:28:11 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/14 13:28:11 DEBUG : rclone: Version "v1.61.0-beta.6565.a61d219bc.fix-local-filter" starting with parameters ["./rclone" "ls" "/home/felix/test" "-vv" "-L" "--include" "test2/**"]
2022/11/14 13:28:11 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/14 13:28:11 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/14 13:28:11 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/11/14 13:28:11 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
2022/11/14 13:28:11 DEBUG : test3: Excluded
        0 test2/test2file
2022/11/14 13:28:11 DEBUG : 2 go routines active

@cberni - give it a go and see how if it works for you as well.

cberni · November 15, 2022, 1:54am

yes. woking fine with...

thanks a lot @Animosity022 and @ncw

ncw · November 15, 2022, 9:05am

Thank you both for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.61 and v1.60.1

system · November 18, 2022, 9:06am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.