--links and filtering

What is the problem you are having with rclone?

Using rclone sync --links together with filtering (e.g. --include or --files-from) fails to create .rclonelink files at the destination. How can I make this work? Or maybe this is a bug?

Run the command 'rclone version' and share the full output of the command.

rclone 1.61.1
- os/version: darwin 13.3.1 (64 bit)
- os/kernel: 22.4.0 (arm64)
- os/type: darwin
- os/arch: arm64
- go/version: go1.19.5
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

pCloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --log-level INFO --files-from-raw files.txt /Users/brechtm/Documents/Routes crypt:Backup/Test/latest

The rclone config contents with secrets removed.

[pcloud]
type = pcloud
hostname = eapi.pcloud.com
token = ...

[crypt]
type = crypt
remote = pcloud:.crypt
filename_encryption = standard
directory_name_encryption = true
password = ...
password2 = ...

A log from the command with the -vv flag

Documents/Routes crypt:Backup/Test/latest
2023/05/27 14:53:51 DEBUG : rclone: Version "1.61.1" starting with parameters ["rclone" "sync" "-vv" "--links" "--files-from-raw" "files.txt" "/Users/brechtm/Documents/Routes" "crypt:Backup/Test/latest"]
2023/05/27 14:53:51 DEBUG : Creating backend with remote "/Users/brechtm/Documents/Routes"
2023/05/27 14:53:51 DEBUG : Using config file from "/Users/brechtm/.config/rclone/rclone.conf"
2023/05/27 14:53:51 DEBUG : local: detected overridden config - adding "{XXX}" suffix to name
2023/05/27 14:53:51 DEBUG : fs cache: renaming cache item "/Users/brechtm/Documents/Routes" to be canonical "local{XXX}:/Users/brechtm/Documents/Routes"
2023/05/27 14:53:51 DEBUG : Creating backend with remote "crypt:Backup/Test/latest"
2023/05/27 14:53:51 DEBUG : Creating backend with remote "pcloud:.crypt/XXXXX"
2023/05/27 14:53:52 DEBUG : Directions: Excluded
2023/05/27 14:53:52 DEBUG : Recorded: Excluded
2023/05/27 14:53:52 DEBUG : brechtm.exclude.rclonelink: Excluded
2023/05/27 14:53:52 DEBUG : rclone-develop.rclonelink: Excluded
... (more excluded)
2023/05/27 14:53:52 DEBUG : test.gpx: Excluded
2023/05/27 14:53:52 DEBUG : Encrypted drive 'crypt:Backup/Test/latest': Waiting for checks to finish
2023/05/27 14:53:52 DEBUG : Encrypted drive 'crypt:Backup/Test/latest': Waiting for transfers to finish
2023/05/27 14:53:52 DEBUG : Waiting for deletions to finish
2023/05/27 14:53:52 INFO  : There was nothing to transfer
2023/05/27 14:53:52 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.7s

files.txt:

rclone-develop
brechtm.exclude

When I add the .rclonelink extension to these filenames in files.txt, the files aren't even listed in the debug log.

EDIT: An interesting observation is that for --include and --include-from, adding brechtm.exclude* (note the asterisk) will create the .rclonelink file on the remote. This doesn't work for --files-from-raw.

I'm more and more convinced that this is a bug. I'm unable to change this thread's category to Suspected Bug, however.

best to update to latest stable, test again.
rclone selfupdate

I can reproduce this with

rclone 1.62.2
- os/version: darwin 13.3.1 (64 bit)
- os/kernel: 22.4.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.4
- go/linking: dynamic
- go/tags: cmount

Your example code doesn't have --links in, so I'm not really sure what you are trying to achieve.

Are either of these symlinks?

You need to use the name rclone uses for a file.

$ echo normal > file.txt
$ ln -s file.txt link.txt
$ ls -l
total 4
-rw-rw-r-- 1 ncw ncw 7 Jun  2 16:08 file.txt
lrwxrwxrwx 1 ncw ncw 8 Jun  2 16:08 link.txt -> file.txt

$ rclone lsf .
2023/06/02 16:09:01 NOTICE: link.txt: Can't follow symlink without -L/--copy-links
file.txt

$ rclone lsf --links .
file.txt
link.txt.rclonelink

$ echo "link.txt.rclonelink" > include.txt

$ rclone lsf --links --files-from-raw include.txt .
link.txt.rclonelink

$ rclone lsf --files-from-raw include.txt . # no files found as no --links
$ 

This is how v1.63 beta behaves.... not v1.62.2. You obviously are always step ahead:)

Thanks for taking the time looking into this, @ncw.

Oops. The actual command line code I'm using is much longer, so I removed the irrelevant flags. Looks like I also removed a very relevant flag :blush:
(Note to self: create a minimal example and verbatim copy it)

Yes, both those files are symlinks. You can see that the DEBUG log says that the .rclonelink files are excluded.

It appears this works for a local destination (I tried it here too), but not a remote one (at least pCloud and pCloud-crypt).

I did now happen to notice that including both link.txt and link.txt.rclonelink in the include.txt makes it work! That could also explains why adding link.txt* to the --include-from file worked too. I'm assuming --files-from doesn't do pattern matching, so that's why it doesn't work there.

I can use that to work around the issue, but I'm assuming this should really work with listing just the .rclonelink in the --files-from file?

Yes you are right (again :slight_smile: ). This is a bug which was fixed here.

So the answer @brechtm is to try the latest beta and hopefully that will fix it for you.

We are due a new rclone release quite soon which is good since I've obviously forgotten all the bug fixes in it already!

1 Like

I tested with the latest beta, which is happy with just the .rclonelink line in the --files-from file :+1:

I'll list both the filename and the filename with .rclonelink for now for backward compatibility.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.