Syncing links with --links between locally mounted drives fails due to: readlink, no such file or directory

Suspected Bug
1

What is the problem you are having with rclone?

Syncing links with --links between locally mounted drives fails due to: readlink ... no such file or directory.

Run the command 'rclone version' and share the full output of the command.

phone:~$ /home/user/rclone-v1.63.0-beta.6937.64cf9ac91-linux-arm64/rclone version
rclone v1.63.0-beta.6937.64cf9ac91
- os/version: alpine 3.17.3 (64 bit)
- os/kernel: 6.1.9 (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

local+local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/home/user/rclone-v1.63.0-beta.6937.64cf9ac91-linux-arm64/rclone \
 sync \
 / \
 /media/t7/phone-backup/ \
 --filter='- home/user/.local/**' \
 --filter='- home/user/.cache/**' \
 --filter='- home/user/.mozilla/**' \
 --filter='+ home/**' \
 --filter='+ media/sdcard/**' \
 --filter='- *' \
 --modify-window 2s \
 --links \
 --bwlimit 30M \
 --progress \
 -vv \
 --log-file rclone-test.log \
 --check-first \
 --delete-excluded \
 --transfers 4 \
 --create-empty-src-dirs \
 --config=/dev/null

The rclone config contents with secrets removed.

None. See the above command.

A log from the command with the -vv flag

2023/04/04 11:15:04 INFO  : Starting bandwidth limiter at 30Mi Byte/s
2023/04/04 11:15:04 DEBUG : rclone: Version "v1.63.0-beta.6937.64cf9ac91" starting with parameters ["/home/user/rclone-v1.63.0-beta.6937.64cf9ac91-linux-arm64/rclone" "sync" "/" "/media/t7/phone-backup/" "--filter=- home/user/.local/**" "--filter=- home/user/.cache/**" "--filter=- home/user/.mozilla/**" "--filter=+ home/**" "--filter=+ media/sdcard/**" "--filter=- *" "--modify-window" "2s" "--links" "--bwlimit" "30M" "--progress" "-vv" "--log-file" "rclone-test.log" "--check-first" "--delete-excluded" "--transfers" "4" "--create-empty-src-dirs" "--config=/dev/null"]
2023/04/04 11:15:04 DEBUG : Creating backend with remote "/"
2023/04/04 11:15:04 DEBUG : Using config file from ""
2023/04/04 11:15:04 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2023/04/04 11:15:04 DEBUG : fs cache: renaming cache item "/" to be canonical "local{b6816}:/"
2023/04/04 11:15:04 DEBUG : Creating backend with remote "/media/t7/phone-backup/"
2023/04/04 11:15:04 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2023/04/04 11:15:04 DEBUG : fs cache: renaming cache item "/media/t7/phone-backup/" to be canonical "local{b6816}:/media/t7/phone-backup"
2023/04/04 11:15:04 INFO  : Local file system at /media/t7/phone-backup: Running all checks before starting transfers
2023/04/04 11:15:04 DEBUG : usr: Excluded
2023/04/04 11:15:04 DEBUG : lib: Excluded
2023/04/04 11:15:04 DEBUG : proc: Excluded
2023/04/04 11:15:04 DEBUG : opt: Excluded
2023/04/04 11:15:04 DEBUG : sbin: Excluded
2023/04/04 11:15:04 DEBUG : lost+found: Excluded
2023/04/04 11:15:04 DEBUG : bin: Excluded
2023/04/04 11:15:04 DEBUG : tmp: Excluded
2023/04/04 11:15:04 DEBUG : mnt: Excluded
2023/04/04 11:15:04 DEBUG : sys: Excluded
2023/04/04 11:15:04 DEBUG : run: Excluded
2023/04/04 11:15:04 DEBUG : root: Excluded
2023/04/04 11:15:04 DEBUG : etc: Excluded
2023/04/04 11:15:04 DEBUG : boot: Excluded
2023/04/04 11:15:04 DEBUG : srv: Excluded
2023/04/04 11:15:04 DEBUG : dev: Excluded
2023/04/04 11:15:04 DEBUG : var: Excluded
2023/04/04 11:15:04 DEBUG : media/usb: Excluded
2023/04/04 11:15:04 DEBUG : media/t7: Excluded
2023/04/04 11:15:04 DEBUG : media/floppy: Excluded
2023/04/04 11:15:04 DEBUG : media/cdrom: Excluded
2023/04/04 11:15:04 DEBUG : home/user/.mozilla: Excluded
2023/04/04 11:15:04 DEBUG : home/user/.local: Excluded
2023/04/04 11:15:04 DEBUG : home/user/.cache: Excluded
2023/04/04 11:15:10 ERROR : home/user/sdcard.rclonelink: Failed to read link size: readlink /media/t7/phone-backup/home/user/sdcard: no such file or directory
2023/04/04 11:15:10 DEBUG : home/user/sdcard.rclonelink: Modification times differ by -5h0m1.167166878s: 2022-07-28 14:18:23.167166878 -0500 CDT, 2022-07-28 09:18:22 -0500 CDT
2023/04/04 11:15:10 ERROR : home/user/sdcard.rclonelink: Failed to calculate dst hash: hash: failed to open: readlink /media/t7/phone-backup/home/user/sdcard: no such file or directory
2023/04/04 11:15:10 DEBUG : home/user/sdcard.rclonelink: md5 differ
2023/04/04 11:16:36 DEBUG : Local file system at /media/t7/phone-backup: Waiting for checks to finish
2023/04/04 11:16:36 INFO  : Local file system at /media/t7/phone-backup: Checks finished, now starting transfers
2023/04/04 11:16:36 DEBUG : Local file system at /media/t7/phone-backup: Waiting for transfers to finish
2023/04/04 11:16:36 ERROR : home/user/sdcard.rclonelink: Failed to copy: symlink /media/sdcard /media/t7/phone-backup/home/user/sdcard: operation not permitted
2023/04/04 11:16:36 ERROR : Local file system at /media/t7/phone-backup: not deleting files as there were IO errors
2023/04/04 11:16:36 ERROR : Local file system at /media/t7/phone-backup: not deleting directories as there were IO errors
2023/04/04 11:16:36 ERROR : Attempt 1/3 failed ........

NOTE: Cruft regarding other symlinks is removed, as this is happening with all of them, and I wanted to focus on one of them.
Also, nothing else was related in the log.

Some Notes

Also, the following:

phone:~$ ls -al sdcard
lrwxrwxrwx 1 user user 13 Jul 28  2022 sdcard -> /media/sdcard

phone:~$ ls -al /media/t7/phone-backup/home/user/sdcard*
-rwxr-xr-x 1 user user 13 Jul 28  2022 /media/t7/phone-backup/home/user/sdcard.rclonelink

Some background:

  • The above is all happening on the same machine. It is just syncing to a mounted external drive for offline backup.
  • However, the sdcard.rclonelink file already exists on the external drive because I used to do this in a different way... I used to have the external drive mounted to a different machine, and I ran rclone sync across the network (sftp).
  • So, this problem does not seem to happen with sftp.
  • It appears to me based on the error message that instead of syncing
    /home/user/sdcard.rclonelink to /media/t7/phone-backup/home/user/sdcard.rclonelink
    it is trying to sync
    /home/user/sdcard.rclonelink to /media/t7/phone-backup/home/user/sdcard
    but only when syncing local<->local

I hope this makes sense!

2

I manually deleted /media/t7/phone-backup/home/user/sdcard.rclonelink and ran the sync again and got this:

2023/04/04 12:32:27 DEBUG : home/user/sdcard.rclonelink: Need to transfer - File not found at Destination
2023/04/04 12:33:54 ERROR : home/user/sdcard.rclonelink: Failed to copy: symlink /media/sdcard /media/t7/phone-backup/home/user/sdcard: operation not permitted

Note that any files that are not symlinks sync just fine.

3

Your sdcard is formatted vfat I'd guess and vfat doesn't support symlinks.

That's the cause of this error.

That is what the --links flag does. Remember it applies to the source and the destination.

If you want links to apply to just the source you'd do this I think

rclone copy :local,links:/source /dest
4

Thanks, @ncw

Maybe I misunderstood. I assumed that, using --link, a real symlink in the source would ALWAYS be translated into a FILE.rclonelink file in the destination (instead of an actual symlink)? This is what occurred when I did the exact same operation local->sftp vs local->local.

The manual says for --links:

Translate symlinks to/from regular files with a '.rclonelink' extension
5

Its a two way thing.

If you are writing a file to a local file system called file.rclonelink it will get translated into a symlink called file.

That is what the "to/from" bit implies.

Here is an example

$ ls -l src
total 0
lrwxrwxrwx 1 ncw ncw 8 Mar 20 12:11 dangling -> not-here

$ rclone -l tree src
/
└── dangling.rclonelink

0 directories, 1 files

$ rclone -l tree dst
/

0 directories, 0 files

$ rclone copy -l -v src dst
2023/04/05 10:54:32 INFO  : dangling.rclonelink: Copied (new)
2023/04/05 10:54:32 INFO  : 
Transferred:   	          8 B / 8 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

$ rclone -l tree dst
/
└── dangling.rclonelink

0 directories, 1 files

$ ls -l dst
total 0
lrwxrwxrwx 1 ncw ncw 8 Mar 20 12:11 dangling -> not-here

$

And here is an example with only setting --links for the source which will copy the .rclonelink file.

$ rm dst/dangling 
$ rclone -l tree dst
/

0 directories, 0 files
$ rclone copy -v :local,links:src dst
2023/04/05 10:55:38 INFO  : dangling.rclonelink: Copied (new)
2023/04/05 10:55:38 INFO  : 
Transferred:   	          8 B / 8 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

$ rclone tree dst
/
└── dangling.rclonelink

0 directories, 1 files

$ ls -l dst
total 4
-rw-rw-r-- 1 ncw ncw 8 Mar 20 12:11 dangling.rclonelink

$ cat dst/dangling.rclonelink 
not-here
$

Hopefully that makes more sense!

6

Oh, I see! Yes, perfectly! I must say that it isn't clear in the docs that "local" always gets a real symlink, though.

Thanks!

7

So, if this is the "backup":

$ rclone copy -v :local,links:src dst

What is the "restore"? This?

$ rclone copy -v dst :local,links:src
8

Yes, that's right.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.