Rclone not handling Symlinks as Symlinks

Help and Support
1

What is the problem you are having with rclone?

I am trying to sync multiple remote directories, that resides on a FreeBSD system/ZFS pool, to a matching set of local directories that are on an EXT4/Linux Ubuntu 22.04 host.

**I have to go that way because the FreeBSD system is too old and Rclone does not run on it. So I am basically running Rclone on the destination to fetch the data from the source. Also, Rsync performance is horrible compared with Rclone. **

Each remote directory contains one sub-directory, and multiple symlinks to that sub-directory like so;

$ ls -lah
drw-r--r--  6 user staff 4.0K Jul 13 06:54 backup
drwxr-xr-x  3 user staff  4.0K Jul 29 05:51 music
drwxrwxrwx  6 user staff  4.0K Sep 25 02:11 plex
lrwxrwxrwx  1 user staff   5 Oct 23 16:08 plex2 -> plex/
lrwxrwxrwx  1 user staff   5 Oct 23 16:09 plex3 -> plex/

Rclone does fetch the data, but it treats each symlink on the source as a directory, resulting in 3X data written on the target.

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.18.1

This seems to be the latest version available to my system.

Which cloud storage system are you using? (eg Google Drive)

None. The sync is done over SSH.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync remotehost:/data /local_backup_path

I also tried with

rclone sync remotehost:/data /local_backup_path --links

Please run 'rclone config redacted' and share the full output.

This shows Rclone's help menu. No errors are displayed.

2

welcome to the forum,

that is a custom compiled version from four years ago.

  1. uninstall that version
  2. https://rclone.org/install/#script-installation
3

...Once you've upgraded use this flag

-l, --links                           Translate symlinks to/from regular files with a '.rclonelink' extension
4

Thank you, both, for the responses. I updated, and now I have version

rclone v1.68.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-122-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: none

Redacted configs are:

[host1]
type = sftp
host = XXX
user = XXX
port = 22
pass = XXX
md5sum_command = md5 -r
sha1sum_command = sha1 -r
shell_type = unix

Commands currently used to sync:

rclone sync remotehost:/data /local_backup_path --links

rclone copy remotehost:/data /local_backup_path --links

Unfortunately, same issue still presists. Both commands still create local directories from remote Symlinks. What else could I try?

5

Ah, the source is SFTP - I thought it was local.

We are in the process of adding symlink support to sftp but it isn't there yet

I'd probably use

  --sftp-skip-links                 Set to skip any symlinks and any other non regular files

And remake the links manually for the moment

6

how about samba/smb. easy to setup, file verification using checksums, rclone can support it multiple ways.
samba has follow symlinks = no

and i have a handy, dandy, howto guide about rclone+smb

7

Thank you @asdffdsa for the idea. Unfortunately, as stated in my original post, the FreeBSD system is beyond obsolete, and the datasets are set up in a complex way that would make attempting SMB exports a nightmare. I don't think I would want to touch that. Still, Thank you, for the hint and the thorough guide. Bookmarking it for the future.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.