SFTP remote server side copy

Help and Support
1

What is the problem you are having with rclone?

rclone is set to use an SSH/SFTP remote to transfer files to. Sometimes there is a need to copy a file in a different path in the same remote and I would like to avoid using ssh to copy the file around.
From the docs it seems like rclone will always try to use a server-side action if it can. I also tried to use --server-side-across-configs but it still ends up being downloaded and uploaded.

edit after comment #5: The below only makes sense if the src and dst remotes are the same - I was considering only my specific scenario
Since cp in sftp is not supported (or still new?), would it be possible to a) use sftp ln instead or b) have rclone send an ssh remote cp src dest, like it does with md5sum when it detects it has shell access.
The above could be done when --server-side-across-configs or some sftp specific flag is used.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

  • os/version: debian 10.13 (64 bit)
  • os/kernel: 4.19.0-25-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

SFTP remote

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vP --server-side-across-configs --fast-list sftp_remote:path1/file sftp_remote:path2/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[sftp_remote]
type = sftp
host = <host>
user = <user>
port = <port>
key_file = ~/.ssh/id_ed25519
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
2

with sftp, rclone does not support server-side copy.
tho, does support server-side move.

DEBUG : sftp://xxx@xxx.xxx:23/zork/: Shell type "unix" from config
DEBUG : sftp://xxx@xxx.xxx:23/zork/: Relative path resolved to "/home/zork"
DEBUG : sftp://xxx@xxx.xxx:23/zork/: Using root directory "/home/zork"
DEBUG : file01.ext: Need to transfer - File not found at Destination
INFO  : file01.ext: Moved (server-side) to: file02.ext
3

Here you are all remotes supported features:

As you can see SFTP does not support Copy meaning server-side copy is not possible.

If it is something you need I would suggest to use webdav by running rclone serve webdav

4

As I mentioned I am aware that SFTP does not support copy.
I suggested some alternatives to that though I understand it is most likely a niche case so it might end up at the bottom of the feature list or not even at all for ncw :slight_smile:

5

that is interesting. that rclone, for a source and dest, would call cp or generic command.

6

Well it is, you are right. My mistake was that I forgot that this is applicable only when the src and dst is the same sftp remote. If they are different, obviously we enter the voodoo land. Which makes it an even more niche or N/A case.

7

Server side copies generally only apply in the same remote or the same cloud storage system.

The sftp library rclone uses has a hardlink function so your option "a" could work quite well as it is standardised.

I did a very quick proof of concept of this - please give it a go! It works with openssh and the sftp that ships with that at least.

v1.65.0-beta.7380.5d0af2274.fix-sftp-copy on branch fix-sftp-copy (uploaded in 15-30 mins)

I'm not sure all sftp servers will support this and certainly not all file systems support hardlinks so it needs to fall back intelligently to a normal copy. I didn't implement that for the POC though!

Whether a hardlink is as good as copying the file is another question that probably needs addressing. Maybe the backend needs an option to explicitly enable this, eg --sftp-copy-is-hardlink or something like that.

8

Thank you for the quick beta! I will gladly try it out but are the linux versions missing?

9

Sorry the tests failed!

I discovered a bug in rclone serve sftp which was just plain ignoring the hardlink command instead of returning an unsupported error.

This changes it so that if the server does report unsupported it will fall back to the old way of copying.

v1.65.0-beta.7383.7e72461ff.fix-sftp-copy on branch fix-sftp-copy (uploaded in 15-30 mins)

Thoughts on that? Do we need a flag?

10

100%. IMO creating hard links should not be default behaviour. If user does not understand hard links concept it can even lead to data loss etc.

11

That would be the safe and sensible thing to do.

12

How does a hard link lead to data loss?

13
  1. lets say I sync local to sftp.
  2. some file is hard linked (via server-side-copy)
  3. editing one of these files will lead to second file changes - as in reality it is all one content
14

It comes down to when you modify the file. So if you are taking immutable backups there is no problem. But if you append data to one of the hardlinks, you'll be appending it to both. Most editors take care to break hardlinks first if you edit hardlinked files, but not everything does.

It occurs to me that I could implement this flag for the local filesystem too. That would then make --copy-dest for instance make a zero sized fully populated backup.

15

I agree with the other peeps that it should have its own flag, as copy is not the same as hardlink.

Would like to chime in just that the documentation/help, as the flag would become quite long? should make it very clear that this flag has an effect only when the two remotes are the same otherwise it will be ignored and work as a normal copy.

16

hmm, that's not data loss to me as that's just a collision and can happen regardless of links or not links. I'm think data gets deleted completely.

Again, I agree as I wouldn't hardlink to default as if I want to hardlink something, I 100% do that with a flag in any other program I use and it's always turned on (Sonarr/Radarr are some examples).

17

@ncw the beta works just fine!

./rclone copy -vP sftp_remote:"path/filename" sftp_remote:"differentpath"

2023-09-22 18:29:38 INFO  : <filename>: Copied (server-side copy)
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Server Side Copies:     1 @ 343.069 MiB
Elapsed time:         6.8s
2023/09/22 18:29:38 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Server Side Copies:     1 @ 343.069 MiB
Elapsed time:         6.8s

and from the remote side we can see it's a hardlink:

-rw-r--r-- 2 loke loke  359733729 Sep 18 14:02 filename
18

I've enabled this with a flag --sftp-copy-is-hardlink or set copy_is_hardlink = true in your config file.

v1.65.0-beta.7411.b556e763a.fix-sftp-copy on branch fix-sftp-copy (uploaded in 15-30 mins)

It it tests out OK I'll merge this version :slight_smile:

19

I had it running for couple of days, I found no errors in the logs and periodically checked that all files are hardlinks. So looks ready to be merged!

Thank you for your great work in rclone!

20

Thanks for testing

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.65

Here is the config item to enable it.

--sftp-copy-is-hardlink

Set to enable server side copies using hardlinks.

The SFTP protocol does not define a copy command so normally server
side copies are not allowed with the sftp backend.

However the SFTP protocol does support hardlinking, and if you enable
this flag then the sftp backend will support server side copies. These
will be implemented by doing a hardlink from the source to the
destination.

Not all sftp servers support this.

Note that hardlinking two files together will use no additional space
as the source and the destination will be the same file.

This feature may be useful backups made with --copy-dest.

Properties:

  • Config: copy_is_hardlink
  • Env Var: RCLONE_SFTP_COPY_IS_HARDLINK
  • Type: bool
  • Default: false
1 Like