SFTP remote server side copy

What is the problem you are having with rclone?

rclone is set to use an SSH/SFTP remote to transfer files to. Sometimes there is a need to copy a file in a different path in the same remote and I would like to avoid using ssh to copy the file around.
From the docs it seems like rclone will always try to use a server-side action if it can. I also tried to use --server-side-across-configs but it still ends up being downloaded and uploaded.

edit after comment #5: The below only makes sense if the src and dst remotes are the same - I was considering only my specific scenario
Since cp in sftp is not supported (or still new?), would it be possible to a) use sftp ln instead or b) have rclone send an ssh remote cp src dest, like it does with md5sum when it detects it has shell access.
The above could be done when --server-side-across-configs or some sftp specific flag is used.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

  • os/version: debian 10.13 (64 bit)
  • os/kernel: 4.19.0-25-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

SFTP remote

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vP --server-side-across-configs --fast-list sftp_remote:path1/file sftp_remote:path2/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

type = sftp
host = <host>
user = <user>
port = <port>
key_file = ~/.ssh/id_ed25519
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

with sftp, rclone does not support server-side copy.
tho, does support server-side move.

DEBUG : sftp://xxx@xxx.xxx:23/zork/: Shell type "unix" from config
DEBUG : sftp://xxx@xxx.xxx:23/zork/: Relative path resolved to "/home/zork"
DEBUG : sftp://xxx@xxx.xxx:23/zork/: Using root directory "/home/zork"
DEBUG : file01.ext: Need to transfer - File not found at Destination
INFO  : file01.ext: Moved (server-side) to: file02.ext

Here you are all remotes supported features:

As you can see SFTP does not support Copy meaning server-side copy is not possible.

If it is something you need I would suggest to use webdav by running rclone serve webdav

As I mentioned I am aware that SFTP does not support copy.
I suggested some alternatives to that though I understand it is most likely a niche case so it might end up at the bottom of the feature list or not even at all for ncw :slight_smile:

that is interesting. that rclone, for a source and dest, would call cp or generic command.

Well it is, you are right. My mistake was that I forgot that this is applicable only when the src and dst is the same sftp remote. If they are different, obviously we enter the voodoo land. Which makes it an even more niche or N/A case.

Server side copies generally only apply in the same remote or the same cloud storage system.

The sftp library rclone uses has a hardlink function so your option "a" could work quite well as it is standardised.

I did a very quick proof of concept of this - please give it a go! It works with openssh and the sftp that ships with that at least.

v1.65.0-beta.7380.5d0af2274.fix-sftp-copy on branch fix-sftp-copy (uploaded in 15-30 mins)

I'm not sure all sftp servers will support this and certainly not all file systems support hardlinks so it needs to fall back intelligently to a normal copy. I didn't implement that for the POC though!

Whether a hardlink is as good as copying the file is another question that probably needs addressing. Maybe the backend needs an option to explicitly enable this, eg --sftp-copy-is-hardlink or something like that.

Thank you for the quick beta! I will gladly try it out but are the linux versions missing?

Sorry the tests failed!

I discovered a bug in rclone serve sftp which was just plain ignoring the hardlink command instead of returning an unsupported error.

This changes it so that if the server does report unsupported it will fall back to the old way of copying.

v1.65.0-beta.7383.7e72461ff.fix-sftp-copy on branch fix-sftp-copy (uploaded in 15-30 mins)

Thoughts on that? Do we need a flag?

100%. IMO creating hard links should not be default behaviour. If user does not understand hard links concept it can even lead to data loss etc.

That would be the safe and sensible thing to do.

How does a hard link lead to data loss?

  1. lets say I sync local to sftp.
  2. some file is hard linked (via server-side-copy)
  3. editing one of these files will lead to second file changes - as in reality it is all one content

It comes down to when you modify the file. So if you are taking immutable backups there is no problem. But if you append data to one of the hardlinks, you'll be appending it to both. Most editors take care to break hardlinks first if you edit hardlinked files, but not everything does.

It occurs to me that I could implement this flag for the local filesystem too. That would then make --copy-dest for instance make a zero sized fully populated backup.

I agree with the other peeps that it should have its own flag, as copy is not the same as hardlink.

Would like to chime in just that the documentation/help, as the flag would become quite long? should make it very clear that this flag has an effect only when the two remotes are the same otherwise it will be ignored and work as a normal copy.

hmm, that's not data loss to me as that's just a collision and can happen regardless of links or not links. I'm think data gets deleted completely.

Again, I agree as I wouldn't hardlink to default as if I want to hardlink something, I 100% do that with a flag in any other program I use and it's always turned on (Sonarr/Radarr are some examples).

@ncw the beta works just fine!

./rclone copy -vP sftp_remote:"path/filename" sftp_remote:"differentpath"

2023-09-22 18:29:38 INFO  : <filename>: Copied (server-side copy)
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Server Side Copies:     1 @ 343.069 MiB
Elapsed time:         6.8s
2023/09/22 18:29:38 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Server Side Copies:     1 @ 343.069 MiB
Elapsed time:         6.8s

and from the remote side we can see it's a hardlink:

-rw-r--r-- 2 loke loke  359733729 Sep 18 14:02 filename