As I often do when I want to understand some feature of rclone, I set up a test with local "remotes" to see what will happen. In doing so, I was surprised when I got an error because a local remote doesn't support server side copy. (version 1.55.0)
Is there a reason for this? It seems like something that should be easily doable, right?
hello,
not sure what you are asking, as between local to local, there is not a server?
you know the drill
I am asking purely from a capability standpoint, why server-side copy is not supported in rclone with a local "remote".
With all due respect, this is not a "why isn't this working". It is a "why isn't this a feature". It is not about an issue I am having. It is documented that a local remote does not support copy.
I am not having a problem to be debugged. There is no command I am working on. I intentionally chose "feature" for the topic and not "help and support". So "the drill" makes sense for those and I 100% empathize with anyone trying to help and not being provided what was asked. But (a) I wasn't asked and (b) it is not about support.
What would you expect to happen for the server side?
A hard link ? A link? A move of the file?
For me, I wouldn't use rclone if I was locally only as there are better tools for local only rather than rclone. I use rclone for cloud based stuff.
sure, but you did write a rclone command and did get a rclone error.
and i agree with @Animosity022, i have never used rclone for local.
Well, again, I am asking about why the capability isn't there but I was trying to better understand --compare-dest and --copy-dest since I wasn't 100% sure I understood it from the documentation. The latter (which basically uses the data already on the remote rather than sync) is the one that failed because it needs server-side copies
No config file for this one
rclone v1.55.0
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.2
- go/linking: dynamic
- go/tags: cmount
$ rclone sync -vv A/ B/new --copy-dest=B/old
2021/04/04 20:07:47 DEBUG : Using config file from "/Users/<USER>/.config/rclone/rclone.conf"
2021/04/04 20:07:47 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "sync" "-vv" "A/" "B/new" "--copy-dest=B/old"]
2021/04/04 20:07:47 DEBUG : Creating backend with remote "A/"
2021/04/04 20:07:47 DEBUG : fs cache: renaming cache item "A/" to be canonical "/Users/<USER>/tmp/A"
2021/04/04 20:07:47 DEBUG : Creating backend with remote "B/new"
2021/04/04 20:07:47 DEBUG : fs cache: renaming cache item "B/new" to be canonical "/Users/<USER>/tmp/B/new"
2021/04/04 20:07:47 DEBUG : Creating backend with remote "B/old"
2021/04/04 20:07:47 DEBUG : fs cache: renaming cache item "B/old" to be canonical "/Users/<USER>/tmp/B/old"
2021/04/04 20:07:47 ERROR : Fatal error received - not attempting retries
2021/04/04 20:07:47 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (fatal error encountered)
Elapsed time: 0.0s
2021/04/04 20:07:47 DEBUG : 4 go routines active
2021/04/04 20:07:47 Failed to sync: can't use --copy-dest on a remote which doesn't support server side copy
Again, I understand why the sync failed. I just don't understand why server-side copy doesn't exist for local.
First of all, while this example was local-to-local, it is very common for one "remote" to be local. What if I was pulling a cloud repo and I wanted to use --copy-dest to save bandwidth? That is equally valid as any other reason to use --copy-dest. The fact that this was local to local was for testing.
However, while there are other tools as well, rclone is pretty good for moving files around local-to-local if you (a) want the integrity checks that come with it and (b) want a threaded speedup. Many tools do not thread a move. This is especially good for moving files on a slow spinning rust drive where there isn't anything already (that would benefit from rsync's diff transfers). But yes, it is not the primary use case for sure.
I actually think this is pretty straigh-forward. I expect a copy. Just like I would get on any other remote that supports copies. Any kind of link is (a) not really in rclone's wheelhouse since most cloud systems don't have that concept and (b) would absolutly break the "principle of least surprise".
With an example it is a straight forward as you didn't provide one earlier.
felix@gemini:~$ rclone sync -vv /home/felix/A /home/felix/B
2021/04/04 22:26:48 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/04/04 22:26:48 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "sync" "-vv" "/home/felix/A" "/home/felix/B"]
2021/04/04 22:26:48 DEBUG : Creating backend with remote "/home/felix/A"
2021/04/04 22:26:48 DEBUG : Creating backend with remote "/home/felix/B"
2021/04/04 22:26:48 DEBUG : Local file system at /home/felix/B: Waiting for checks to finish
2021/04/04 22:26:48 DEBUG : Local file system at /home/felix/B: Waiting for transfers to finish
2021/04/04 22:26:48 DEBUG : hosts: MD5 = e9b49c993fe22326c398ecea2fd9b219 OK
2021/04/04 22:26:48 INFO : hosts: Copied (new)
2021/04/04 22:26:48 DEBUG : Waiting for deletions to finish
2021/04/04 22:26:48 INFO :
Transferred: 221 / 221 Bytes, 100%, 131.571 kBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 0.0s
2021/04/04 22:26:48 DEBUG : 3 go routines active
My test worked fine, but you are doing something a bit different so not sure if that's a bug or defect.
Rclone doesn't implement server side copy for the local backend because there isn't an OS function "copy this file".
Rclone would have to open the file, open the destination, read chunks from one and write to the other, etc.
This is exactly what the non server side fallback copy does, so there is no need to duplicate that code.
Yes it might be useful for testing sometimes!
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.