Using the move command to a destination directory which happens to be the source directory but not with the same path (with use of symlinks) results in directory content removal.
Yes, I did do this
What is your rclone version (output from rclone version)
$ rclone version
rclone v1.57.0
os/version: ubuntu 18.04 (64 bit)
os/kernel: 4.4.0-19041-Microsoft (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.17.2
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Local
The command you were trying to run (eg rclone copy /tmp remote:tmp)
$ rclone move dir1/ dir2/
dir2 is a symlink pointing to dir1:
$ ls -l
total 0
drwxr-x--- 1 danbei danbei 4.0K Feb 24 21:11 dir1
lrwxrwxrwx 1 danbei danbei 4 Feb 24 21:10 dir2 -> dir1
A log from the command with the -vv flag
$ rclone -vv move dir1/ dir2/
2022/02/24 21:11:13 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "move" "dir1/" "dir2/"]
2022/02/24 21:11:13 DEBUG : Creating backend with remote "dir1/"
2022/02/24 21:11:13 DEBUG : Using config file from "/home/danbei/.config/rclone/rclone.conf"
2022/02/24 21:11:13 DEBUG : fs cache: renaming cache item "dir1/" to be canonical "/mnt/d/test/dir1"
2022/02/24 21:11:13 DEBUG : Creating backend with remote "dir2/"
2022/02/24 21:11:13 DEBUG : fs cache: renaming cache item "dir2/" to be canonical "/mnt/d/test/dir2"
2022/02/24 21:11:13 DEBUG : Local file system at /mnt/d/test/dir2: Using server-side directory move
2022/02/24 21:11:13 INFO : Local file system at /mnt/d/test/dir2: Server side directory move failed - fallback to file moves: can t copy directory - destination already exists
2022/02/24 21:11:13 DEBUG : Local file system at /mnt/d/test/dir2: Waiting for checks to finish
2022/02/24 21:11:13 DEBUG : file: Size and modification time the same (differ by 0s, within tolerance 100ns)
2022/02/24 21:11:13 DEBUG : file: Unchanged skipping
2022/02/24 21:11:13 INFO : file: Deleted
2022/02/24 21:11:13 DEBUG : Local file system at /mnt/d/test/dir2: Waiting for transfers to finish
2022/02/24 21:11:13 INFO : There was nothing to transfer
2022/02/24 21:11:13 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 2 / 2, 100%
Deleted: 1 (files), 0 (dirs)
Elapsed time: 0.0s
2022/02/24 21:11:13 DEBUG : 2 go routines active
This might not be feasible for all remotes combinations, but maybe for local remote to local remote?
You can't always prevent a user from doing something unexpected and I wouldn't expect rclone to be able to check for that it's a super edge case.
I always make sure to test things before doing destructive operations as dry-run and some validation before goes a long way to ensure you aren't breaking or deleting data as over the years, I've seen many examples of a rm -rf * command.
In my upload scripts, I make sure I don't goof and do a move loop and blow away my cloud stuff:
# Local Drive - This must be a local mount point on your server that is used for the source of files
# WARNING: If you make this your rclone Google Drive mount, it will create a move loop
# and DELETE YOUR FILES!
# Make sure to set this to the local path you are moving from!!
LOCAL=/local
# Exit if running
if [[ $(pidof -x "$(basename "$0")" -o %PPID) ]]; then
echo "Already running, exiting..."; exit 1; fi
# Is $LOCAL actually a local disk?
if /bin/findmnt $LOCAL -o FSTYPE -n | grep fuse; then
echo "FUSE file system found, exiting..."; exit 1; fi
That's fair, thank you for your script example, I definitively should implement this kind of checks in my scripts.
Though, in the particular case of local remote to local remote, rclone may compute the realpaths of source and destination.
I'm thinking of this part:
rclone allready detects if if use the exact same canonical directory:
$ rclone move dir1/ /mnt/d/test/dir1/
2022/02/24 22:19:04 ERROR : Local file system at /mnt/d/test/dir1/: Nothing to do as source and destination are the same
Move does the same thing as sync and deletes the file if you have a loop:
drwxrwxr-x 1 felix felix 0 Feb 24 17:24 test1
lrwxrwxrwx 1 felix felix 5 Feb 24 17:24 test2 -> test1
and
felix@gemini:~$ rclone move test2 test1 -vvv
2022/02/24 17:25:07 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/02/24 17:25:07 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "move" "test2" "test1" "-vvv"]
2022/02/24 17:25:07 DEBUG : Creating backend with remote "test2"
2022/02/24 17:25:07 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/02/24 17:25:07 DEBUG : fs cache: renaming cache item "test2" to be canonical "/home/felix/test2"
2022/02/24 17:25:07 DEBUG : Creating backend with remote "test1"
2022/02/24 17:25:07 DEBUG : fs cache: renaming cache item "test1" to be canonical "/home/felix/test1"
2022/02/24 17:25:07 DEBUG : Local file system at /home/felix/test1: Using server-side directory move
2022/02/24 17:25:07 INFO : Local file system at /home/felix/test1: Server side directory move failed - fallback to file moves: can't copy directory - destination already exists
2022/02/24 17:25:07 DEBUG : testfile: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/02/24 17:25:07 DEBUG : testfile: Unchanged skipping
2022/02/24 17:25:07 DEBUG : Local file system at /home/felix/test1: Waiting for checks to finish
2022/02/24 17:25:07 INFO : testfile: Deleted
2022/02/24 17:25:07 DEBUG : Local file system at /home/felix/test1: Waiting for transfers to finish
2022/02/24 17:25:07 INFO : There was nothing to transfer
2022/02/24 17:25:07 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 2 / 2, 100%
Deleted: 1 (files), 0 (dirs)
Elapsed time: 0.0s
2022/02/24 17:25:07 DEBUG : 3 go routines active
For local stuff, you can check if it's a directory and not a link or something.
Offhand, I'm not sure how rclone would validate that albeit I'm sure someone smarter than me can figure out a way.
I think it's an edge case though as for local things, I don't use rclone as I only use rclone for a cloud type remote.
Yes, my point is that checking if source and destination are equal is already implemented.
If this request is accepted, the remaining thing to do would be to evaluate symlinks when computing the root path of the local remote, something like that:
diff --git a/backend/local/local.go b/backend/local/local.go
index 3c00d5389..f90c71f45 100644
--- a/backend/local/local.go
+++ b/backend/local/local.go
@@ -1327,6 +1327,10 @@ func cleanRootPath(s string, noUNC bool, enc encoder.MultiEncoder) string {
if err == nil {
s = s2
}
+ s2, err := filepath.EvalSymlinks(s)
+ if err == nil {
+ s = s2
+ }
}
s = filepath.ToSlash(s)
vol := filepath.VolumeName(s)
@@ -1344,6 +1348,10 @@ func cleanRootPath(s string, noUNC bool, enc encoder.MultiEncoder) string {
if err == nil {
s = s2
}
+ s2, err := filepath.EvalSymlinks(s)
+ if err == nil {
+ s = s2
+ }
}
s = enc.FromStandardPath(s)
return s
Is there a particular reason why this has not been done in the first place that I'm not aware of?
You are right rclone does check to see if source and dest are equal to help avoid accidents.
However it checks them at a higher level than the local backend so it doesn't have any knowledge of symlinks etc at that point, it is comparing rclone urls to see if they are equal. So you could do rclone move drive:source drive:source and rclone would notice that.
So even if the local backend did evaluate the symlinks, it is too late at that point for the duplicate detection I think.
Don't the function that do the check gets the paths to compare from the backend?
File cmd/move/move.go line 60 -> calls MoveDir fs/sync/sync.go line 1103 -> calls Same with file system as arg wich I suppose are bind to the backends.
I build the patch mentioned in my last post (minus the syntax error) and it does what I intend:
The directory content:
$ ls -l
total 0
drwxr-x--- 1 danbei danbei 4.0K Feb 25 13:02 dir1
lrwxrwxrwx 1 danbei danbei 4 Feb 25 08:31 dir2 -> dir1
The erroneous move command:
$ /mnt/d/depots/rclone/rclone -vv move dir1/ dir2/
2022/02/25 13:03:05 DEBUG : rclone: Version "v1.58.0-DEV" starting with parameters ["/mnt/d/depots/rclone/rclone" "-vv" "move" "dir1/" "dir2/"]
2022/02/25 13:03:05 DEBUG : Creating backend with remote "dir1/"
2022/02/25 13:03:05 DEBUG : Using config file from "/home/danbei/.config/rclone/rclone.conf"
2022/02/25 13:03:05 DEBUG : fs cache: renaming cache item "dir1/" to be canonical "/mnt/d/test/dir1"
2022/02/25 13:03:05 DEBUG : Creating backend with remote "dir2/"
2022/02/25 13:03:05 DEBUG : fs cache: renaming cache item "dir2/" to be canonical "/mnt/d/test/dir1"
2022/02/25 13:03:05 ERROR : Local file system at /mnt/d/test/dir1: Nothing to do as source and destination are the same
2022/02/25 13:03:05 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.0s
2022/02/25 13:03:05 DEBUG : 2 go routines active
The important line in the 6th: 2022/02/25 13:03:05 DEBUG : fs cache: renaming cache item "dir2/" to be canonical "/mnt/d/test/dir1"
Whereas without the symlinks evaluation it is: 2022/02/25 13:20:25 DEBUG : fs cache: renaming cache item "dir2/" to be canonical "/mnt/d/test/dir2"
The canonicalisation has changed the backend path which has indeed got picked up. That won't work for paths under the root, but it will work for the root.
So what you are saying is that if the root is a symlink, we should canonicalise it. I don't think that is entirely unreasonable, but I'm concerned about backwards compatibility problems.
Indeed, I was thinking about the root path.
I really don't know enough about rclone to anticipate side effects, I'm curious about what scenarios you suspect there would be problems.
Canonicalising the symlink will interact in some way with the -l and -L flags but I don't know exactly how.
Assuming we can sort that out, there is always a risk of breaking people's workflows when changing stuff, or fixing stuff as rclone has many thousands of users!
I always think of this xkcd at this moment
If you wanted to propose a pull request, then we can discuss further on it!
