`sync' command doesn't track file movements but deletes and completely re-uploads moved files

Help and Support
1

Greetings!

Using rclone v1.33 on OS X El Capitan.

I uploaded a few hundred files to my Google Drive account. Then on my local server I moved some of those files to a new folder.

Original:
/Movies/A.mp4
/Movies/B.mp4
/Movies/C.mp4

New:
/Movies/A/A.mp4
/Movies/B/B.mp4
/Movies/C/C.mp4

Running the command: rclone sync Movies/ gdrive:Movies/ causes each file to be removed from Movies folder of gdrive and then re-uploaded to each newly created folder.

Expected behavior would be for rclone to realize that these are the same files and instead of wasting bandwidth to reupload them, it would tell google drive to just move the file.

Can anyone confirm this? Am I doing something wrong? It feels to me like this is not the right behavior.

1 Like
2

This is normal. This would be the same with any tool like rsync or acd_cli.

3

Thanks for the answer.
That’s too bad. I wonder if this might make a good feature that differentiates rclone. Considering that it can compare md5s why would it need to do this? I don’t think an answer of “because this is how rsync does it” isn’t actually a great answer.

4

Rclone could do this certainly. It is a complex feature though.

If you make an issue at github.com/ncw/rclone we can gauge interest for the feature.

5

Done!

6

Thanks for doing that :slight_smile:

7

hopefully it gains some traction :slight_smile: if i knew what “coding” was i’d help

1 Like
8

I hope this is still being worked on! Being unable to make changes to my folder structures does leave me handicapped.

9

It is! Thanks to bep on github, it’s pretty much done from the sounds of things. It looks like it’s going to get into a future release (but not this upcoming one just yet). See https://github.com/ncw/rclone/issues/888

10

exciting! Can’t wait for it. Is it in the Beta yet?

11

Not quite! I’ll merge it in a day or two most likely.

1 Like
12

Great, after I synced a folder from acd to gdrive I went through the folder renaming a few hundred files, about 2TB in size. I can’t wait to see if a reverse sync would recognize that only the names have changed and nothing else.
This will speedup syncing immensely!

13

This is merged now…

Please find a beta with this in here: http://beta.rclone.org/v1.35-07-gf1221b5/ (uploaded in 15-30 mins)

14

I assume this will be included in all future beta updates, right?

15

Yes that is the way it works, yes.

I’ll post an update in a bit with much better performance - I made a mistake in the patch above…

16
  1. So I had two identical folders on gdrive & acd.
  2. I renamed a few hundred files in the gdrive folder.
  3. I used “rclone sync gdrive:folder acd:folder” to see if it will simply rename the files
  4. It looks like it isn’t working. rclone has transferred over 500GB.

Are there any modifiers that I need to add to the sync function for this to work?

17

Unfortunately Amazon Drive and Google Drive don’t have a compatible hash - you should have seen a message in the log about that.

This means that you can’t use --track-renames between them unfortunately :frowning:

18

What a bummer. So it only works within a given Cloud Storage provider but never between two?

19

That is actually rubbish as you can see in this table!: http://rclone.org/overview/#features

So I was wrong and this should work between Amzon Drive and Google Drive…

Here is me testing it and showing that it does work for me at least…

$ mkdir rename-test
$ echo potato > rename-test/file1
$ rclone sync rename-test drive:rename-test
2017/01/05 13:02:02 Google drive root 'rename-test': Waiting for checks to finish
2017/01/05 13:02:02 Google drive root 'rename-test': Waiting for transfers to finish
2017/01/05 13:02:03 Waiting for deletions to finish
2017/01/05 13:02:03 
Transferred:      7 Bytes (2 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:          3s
$ mv rename-test/file1 rename-test/file2
$ rclone sync rename-test acd:rename-test
2017/01/05 13:02:23 amazon drive root 'rename-test': Waiting for checks to finish
2017/01/05 13:02:23 amazon drive root 'rename-test': Waiting for transfers to finish
2017/01/05 13:02:24 Waiting for deletions to finish
2017/01/05 13:02:24 
Transferred:      7 Bytes (2 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:        3.3s
$ rclone sync --track-renames -v acd:rename-test drive:rename-test
2017/01/05 13:02:45 rclone: Version "v1.35-DEV" starting with parameters ["rclone" "sync" "--track-renames" "-v" "acd:rename-test" "drive:rename-test"]
2017/01/05 13:02:46 amazon drive root 'rename-test': Modify window not supported
2017/01/05 13:02:46 Google drive root 'rename-test': Reading ""
2017/01/05 13:02:47 Google drive root 'rename-test': Finished reading ""
2017/01/05 13:02:47 Google drive root 'rename-test': Making map for --track-renames
2017/01/05 13:02:47 Google drive root 'rename-test': Finished making map for --track-renames
2017/01/05 13:02:47 amazon drive root 'rename-test': Reading ""
2017/01/05 13:02:47 amazon drive root 'rename-test': Finished reading ""
2017/01/05 13:02:47 Google drive root 'rename-test': Waiting for checks to finish
2017/01/05 13:02:47 Google drive root 'rename-test': Waiting for renames to finish
2017/01/05 13:02:47 file1: Couldn't find file - need to transfer
2017/01/05 13:02:48 file1: Moved (server side)
2017/01/05 13:02:48 file2: Renamed from "file1"
2017/01/05 13:02:48 Google drive root 'rename-test': Waiting for transfers to finish
2017/01/05 13:02:48 Waiting for deletions to finish
2017/01/05 13:02:48 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 2
Transferred:            0
Elapsed time:        2.2s
2017/01/05 13:02:48 Go routines at exit 12

Note the rename in the log and the 2 checks and no transfers.

20

Does this work with encrypted dirs ?