Get real transfer size and path for moved items in 'rclone sync'

Hi there!

I've currently run into a little problem when using the sync subcommand:

Let's say I have two files that changed since the last sync, one that is new (A) with a size of 10 MB, and one (B) that was just moved to another folder with a size of 20 MB.

I run rclone sync <source> <dest> --track-renames --track-renames-strategy leaf,size so (B) is correctly detected as just moved, and if actually run the command it won't be transferred ; the file will just be moved on the remote location, which is the expected behaviour.

The problem is with the displayed path and size. Using --dry-run, the displayed file's path is the old one, not the new one. So let's say I moved the file from /dir1/B to /dir2/B, the transfer path will be shown as being /dir1/B, which is not ideal.

The biggest problem I have though is that the total transfer size is calculated using the file's size. So as (B) is not going to be transferred, the total transfer size is 10 MB. But RClone actually displays a size of 30MB:

Transferred:      30M / 30 MBytes, 100%, 1.133 TBytes/s, ETA 0s

While it's going to actually transfer 10 MB over the network.

Is there a way to:

  1. Display the moved file's path instead of its old one ;
  2. Display the correct total transfer size

I'm using RClone v1.55.1 on Windows 10.

Thanks in advance for your help!

hello,

can you test the sync commands using a single file.

can you post the full debug logs.

Here is the output log when with --dry-run:

2021/07/09 14:28:39 DEBUG : Using config file from "C:\\Users\\XXXXX\\.config\\rclone\\rclone.conf"
2021/07/09 14:28:39 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone.exe" "sync" "C:/Temp/megatest" "mega_nz:/tmp" "--track-renames" "--track-renames-strategy" "leaf,size" "-vv" "--dry-run"]
2021/07/09 14:28:39 DEBUG : Creating backend with remote "C:/Temp/megatest"
2021/07/09 14:28:39 DEBUG : fs cache: renaming cache item "C:/Temp/megatest" to be canonical "//?/C:/Temp/megatest"
2021/07/09 14:28:39 DEBUG : Creating backend with remote "mega_nz:/tmp"
2021/07/09 14:28:45 DEBUG : fs cache: renaming cache item "mega_nz:/tmp" to be canonical "mega_nz:tmp"
2021/07/09 14:28:45 INFO  : mega root 'tmp': Making map for --track-renames
2021/07/09 14:28:45 INFO  : mega root 'tmp': Finished making map for --track-renames
2021/07/09 14:28:45 DEBUG : mega root 'tmp': Waiting for checks to finish
2021/07/09 14:28:45 DEBUG : mega root 'tmp': Waiting for renames to finish
2021/07/09 14:28:45 NOTICE: dir1/file1.txt: Skipped move as --dry-run is set (size 1.855M)
2021/07/09 14:28:45 INFO  : dir2/file1.txt: Renamed from "dir1/file1.txt"
2021/07/09 14:28:45 DEBUG : mega root 'tmp': Waiting for transfers to finish
2021/07/09 14:28:45 DEBUG : Waiting for deletions to finish
2021/07/09 14:28:45 INFO  : There was nothing to transfer
2021/07/09 14:28:45 NOTICE:
Transferred:        1.855M / 1.855 MBytes, 100%, 3.514 GBytes/s, ETA 0s
Checks:                 2 / 2, 100%
Renamed:                1
Elapsed time:         5.3s

2021/07/09 14:28:45 DEBUG : 8 go routines active

And with no --dry-run:

2021/07/09 14:28:16 DEBUG : Using config file from "C:\\Users\\XXXX\\.config\\rclone\\rclone.conf"
2021/07/09 14:28:16 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone.exe" "sync" "C:/Temp/megatest" "mega_nz:/tmp" "--track-renames" "--track-renames-strategy" "leaf,size" "-vv"]
2021/07/09 14:28:16 DEBUG : Creating backend with remote "C:/Temp/megatest"
2021/07/09 14:28:16 DEBUG : fs cache: renaming cache item "C:/Temp/megatest" to be canonical "//?/C:/Temp/megatest"
2021/07/09 14:28:16 DEBUG : Creating backend with remote "mega_nz:/tmp"
2021/07/09 14:28:21 DEBUG : fs cache: renaming cache item "mega_nz:/tmp" to be canonical "mega_nz:tmp"
2021/07/09 14:28:21 INFO  : mega root 'tmp': Making map for --track-renames
2021/07/09 14:28:21 INFO  : mega root 'tmp': Finished making map for --track-renames
2021/07/09 14:28:21 DEBUG : mega root 'tmp': Waiting for checks to finish
2021/07/09 14:28:21 DEBUG : mega root 'tmp': Waiting for renames to finish
2021/07/09 14:28:22 INFO  : dir1/file1.txt: Moved (server-side) to: dir2/file1.txt
2021/07/09 14:28:22 INFO  : dir2/file1.txt: Renamed from "dir1/file1.txt"
2021/07/09 14:28:22 DEBUG : mega root 'tmp': Waiting for transfers to finish
2021/07/09 14:28:22 DEBUG : Waiting for deletions to finish
2021/07/09 14:28:22 INFO  : There was nothing to transfer
2021/07/09 14:28:22 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 2 / 2, 100%
Renamed:                1
Elapsed time:         5.9s

I can give you some input on this

It is my understanding that the Transfered stats currently includes uploads, downloads and server-side copies.

The server-side copies were added as result of this forum thread:
B2 server side copy doesn't show cumulative progress - #12 by ctonsing

There is a pending issue to include server-side moves:
--stats-one-line-date does not increment transfers when doing a server-side move · Issue #5430 · rclone/rclone · GitHub

Personally, I can see many good reasons both to exclude and include server-side stats in the transfer stats. Ideally they probably ought to be split, but then on the other hand I don't want stats to be too complicated to code or understand. Still thinking...

Well I think there should be a flag for this, given that we want either one thing or the other depending on what we're trying to do.

But I still think the value is invalid, as no server-side copy happens, it's only a moving (so like a renaming) so it shouldn't be shown in the total transfer's size IMO.

Just a small clarification, server-side copy stats were not added as a result of this thread (B2 server side copy doesn't show cumulative progress - #12 by ctonsing)
As @ncw states in one of the responses (https://forum.rclone.org/t/b2-server-side-copy-doesnt-show-cumulative-progress/11154/13?u=ctonsing), "Relatively recently rclone started accounting for server side copies too", which was before I submitted the question.

I only mentioned a discrepancy in the stats which turned out to be an error in the stats accounting which was subsequently fixed.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.