[Bug discussion before issue] dedupe allows -P but is incompatible with it (graphical issue?)

Attn: @ncw , you probably want to have a look at this

Hey all,

Recently started noticing dedupe looked like it was getting stuck for no reason and not even giving error output with -v . After some troubleshooting I figured out it was a problem with using -P (progress indicator).

What seems to happen is that when dedupe (in interactive mode) pauses to ask the user for input -P will overwrite most of the question and "menu", thus basically breaking it and halting indefinitely for no apparent reason.

It looks something like this:

C:\rclone>rclone dedupe TD1: -vv --tpslimit 5 -P
2019/08/01 16:46:44 DEBUG : rclone: Version "v1.48.0-082-ga35aa136-beta" starting with parameters ["rclone" "dedupe" "TD1:" "-vv" "--tpslimit" "5" "-P"]
2019/08/01 16:46:44 DEBUG : Using config file from "C:\rclone\rclone.conf"
2019/08/01 16:46:44 INFO : Starting HTTP transaction limiter: max 5 transactions/s with burst 1
2019-08-01 16:46:44 INFO : Google drive root '': Looking for duplicates using interactive mode.
2019-08-01 16:47:03 NOTICE: Crypt1/nlc28rgi1rdqf3mf8b7h5ut314a5kejs41j9usivvcdv0iib4rb0: Found 2 duplicates - deleting identical copies
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 18.5sCrypt1/nlc28rgi1rdqf3mf8b7h5ut314a5kejs41j9usivvcdv0iib4rb0: 2 duplicates remain
1: 2376763893 bytes, 2019-07-31 15:02:11.802000000, MD5 cfbd654ba0d868ede3d6389ad13aa4c5
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -

Note that I am on Windows10, so it's not impossible that Linux output is slightly different and would not have an identical problem. It would be really nice if someone on Linux could test and see if they see the same thing there (you probably will need to have an actual duplicate to clean up to see the issue though).

If the same problem happens on Linux then the solution is probably to just ignore -P when interactive mode is used on dedupe.

If this is a windows specific issue then I'm not sure what the best fix is, except to maybe look into if -P can be modified to work better on this OS. If not (or if it's just not worth the time to do so) then I assume it's probably possible ignore -P with dedupe in interactive mode only for the windows version.

Ultimately this is not such a big problem since it's easy to avoid the whole issue, but it is a bit of a user-trap to allow this to happen as it will cause a lot of confusion and necessary frustration for less technical users.

I will create an issue for this once i have confirmation regarding how it looks on Linux.

It looks quite similar

$ rclone -P dedupe drive:test/dupes/dupe1
2019-08-02 15:36:34 NOTICE: two.txt: Found 2 duplicates - deleting identical copies
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:          0stwo.txt: 2 duplicates remain
  1:      6048320 bytes, 2019-08-02 15:36:05.451000000, MD5 1eedaa9fe86fd4b8632e2ac549403b36
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:          0s^C

Probably what we should do is disable --progress if any of the functions to read input get called - that would be fairly issue.

Please make a new issue on github about this :slight_smile:

Copy that. I will try to get this up on github shortly.

1 Like