Bugs with --progress transferred bytes and speed

Suspected Bug
1

What is the problem you are having with rclone?

I've been on rclone 1.57.0 because of 2 bugs with --progress. But i'm required to upgrade for filename_encoding base32768 since i'm moving from gdrive to dropbox.

  1. Renamed or moved files are counted as transferred bytes. You can see this issue in the attached screenshot. I've limited rclone to 3Mbytes upload so this speed and amount of transferred data is technically impossible. (Tested on 1.63.0)
    image

  2. The transfer speed in --progress get stuck on a couple kbytes after it starts the retry if errors occurred. Individual file transfer speed are still working properly after the retry starts. (The last time i tested this issue was with 1.62.0. I haven't tested it yet on 1.63.0) (might be the same as Speed and ETA are incorrect after a rescan · Issue #7033 · rclone/rclone · GitHub)

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.3086 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.5
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe sync --bwlimit 3M:off --fast-list --track-renames --track-renames-strategy modtime --check-first --delete-after --delete-excluded --exclude-from "Exclusions.txt" --backup-dir cryptGDrive:Archive/Union --progress --log-file "Rclone Mirror_Union.log" --log-level INFO local:E:/Union/ cryptGDrive:Union

The rclone config contents with secrets removed.

[local]
type = local
encoding = Slash,InvalidUtf8
copy_links = true

[remoteGDrive]
type = drive
client_id = ***
client_secret = ***
scope = drive
root_folder_id = ***
service_account_file = 
token = ***
encoding = Slash,InvalidUtf8

[cryptGDrive]
type = crypt
remote = remoteGDrive:Crypt
filename_encryption = standard
directory_name_encryption = true
password = ***
password2 = ***

A log from the command with the -vv flag

n/a

2

Rclone does indeed count server side moves and copies as transfers. This was through popular demand - it didn't used to.

That could be the same issue. Can you add your findings to that issue please. In particular try to answer the last question I asked on that issue - thanks.

3

Popular demand? Lacks any logic for me. So that means it will not be changed?

1 Like
4

I agree with you - why anybody would like to have some phantom transfers included in very clear indicator - bytes transferred and speed? I want to see what is happening on my NIC. Not some imaginary numbers. Any metrics provided should have some practical meaning - what meaning it has when server side operations are included?

At the same time please do not change anything in format of transfers metrics - many people scripts parse them using regexp and any change break a lot of carefully written programs.

5

On most backends server side copy (if available) is quite a time intensive operation so having progress stats for it is useful.

So I think keeping these stats for server side copy is useful.

Here is where the stats for server side move got added (by @Ole) so people could see the progress of large server side moves better

In the thread there are suggestions for some more sophisticated stats - maybe we need that!

Or we could add a flag...

6

I can understand that it would need stats. But its still not transferred.
Transferred is upload or download.

My stats are quite skewed at the moment because of this. At maximum i am able to upload ~250GB a day (average speed 3mbytes/s). But because of these serverside moves being counted, i've got stats like this (this one had one retry because of an error) (it also contains an example of the transfer speed being stuck because of the error)

Transferred:   	  425.212 GiB / 425.212 GiB, 100%, 61.621 KiB/s, ETA 0s
Checks:            308234 / 308234, 100%
Deleted:               51 (files), 9 (dirs)
Renamed:               58
Transferred:           78 / 78, 100%
Elapsed time:  21h37m30.7s

This is when there is nothing to do

Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:            154097 / 154097, 100%
Elapsed time:      3m44.1s

I do quite a bit of serverside moves. But this makes the stats practically useless.
Are you able to tell how much data was actually uploaded? I can make estimates but the stats won't tell me.
maybe add an extra line with "Moved bytes" or something like that. I find the Renamed entry enough for my use though.

7

It really depends on your source and destination.

If the read the PR there, they are moving from one GD to another GD so it's happening server side but it was definitely moved from one to another so that's definitely a transfer.

In your use case, maybe not. Both can be right.

8

I've read it yes.
Can you measure the speed per second in that situation? Correct me if I'm wrong but i don't think it's the case.
I believe the real issue is that the single line stats do not mention how many renames are performed.

9

Can I personally? Probably not but data is definitely downloading from one and to another. Just because I can't measure it does not mean it isn't happening.

That's why I said you are "BOTH" right.

Feel free to submit a PR, let other folks discuss it and figure out what you'd want to do. It's open source :slight_smile:

10

I meant can the program measure the speed and remaining time for a rename.

But thank you for your time. I know enough. It's not a bug and rather a deliberate decision that has been made.

11

We could make a flag to turn off this feature? Maybe --stats-ignore-server-side to not put server side moves and copies into the stats?

1 Like
12

I would really appreciate it.

13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.