Measure rclone process network use on linux

kb.claim · June 13, 2024, 7:46pm

What is the problem you are having with rclone?

not really a problem, i would like to measure the total network/internet usage of my weekly rclone -sync task.

I tried several tools, top, nethogs, iftop, and googled a lot, but cannot find any way to record the total traffic rclone creates for a complete sync run. I assume rclone needs to transfer data beetween the remote and local at least to compare the data, so the statistics output for "Transferred:" is understood as the new/changed data, but this is assumed not to contain additional network bandwidth for the rclone checks, etc.

I have around 300 GB of data, synced weekly to googleDrive and Mega clouds, and a sync transfers around 2GB. But i want to find out what the total network/internet usage is as I sit sometimes with a 5G SIM connection to the net and the network usage counts.

Any hint on a tool that can measure the send / received of rclone during the sync would be excellent.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

os/version: ubuntu 24.04 (64 bit)
os/kernel: 6.8.0-35-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.22.1
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

googleDrive and Mega

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync localDir encrypted-union-remote: --progress --links --create-empty-src-dirs --delete-during --delete-excluded --human-readable --log-file=~/log_it_baby.log

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

-- can be sent on request --

A log from the command that you were trying to run with the `-vv` flag

-- if required --

kapitainsky · June 13, 2024, 7:53pm

This is not really something you can get from rclone itself - I do not think anybody planned for it.

Look at your OS specific tools - for Linux try for example nethogs or dstat. Both allow to measure cumulative network traffic for specific process.

kb.claim · June 13, 2024, 8:01pm

Thanks, as mentioned, i tried several tools, but failed. nethogs sums up the bandwith for an interval, but does not record the sum total. I never asked for this to be done by rclone, a tool that can do that would be my goal to find.

kapitainsky · June 13, 2024, 8:21pm

There is other very simple way:) I am sure your mobile device records your traffic - if not device then your operator for billing purposes (otherwise I guess you would not worry about few GB on 5G network).

Make sure nothing else is using network. Get initial stats. run rclone sync. Get stats. Do math. Done.

kb.claim · June 15, 2024, 11:08am

thanks for trying. I seek a TOOL to record incoming and outgoing traffic of the rclone process on ubuntu and sum up the totals. if you know such a tool welcome. thanks.

kapitainsky · June 15, 2024, 11:25am

Try on ubuntu forums. This is general linux problem.

kb.claim · June 15, 2024, 1:15pm

thanks, have you ever heard that your comments might be pretty useless and embarrassing? I am sure if you would sit on the "ubuntu forums" you refer to you would write "this is a rclone problem, try on rclone forums". Does not make sense sir. Get a grip on what you write!

kapitainsky · June 15, 2024, 1:24pm

It is not rclone problem. You are asking how to measure network interface traffic for specific process on your OS:) If you ask your questions in the right place you might have some meaningful answers.

kb.claim · June 15, 2024, 1:43pm

well, thanks fro your opinion. my view differs, as neither the forum nor the question are the problem, but your repeated empty response.

kb.claim · June 15, 2024, 3:20pm

I have found the solution now.

One needs 2 shells / terminals.

Terminal 1:
start rclone: rclone sync localDir remote: -your flags and parameters

Terminal 2:
start nethogs: sudo nethogs -P pgrep rclone -v 3

pgrep gets the pid of the rclone process (if you have more rclone processes running you need another way to get the pids and feed it to nethogs via -P)

-v 3 tells nethogs to sum the total (this is what i missed). The default is -v 0 which lists the bandwidth.

After the rclone job terminates you can see in terminal 1 how much files were updated (the "Transferred" data), and in terminal 2 you see the real network usage, e.g. "Transferred" data + rclone protocol overheads.

So far as it should be, proof to be made, and is already running.

Dual-O · June 15, 2024, 6:40pm

look at my post hope it helps you.

kb.claim · June 16, 2024, 4:36am

thanks, thats very impressive - great job!

kb.claim · June 17, 2024, 7:36am

First run's results - 262.667 GiB archive on a googleDrive encrypted union, synchronised against the local folder.

982 MiB were changed and thus transferred from local to googleDrive.

nethogs recorded 3.151 GB sent and 2.16 GB received, so i assume one sync round takes ~5.5 GB (i am not sure how billing/accounting is done at the network provider, whether its the sum of up/download).

gd_stat