I would like to dump the output of rclone ls to a file, but I would also like to see the progress. It appears that the progress is also printed to stdout, is that correct? I would have assumed that redirecting stdout to a file would give me the file listing, while the progress would still be shown on screen, is this not possible?
What are you running and what's not working?
felix@gemini:~$ rclone ls GD: --progress >> blah 2>&1
felix@gemini:~$ cat blah
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
-1 Joeisms.docx
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
278 hosts
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
0 test.tar
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
-1 TestDoc.docx
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
184 testcopy
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
306520 RDManager
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
1504953150 jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
-1 test/testsheet.xlsx
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6sTransferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
Yeah, that's the problem. I don't want the
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.6s
progress part to be redirected to the file. Only the file listing. Usually, programs will use stderr for the debug and info stuff, while only the proper output of the program will use stdout. Is there a way to get the progress to use stderr?
I ended up just not using progress, but to me this seems not proper to mix outputs.
You asked for progress, which goes to stdout and you want the output, which also goes to stdout.
I'm not sure I see anything being mixed.
Progress isn't an error though as it's output you asked for so it should be on stdout.
This issue looks relevant as well:
Progress report does not honor --syslog · Issue #4840 · rclone/rclone (github.com)
So I think what you are asking might be in the works.
You asked for progress, which goes to stdout and you want the output, which also goes to stdout.
That's what I'm saying is incorrect. Progress, errors, info, diagnostic, etc that is not the main output should NOT go to stdout. Now you're saying it does, and I am saying is "there a way" and if not "can we make a way" for the progress to go to stderr.
I'm not sure I see anything being mixed.
Seriously? You don't see the progress and the file listing, obviously two different outputs (and are separately outputted to stdout when not pipping), being intertwined in the file?
Progress isn't an error though as it's output you asked for so it should be on stdout.
No, this is not correct. stderr isn't just for errors, but for any diagnostic or logging info, such as a progress bar. A quick google will show you this is standard Unix/Linux, POSIX, and C standards. It's the whole reason stderr was invented, so you can pipe the program output without having to get a bunch of error, warning, info, diagnostic, progress, whatever into the next program or file.
Besides, using stderr correctly can still allow you to easily mix the output if you so choose.
So, rclone obviously doesn't do this. How to we move forward here? There are a few ways to deal with this:
- change it to use stderr (possible bc break if anyone is relying on it)
- provide and option to use stderr for all the aforementioned info.
- detect when output is not going to terminal (but being piped), and only then use stderr for the additional
This issue looks relevant as well:
Progress report does not honor --syslog · Issue #4840 · rclone/rclone (github.com)
So I think what you are asking might be in the works.
Interesting. ivandeex seems to agree with what I'm saying:
This is by design. Progress report is intended as a terminal (stderr) only output, optionally piped up an embedding GUI.
But that's not what we're seeing?
Maybe my first search was bad? 
or second one?
I'm not sure I agree with his statement either though, but in this case, I really have zero skin in the game and have my opinion and yours is different.
I wouldn't trust that, it's probably written by a student, or a professor that hasn't written a program in decades. It's very easy to just call it "error", but that's not the whole picture.
standard error (for writing diagnostic output)
POSIX standard
The standard error stream, which is used for error messages and diagnostics issued by the program.
GNU C Standard
And this one is a particularly good answer, quoting the inventor of Unix pipes explaining why stderr was invented:
Not sure there's much value in debating your perception of why it should be in stdout or stderr vs mine adds much value as we are both right based on how we "expect" it to work. I think you typed in -P and you want the output as you asked for it so that's the normal ouptut. You think it's diagnostic info. Both of us are right.
The pull for the stdout was done by @ncw so really ask him why he choose it and that would be the answer.
Here's a way to get the output I expect:
rclone ls 'S3:' | pv -l > files.txt
pv will pipe the output, but show the progress in the terminal.
This is also relevant:
Yes that is correct.
What you are asking for there is rclone to write the progress direct to the console, rather than to stdout. This is (for example) how some programs prompt for passwords. If rclone did that then you could redirect stdout to a file and still see the progress.
This looks to be how you do it:
Fancy having a go at implementing it?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.