Non-JSON'ed output when using --use-json-log

When using "--use-json-log" and "-q", I'm getting non-JSON'ed output where there is an error. For example:

2021/11/03 10:54:22 Failed to sync with 2 errors: last error was: failed to delete 1 files

It appears immediately after a similarly formatted JSON text:

{"level":"error","msg":"Attempt 3/3 failed with 2 errors and: failed to delete 1 files","source":"cmd/cmd.go:284","time":"2021-11-03T10:54:22.261325-04:00"}

This is borking some inline JSON processing (rclone->mqtt pub->mqtt sub via javascript) I'm doing. Is there a way to get rid of all non-JSON'ed output given that both the json messages and the "error" messages are being sent to standard error? On the JS side, I can just dump it, but I kind of prefer to just not even see the message.

thx.

jrd

sftp backend
RHEL 7.9

rclone-1.55.1-1.el7.x86_64 (via EPEL)
rclone v1.55.1-DEV

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.15.5
  • go/linking: dynamic
  • go/tags: none

I always wonder why folks hate using the template as there is stuff missing :frowning:

You have an old version, can you please update it?
Can you share the command you ran?
Can you share your rclone.conf?
Can you share a full debug log with -vv on it?

Basically, the template that was deleted.

Updating is a potentially long and involved process which is out of my control. I read through the changelog and there haven't been any JSON related fixes, nor did I see anything related to this in the forum. If this is an issue that looks familiar and has been addressed, I'd be happy to start that process.

Perhaps you didn't understand my issue. I'm not talking about the error itself. That is correct. I'm just trying to figure out if there is away around displaying non-json'ed error output when using the --use-json-log flag. If not with rclone itself, then perhaps someone might have a trick with error redirection, pipes, jq. etc. The command, rconf (not being used) and -vv really don't matter. For example, I can replicate the problem with something as simple as:

rclone --use-json-log blahblahblah -vv

Error: unknown command "blahblahblah" for "rclone"
Run 'rclone --help' for usage.
You could use 'rclone selfupdate' to get latest features.

2021/11/03 13:11:23 Fatal error: unknown command "blahblahblah" for "rclone"

None of this is JSON-formatted. In particular, the last line is the same type/style that I'm getting with my actual usage (Date Time Error).

FWIW, I also tried --log-file flag, but this just sends the non-json text to a file.

jrd

Updating involves downloading a single package, unzipping somewhere on your system and running the same command again and sharing the output.

Downloads are here: Rclone downloads as you don't have to install, you can just download, unzip, test and share output.

Correct as there isn't a command or anything I read or replicate to validate what I think you are saying. That's why we spent a lot of time to develop a help and support template to collect all the information up front that gives a complete problem description and saves time.

An example of JSON logging in use:

felix@gemini:~$ rclone copy /etc/hosts GD: --use-json-log --log-file /tmp/blah.log -v
felix@gemini:~$ cat /tmp/blah.log
{"level":"info","msg":"\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA -\nChecks:                 1 / 1, 100%\nElapsed time:         0.3s\n\n","source":"accounting/stats.go:480","stats":{"bytes":0,"checks":1,"deletedDirs":0,"deletes":0,"elapsedTime":0.305284031,"errors":0,"eta":null,"fatalError":false,"renames":0,"retryError":false,"speed":0,"totalBytes":0,"totalChecks":1,"totalTransfers":0,"transferTime":0,"transfers":0},"time":"2021-11-03T13:31:23.006846-04:00"}
felix@gemini:~$

All software used at my institution must be "risk assessed". Software that is provided through an previously approved distribution service (ie a provided RHEL repository) is exempt. Therefore were are prohibited by policy from simply downloading new/updated/whatever software without going through a formal review process. Like I said, if this issue has been addressed, then I'll start that process....but I'd be fine (and in reality this would be easier) with a standard linux o/s program/work around if someone has happened to have run across this issue before as well.

My previous response included an example command and output (rclone --use-json-log blahblahblah -vv). Do you get something different with the latest version of rclone than what I provided? That is, do you get an output which is fully json'ed? In particular, the last line. If so, I'll work on getting the software exception and upgrade it.

jrd

Sure. You can download on anything though as rclone is portable as I'm sure you have another computer, laptop, spin up a free VM in Azure, Google, VirtualBox, etc as there are a plethora of ways to test on a version to validate a situation before moving into a 'secure/production' environment.

That's not a valid command. Do you have an actual command you are trying to get working that is not?

Here is simple (more practical) example to replicate the issue (see the last line for the non-json'ed text). Basically, this creates a file and then tries to sync it while the file is still writing due to the background dd command. As expected, an error is produced which is reflected in the json'ed output. However, the last line is not JSON formatted.

dd if=/dev/zero of=file bs=1 count=10240000 &
rclone sync file file-copy --use-json-log -vv

(bunch of json formatted errors redacted)
...
{"level":"error","msg":"Failed to copy: can't copy - source file is being updated (size changed from 59075 to 63530)","object":"file","objectType":"*local.Object","source":"operations/operations.go:509","time":"2021-11-03T13:57:25.062848-04:00"}
{"level":"error","msg":"Attempt 3/3 failed with 1 errors and: can't copy - source file is being updated (size changed from 59075 to 63530)","source":"cmd/cmd.go:284","time":"2021-11-03T13:57:25.062895-04:00"}
{"level":"info","msg":"\nTransferred: \t 0 / 0 Bytes, -, 0 Bytes/s, ETA -\nErrors: 1 (retrying may help)\nElapsed time: 1.1s\n\n","source":"accounting/stats.go:417","stats":{"bytes":0,"checks":0,"deletedDirs":0,"deletes":0,"elapsedTime":1.178476401,"errors":1,"eta":null,"fatalError":false,"lastError":"can't copy - source file is being updated (size changed from 59075 to 63530)","renames":0,"retryError":true,"speed":0,"totalBytes":0,"totalChecks":0,"totalTransfers":0,"transferTime":1.141352106,"transfers":0},"time":"2021-11-03T13:57:25.063022-04:00"}
{"level":"debug","msg":"2 go routines active\n","source":"cmd/cmd.go:299","time":"2021-11-03T13:57:25.063114-04:00"}
2021/11/03 13:57:25 Failed to sync: can't copy - source file is being updated (size changed from 59075 to 63530)

Nice!

Now have a command, test case and output.

I noticed that this pull request added it in: Add a json format log by justina777 · Pull Request #3373 · rclone/rclone · GitHub

@Justina @ncw

Here is the way to recreate as it does look like a bug to me:

I did:

dd if=/dev/zero of=file bs=1 count=10240000 & rclone sync file file-copy --use-json-log -vv --log-file /tmp/blah.log

which the log looks like:

felix@gemini:~$ cat /tmp/blah.log
{"level":"debug","msg":"Version \"v1.57.0\" starting with parameters [\"rclone\" \"sync\" \"file\" \"file-copy\" \"--use-json-log\" \"-vv\" \"--log-file\" \"/tmp/blah.log\"]","object":"rclone","objectType":"string","source":"cmd/cmd.go:408","time":"2021-11-03T14:08:28.630837-04:00"}
{"level":"debug","msg":"Creating backend with remote \"file\"","source":"fs/newfs.go:28","time":"2021-11-03T14:08:28.631014-04:00"}
{"level":"debug","msg":"Using config file from \"/opt/rclone/rclone.conf\"","source":"config/config.go:353","time":"2021-11-03T14:08:28.631087-04:00"}
{"level":"debug","msg":"fs cache: adding new entry for parent of \"file\", \"/home/felix\"","source":"cache/cache.go:86","time":"2021-11-03T14:08:28.631184-04:00"}
{"level":"debug","msg":"Creating backend with remote \"file-copy\"","source":"fs/newfs.go:28","time":"2021-11-03T14:08:28.631202-04:00"}
{"level":"debug","msg":"fs cache: renaming cache item \"file-copy\" to be canonical \"/home/felix/file-copy\"","source":"cache/cache.go:79","time":"2021-11-03T14:08:28.631281-04:00"}
{"level":"debug","msg":"Need to transfer - File not found at Destination","object":"file","objectType":"*local.Object","source":"operations/operations.go:1577","time":"2021-11-03T14:08:28.63132-04:00"}
{"level":"warning","msg":"Removing partially written file on error: can't copy - source file is being updated (size changed from 11102 to 11160)","object":"file","objectType":"*local.Object","source":"local/local.go:1202","time":"2021-11-03T14:08:28.631411-04:00"}
{"level":"error","msg":"Failed to copy: can't copy - source file is being updated (size changed from 11102 to 11160)","object":"file","objectType":"*local.Object","source":"operations/operations.go:509","time":"2021-11-03T14:08:28.631556-04:00"}
{"level":"error","msg":"Attempt 1/3 failed with 1 errors and: can't copy - source file is being updated (size changed from 11102 to 11160)","source":"cmd/cmd.go:283","time":"2021-11-03T14:08:28.631589-04:00"}
{"level":"debug","msg":"Need to transfer - File not found at Destination","object":"file","objectType":"*local.Object","source":"operations/operations.go:1577","time":"2021-11-03T14:08:28.631613-04:00"}
{"level":"warning","msg":"Removing partially written file on error: can't copy - source file is being updated (size changed from 11247 to 11273)","object":"file","objectType":"*local.Object","source":"local/local.go:1202","time":"2021-11-03T14:08:28.631666-04:00"}
{"level":"error","msg":"Failed to copy: can't copy - source file is being updated (size changed from 11247 to 11273)","object":"file","objectType":"*local.Object","source":"operations/operations.go:509","time":"2021-11-03T14:08:28.631717-04:00"}
{"level":"error","msg":"Attempt 2/3 failed with 1 errors and: can't copy - source file is being updated (size changed from 11247 to 11273)","source":"cmd/cmd.go:283","time":"2021-11-03T14:08:28.631736-04:00"}
{"level":"debug","msg":"Need to transfer - File not found at Destination","object":"file","objectType":"*local.Object","source":"operations/operations.go:1577","time":"2021-11-03T14:08:28.631761-04:00"}
{"level":"warning","msg":"Removing partially written file on error: can't copy - source file is being updated (size changed from 11327 to 11354)","object":"file","objectType":"*local.Object","source":"local/local.go:1202","time":"2021-11-03T14:08:28.63181-04:00"}
{"level":"error","msg":"Failed to copy: can't copy - source file is being updated (size changed from 11327 to 11354)","object":"file","objectType":"*local.Object","source":"operations/operations.go:509","time":"2021-11-03T14:08:28.63185-04:00"}
{"level":"error","msg":"Attempt 3/3 failed with 1 errors and: can't copy - source file is being updated (size changed from 11327 to 11354)","source":"cmd/cmd.go:283","time":"2021-11-03T14:08:28.63187-04:00"}
{"level":"info","msg":"\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA -\nErrors:                 1 (retrying may help)\nElapsed time:         0.0s\n\n","source":"accounting/stats.go:480","stats":{"bytes":0,"checks":0,"deletedDirs":0,"deletes":0,"elapsedTime":0.0133494,"errors":1,"eta":null,"fatalError":false,"lastError":"can't copy - source file is being updated (size changed from 11327 to 11354)","renames":0,"retryError":true,"speed":0,"totalBytes":0,"totalChecks":0,"totalTransfers":0,"transferTime":0,"transfers":0},"time":"2021-11-03T14:08:28.631905-04:00"}
{"level":"debug","msg":"3 go routines active\n","source":"cmd/cmd.go:298","time":"2021-11-03T14:08:28.631941-04:00"}
2021/11/03 14:08:28 Failed to sync: can't copy - source file is being updated (size changed from 11327 to 11354)
felix@gemini:~$

Glad to know it's not just me!

Related to this, I notice that the error code that is being returned is 1. (ie add "echo $?" to the example) This doesn't seem to be the correct number. However, I see in Wrong Exit code that the issue of incorrect error codes has been brought up before. Has this particular error's code (perhaps it should be 6?) been fixed in the latest version of rclone?

Also, FWIW, I just figured out a way around this issue for my pipeline by first validating json with jq:

rclone blah (json sent to std err) 2>&1 | jq -c 2> /dev/null | (remainder of pipeline)

First I convert rclone json std error to std output and then pipe the json through jq. jq uses std output and the bad line goes to std error which I can then divert to /dev/null

Yes I think this is a bug!

@JRDavisUF can you open a new issue on Github please and we'll fix it :slight_smile:

It shouldn't be very hard to fix (hopefully!)

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.