Separate error message from console output

[tehsphinx]

According to the test runner interface there is a distinction to be made between the Message (error message in case the test failed) and Output (the output of the test including user output).

When parsing the output of go test --json we currently write all lines marked with "Action":"output" into the Output field. Instead we should additionally check the "Output" if it has the following pattern and write that in the Message field.

Sample of line to be written to the Message field:

{"Time":"2020-01-03T14:05:53.978508+01:00","Action":"output","Package":"github.com/exercism/go-test-runner/tests/5","Test":"TestSample/first_test","Output":"        sample_test.go:30: Output should be output: first test, got output: third test\n"}

Suggestion is to use a regex to search for the prefix ^\t+(.+?)_test\.go:/d+: (or similar) and use these line(s) to fill Message in the test runner output.

[junedev]

I think Output and Message are mixed up in some places in the description above or the logic was already changed since that was written.

Currently we write everything into message in the the final JSON file.
I will list the three possible types that can appear here with an example from the JSON that the test run produces:

1. The test run report
   {"Time":"...","Action":"output","Package":"gigasecond","Test":"TestTrivialPass1","Output":"--- PASS: TestTrivialPass1 (0.00s)\n"}
2. the error message from the t.Fatal or t.Error calls
   {"Time":"...","Action":"output","Package":"gigasecond","Test":"TestTrivialPass2","Output":" passing_test.go:31: This assertion failed ...\n"}
3. The result of fmt.Println statements that the user added to the solution code for debugging
   {"Time":"...","Action":"output","Package":"gigasecond","Test":"TestTrivialPass2","Output":"I am debugging something\n"}

The test runner docs state

The per-test output key should be used to store and output anything that a user deliberately outputs for a test.

That means, what we need to separate from the rest is point 3 above. However, 3 has no criteria by which we can identify it. That means the only chance we have is to identify 1 and 2 and categorize everything else as 3.

2 can be identified as suggested above by @tehsphinx.
I am not sure about 1. It can start with === RUN, --- PASS or with --- FAIL (after trimming whitespace) but are these all the possible options? So far, I couldn't find any official documentation on this.

[andrerfcsantos]

However, 3 has no criteria by which we can identify it.

If we detect that is not 1 or 2, maybe we can assume it's 3?

[andrerfcsantos]

It can start with === RUN, --- PASS or with --- FAIL (after trimming whitespace) but are these all the possible options?

There's also --- SKIP:

{"Time":"2022-02-07T20:35:20.3824835Z","Action":"output","Package":"sorting","Test":"TestSkip","Output":"--- SKIP: TestSkip (0.00s)\n"}

[junedev]

If we detect that is not 1 or 2, maybe we can assume it's 3

@andrerfcsantos Yes, exactly, that is what I meant with the next sentence.

That means the only chance we have is to identify 1 and 2 and categorize everything else as 3.

[junedev]

Re 1: It turns out there are more. I found this list in the source code for test2json:
https://github.com/golang/go/blob/2ebe77a2fda1ee9ff6fd9a3e08933ad1ebaea039/src/cmd/internal/test2json/test2json.go#L150-L161
Using that list is probably a good bet.