Rendering differences of Skipped Scenarios in v24

[clrudolphi]

👓 What did you see?

The consumption chain is Reqnroll v3.3.1 -> HtmlFormatter v22.2 -> ReactComponents v24.

When a feature contains multiple scenarios and when one or more of those scenarios were skipped, the rendering is now different from the prior releases.

In the prior release, the skipped scenarios would be noted as Skipped:

With ReactComponents v24, the skipped scenarios are noted as Skipped AND now include the exception detail that caused the scenario to be skipped:

✅ What did you expect to see?

In Reqnroll, exceptions are used to indicate to the test runner that tests have failed and when skipped. So it is expected that they would be present. I have confirmed that the ndjson streams are structurally identical between these two renderings.

Under the previous version, when a step was marked as Skipped, the rendering would not include any exception detail.

Is this change in rendering behavior intentional?

📦 Which tool/library version are you using?

.NET Framework 4.8
Reqnroll 3.3.1
HtmlFormatter v22.2
ReactComponents v24.0

🔬 How could we reproduce it?

See reproduction project here

📚 Any additional context?

No response

[mpkorstanje]

Under the previous version, when a step was marked as Skipped, the rendering would not include any exception detail.
Is this change in rendering behavior intentional?

Yes. That is expected. Instead of rendering a facimily of the feature file, we switched to testcase centric rendering in #396. This splits out examples, and also hooks. This was a bit noisy so we hid successful hooks (#414, #415, and very specifically #415 (comment))

In Reqnroll, exceptions are used to indicate to the test runner that tests have failed and when skipped. So it is expected that they would be present. I have confirmed that the ndjson streams are structurally identical between these two renderings.

Using exceptions to abort a scenario is supported. So I think we'll have to come up with a solution that makes this less cluttered. Perhaps we can collapse the body of the first skipped hook but still allow it to be expanded to view the reason for skipping?

[mpkorstanje]

@davidjgoss it also looks we're also not showing the regular steps after a skipped hook. That doesn't look correct.

[clrudolphi]

Perhaps we can collapse the body of the first skipped hook but still allow it to be expanded to view the reason for skipping?

Yes, I think that would be acceptable.

[SNBogdanov]

So the question is:
How to produce clean and readable report with

• Fully completed scenarios
• Failed on step in the middle scenarios and skipped beacuse of that steps
• Scenarios with skip information in attributes
• Scenarios with skip assert in before scenario hook

Ideally I want to see every step with distingushed statuses on scenarios and each step in scenario

[clrudolphi]

• Scenarios with skip information in attributes

What do you mean by 'skip information in attributes'?

[clrudolphi]

it also looks we're also not showing the regular steps after a skipped hook. That doesn't look correct.

If you are commenting on the html rendering above and what is shown for Scenario 2, that is an artifact of how Reqnroll operates. When a Before hook causes a skip, execution halts for that scenario at that point, and thus the ndjson stream does not include any StepStarted/StepFinished for that scenario (beyond the aborting hook).

[SNBogdanov]

• Scenarios with skip information in attributes

What do you mean by 'skip information in attributes'?

Ignore attribute

[clrudolphi]

• Scenarios with skip information in attributes

What do you mean by 'skip information in attributes'?

Ignore attribute

Do you mean when a Feature or Scenario uses an '@ignore' tag? In that situation, Reqnroll will generate code that includes the test-framework appropriate mechanism for skipping that item (usually an [Ignore] attribute on the generated test method). When that happens the test framework does not execute the test, and therefore there is no record of its execution (or lack of execution). There is nothing that we can render in the HTML report.

[SNBogdanov]

• Scenarios with skip information in attributes

What do you mean by 'skip information in attributes'?

Ignore attribute

Do you mean when a Feature or Scenario uses an '@ignore' tag? In that situation, Reqnroll will generate code that includes the test-framework appropriate mechanism for skipping that item (usually an [Ignore] attribute on the generated test method). When that happens the test framework does not execute the test, and therefore there is no record of its execution (or lack of execution). There is nothing that we can render in the HTML report.

That I understand, and I don't expect solution for this.

[luke-hill]

I noticed a difference in cucumber-ruby (Sorry if polluting here). Whereby ones that are skipped by the executor (i.e. exclusion on tag filters).

In previous versions of react components, it would show the scenario - but greyed out.
Now it doesn't show anything.

For ☝️ I'm not that fussed though. But it was "nicer" imo before when i could see a greyed out set of steps because I knew I needed to do something (Enable the scenario)

[mpkorstanje]

@luke-hill without knowing which versions you are comparing it's hard to say if this related. Sounds like you're talking about the switch to the test-case based view instead of the feature file based view. Eitherway a separate issue would be better. We can close duplicates, but we can't unpick off topic threads.