Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,16 @@ jobs:
run: |
uv run behave features/ -f modern -o reports/e2e-report.html
# Module-scoped Cucumber JSON for SpecTracer dogfood
uv run behave features/linking.feature --tags=-@FC-004 -f json -o reports/e2e-linker.json
uv run behave features/edge_cases.feature --tags=-@FC-EDGE-006 -f json -o reports/e2e-linker-edge.json
uv run behave features/linking.feature --tags=@FC-004 -f json -o reports/e2e-collectors.json
uv run behave features/linking.feature --tags="not @scenario:FC-004" -f json -o reports/e2e-linker.json
uv run behave features/edge_cases.feature --tags="not @scenario:FC-EDGE-006" -f json -o reports/e2e-linker-edge.json
uv run behave features/linking.feature --tags=@scenario:FC-004 -f json -o reports/e2e-collectors.json
uv run behave features/health.feature -f json -o reports/e2e-aggregator.json
uv run behave features/internals.feature --tags=@FC-008 -f json -o reports/e2e-aggregator-internals.json
uv run behave features/internals.feature --tags=@scenario:FC-008 -f json -o reports/e2e-aggregator-internals.json
uv run behave features/dashboard.feature -f json -o reports/e2e-renderers.json
uv run behave features/internals.feature --tags=@FC-009 -f json -o reports/e2e-renderers-internals.json
uv run behave features/internals.feature --tags=@scenario:FC-009 -f json -o reports/e2e-renderers-internals.json
uv run behave features/module_scope.feature -f json -o reports/e2e-parsers.json
uv run behave features/edge_cases.feature --tags=@FC-EDGE-006 -f json -o reports/e2e-parsers-edge.json
uv run behave features/internals.feature --tags=@FC-007 -f json -o reports/e2e-parsers-internals.json
uv run behave features/edge_cases.feature --tags=@scenario:FC-EDGE-006 -f json -o reports/e2e-parsers-edge.json
uv run behave features/internals.feature --tags=@scenario:FC-007 -f json -o reports/e2e-parsers-internals.json
uv run behave features/json_output.feature -f json -o reports/e2e-report-model.json

- name: Generate SpecTracer report
Expand Down
24 changes: 13 additions & 11 deletions docs/architecture.html
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ <h4>parsers.py</h4>
<div class="feature-card">
<div class="feature-icon">&#128279;</div>
<h4>linker.py</h4>
<p>OR-based tag matching. Links scenarios to test results by shared tags. 17 lines — the simplest module.</p>
<p>Identity-based pairing. Matches <code>@scenario:VALUE</code> on test results against <code>@id:VALUE</code> on scenarios. 17 lines — the simplest module.</p>
</div>
<div class="feature-card">
<div class="feature-icon">&#128202;</div>
Expand Down Expand Up @@ -134,7 +134,7 @@ <h3>FeatureParser</h3>
<ul>
<li><code>Feature:</code> lines for the feature name</li>
<li>Scenario names (line following <code>Scenario:</code>)</li>
<li>Tags on lines above scenarios (distinguishes <span class="tag tag-link">linking tags</span> from <span class="tag tag-require">@require-*</span> tags)</li>
<li>Tags on lines above scenarios (identity, classification, and <span class="tag tag-require">@require-*</span> layer requirement tags)</li>
<li>Given/When/Then steps for display in the report</li>
</ul>
<p><strong>Default behavior:</strong> Scenarios without any <code>@require-*</code> tags default to <code>@require-e2e</code>.</p>
Expand Down Expand Up @@ -163,21 +163,23 @@ <h3>CucumberParser</h3>
</div>

<h2 id="linker">Linker</h2>
<p>The linker is the simplest but most important module. It performs <strong>OR-based tag matching</strong>:</p>
<p>The linker is the simplest but most important module. It performs <strong>identity-based pairing</strong> using a two-tag convention:</p>
<ul>
<li>For each scenario, find all test results where <strong>any</strong> tag appears in the scenario's linking tags</li>
<li><code>@require-*</code> tags are stripped before linking &mdash; they never participate</li>
<li>Tag matching is <strong>exact string equality</strong> only &mdash; no wildcards, no expressions</li>
<li>One test result can link to multiple scenarios (tag collision)</li>
<li>Scenarios carry <span class="tag tag-link">@id:VALUE</span> (e.g. <code>@id:FC-42</code>) as their stable identity</li>
<li>Test results carry <span class="tag tag-link">@scenario:VALUE</span> (e.g. <code>@scenario:FC-42</code>) to declare which scenario they satisfy</li>
<li>Linking matches <strong>only</strong> <code>@scenario:</code> values from results against <code>@id:</code> values from scenarios &mdash; no accidental matches from other tags</li>
<li><code>@require-*</code> tags never participate in linking</li>
<li>One test result can link to multiple scenarios (same <code>@scenario:</code> value matching multiple <code>@id:</code> values)</li>
</ul>

<div class="code-header"><span>Linking logic (abridged)</span></div>
<pre><code><span class="hl-kw">def</span> <span class="hl-fn">link</span>(scenarios, results):
links = {}
<span class="hl-kw">for</span> scenario <span class="hl-kw">in</span> scenarios:
ids = {t[<span class="hl-num">3</span>:] <span class="hl-kw">for</span> t <span class="hl-kw">in</span> scenario.tags <span class="hl-kw">if</span> t.<span class="hl-fn">startswith</span>(<span class="hl-str">"@id:"</span>)}
linked = [
r <span class="hl-kw">for</span> r <span class="hl-kw">in</span> results
<span class="hl-kw">if</span> <span class="hl-fn">any</span>(tag <span class="hl-kw">in</span> scenario.tags <span class="hl-kw">for</span> tag <span class="hl-kw">in</span> r.tags)
<span class="hl-kw">if</span> <span class="hl-fn">any</span>(t.<span class="hl-fn">startswith</span>(<span class="hl-str">"@scenario:"</span>) <span class="hl-kw">and</span> t[<span class="hl-num">10</span>:] <span class="hl-kw">in</span> ids <span class="hl-kw">for</span> t <span class="hl-kw">in</span> r.tags)
]
links[<span class="hl-fn">id</span>(scenario)] = linked
<span class="hl-kw">return</span> links</code></pre>
Expand Down Expand Up @@ -232,7 +234,7 @@ <h2 id="data-flow">Data Flow</h2>
<li><strong>Config loading</strong> &mdash; <code>cli.py</code> discovers and reads the JSON config file. Validates required keys (<code>features</code>, <code>output</code>). <code>unit</code>, <code>integration</code>, and <code>e2e</code> are module-keyed objects (<code>""</code> = unscoped).</li>
<li><strong>File collection</strong> &mdash; <code>FileCollector</code> (<code>collectors.py</code>) resolves literal file paths and recursively searches directories for <code>.feature</code>, <code>.xml</code>, and <code>.json</code> files, once per module key.</li>
<li><strong>Parsing</strong> &mdash; Each file type is parsed by its dedicated parser. Scenarios and test results are extracted into typed data classes; results carry the module key they were collected under.</li>
<li><strong>Linking</strong> &mdash; <code>ResultLinker</code> cross-references scenario tags against test result tags. OR logic: any shared tag creates a link.</li>
<li><strong>Linking</strong> &mdash; <code>ResultLinker</code> matches <code>@scenario:VALUE</code> on test results against <code>@id:VALUE</code> on scenarios. Only explicit identity pairs create a link.</li>
<li><strong>Aggregation</strong> &mdash; <code>ReportAggregator</code> computes all stats, views, breakdowns, and health checks from the linked data.</li>
<li><strong>Rendering</strong> &mdash; <code>HtmlRenderer</code> feeds the aggregated data into a Jinja2 template and writes the HTML output file.</li>
<li><strong>JSON output (optional)</strong> &mdash; if <code>output_json</code> is set, <code>report_model.build_report()</code> reshapes the same aggregated data into the schema-conformant JSON file.</li>
Expand All @@ -243,7 +245,7 @@ <h2 id="tag-resolution">Tag Resolution Order</h2>
<ol>
<li>All tags on lines immediately above a <code>Scenario:</code> line are collected</li>
<li>Tags matching the pattern <code>@require-(unit|integration|e2e)(:module)?</code> are classified as <strong>layer requirements</strong> (optional <code>:module</code> suffix for all three layers)</li>
<li>All remaining tags become <strong>linking tags</strong></li>
<li>All remaining tags are retained as-is (including <code>@id:VALUE</code> for scenario identity and any classification tags like <code>@smoke</code>)</li>
<li>If zero requirement tags were found, the scenario defaults to bare <code>@require-e2e</code> (unscoped)</li>
<li>A module-scoped requirement is only satisfied by a linked result whose <code>module</code> matches that key; unscoped results never satisfy it</li>
</ol>
Expand Down Expand Up @@ -274,7 +276,7 @@ <h2 id="testing">Testing Strategy (Dogfooding)</h2>
uv run pytest tests/integration --junitxml=reports/int.xml

<span class="hl-com"># E2E is split by module so @require-e2e:&lt;module&gt; can be dogfooded</span>
uv run behave features/coverage_linking.feature --tags=-@FC-004 -f json -o reports/e2e-linker.json
uv run behave features/coverage_linking.feature --tags="not @scenario:FC-004" -f json -o reports/e2e-linker.json
uv run behave features/coverage_health.feature -f json -o reports/e2e-aggregator.json
uv run behave features/coverage_dashboard.feature -f json -o reports/e2e-renderers.json
uv run behave features/coverage_module_scope.feature -f json -o reports/e2e-parsers.json
Expand Down
11 changes: 6 additions & 5 deletions docs/ci-cd.html
Original file line number Diff line number Diff line change
Expand Up @@ -260,8 +260,8 @@ <h2 id="team-workflows">Team Workflows</h2>
<div class="feature-block">
<h3>&#127891; Onboarding a New Project</h3>
<ol>
<li><strong>Write feature files</strong> &mdash; your team's QA or product owner creates <code>.feature</code> files describing every business scenario. Each scenario gets a unique tag like <span class="tag tag-link">@FC-001</span>.</li>
<li><strong>Add tags to tests</strong> &mdash; developers add the matching tags to their unit/integration/E2E tests. A quick PR review ensures tag naming consistency.</li>
<li><strong>Write feature files</strong> &mdash; your team's QA or product owner creates <code>.feature</code> files describing every business scenario. Each scenario gets an identity tag like <span class="tag tag-link">@id:FC-001</span>.</li>
<li><strong>Add tags to tests</strong> &mdash; developers add <span class="tag tag-link">@scenario:FC-001</span> to their unit/integration/E2E tests. A quick PR review ensures tag naming consistency.</li>
<li><strong>Set up the config</strong> &mdash; create <code>spectracer.config.json</code> pointing at your feature files and test outputs.</li>
<li><strong>Add to CI</strong> &mdash; add the SpecTracer step to your pipeline. The report becomes part of every build.</li>
</ol>
Expand All @@ -284,10 +284,11 @@ <h3>&#128220; Tag Naming Convention</h3>
<tr><th>Pattern</th><th>Example</th><th>Use</th></tr>
</thead>
<tbody>
<tr><td><code>@FC-&lt;N&gt;</code></td><td><code>@FC-42</code></td><td>Feature-case ID — unique per scenario (linking)</td></tr>
<tr><td><code>@id:&lt;VALUE&gt;</code></td><td><code>@id:FC-42</code></td><td>Scenario identity — placed on Gherkin scenarios (used for linking)</td></tr>
<tr><td><code>@scenario:&lt;VALUE&gt;</code></td><td><code>@scenario:FC-42</code></td><td>Test reference — placed on test results (used for linking)</td></tr>
<tr><td><code>@require-&lt;layer&gt;</code></td><td><code>@require-unit</code>, <code>@require-e2e</code></td><td>Required layer coverage (not used for linking)</td></tr>
<tr><td><code>@require-&lt;layer&gt;:&lt;module&gt;</code></td><td><code>@require-e2e:checkout</code></td><td>Required layer + module key from config</td></tr>
<tr><td><code>@&lt;type&gt;</code></td><td><code>@regression</code>, <code>@smoke</code></td><td>Optional linking / classification tags</td></tr>
<tr><td><code>@&lt;type&gt;</code></td><td><code>@regression</code>, <code>@smoke</code></td><td>Optional classification tags (ignored during linking)</td></tr>
</tbody>
</table>
</div>
Expand Down Expand Up @@ -328,7 +329,7 @@ <h2 id="standup">The Standup Metric</h2>
<p>SpecTracer's headline metric — <strong>Tested: X / Y scenarios (Z%)</strong> — is designed for daily standup discussion. Unlike code coverage (which measures lines executed), this metric measures <strong>business scenario coverage</strong>:</p>
<ul>
<li>It's <strong>meaningful to product managers</strong> — "we haven't tested the password reset flow yet"</li>
<li>It's <strong>actionable for developers</strong> — "scenario @FC-43 needs a unit test"</li>
<li>It's <strong>actionable for developers</strong> — "scenario @id:FC-43 needs a unit test"</li>
<li>It's <strong>visible to leadership</strong> — a single number that reflects the team's testing health</li>
</ul>
<p>Teams that use SpecTracer find themselves discussing coverage in standup naturally: "Our coverage dropped from 85% to 78% this sprint &mdash; let's add tests for the new billing scenarios before the sprint ends."</p>
Expand Down
29 changes: 18 additions & 11 deletions docs/features.html
Original file line number Diff line number Diff line change
Expand Up @@ -165,12 +165,12 @@ <h3>&#9888; What You See</h3>
</div>

<h2 id="unlinked">Unlinked Tests</h2>
<p>Test results whose tags didn't match any scenario end up here — regardless of tag prefix. Any tagged result that fails to link to a scenario is reported. Results with no tags at all are excluded (they can never participate in tag-based linking).</p>
<p>Test results whose <code>@scenario:VALUE</code> tag doesn't match any scenario's <code>@id:VALUE</code> end up here. Results without a <code>@scenario:</code> tag are excluded (they can never participate in identity-based linking).</p>

<p>This is invaluable for catching:</p>
<ul>
<li><strong>Orphaned tests</strong> — tests that were written for a scenario that no longer exists</li>
<li><strong>Mis-tagged tests</strong> — tests with typos in their tags (<code>@FC-42</code> vs <code>@FC-43</code>)</li>
<li><strong>Mis-tagged tests</strong> — tests with typos in their tags (<code>@scenario:FC-42</code> vs <code>@scenario:FC-43</code>)</li>
<li><strong>Coverage gaps</strong> — areas where test results exist but no feature file scenario describes them</li>
</ul>

Expand Down Expand Up @@ -232,24 +232,29 @@ <h4>End to end Runtime</h4>
<div class="feature-card">
<div class="feature-icon">&#9872;</div>
<h4>Unlinked Tests</h4>
<p>Orphaned test results with any tags that didn't match any scenario. Pass at 0, warn at 1&ndash;3, fail at 4+.</p>
<p>Test results with <code>@scenario:VALUE</code> that matched no scenario's <code>@id:VALUE</code>. Pass at 0, warn at 1&ndash;3, fail at 4+.</p>
</div>
</div>

<h2 id="tag-linking">Tag Linking System</h2>
<p>The heart of SpecTracer. Feature files and test results connect through <strong>shared tags</strong> on scenarios.</p>
<p>The heart of SpecTracer. Feature files and test results connect through an <strong>explicit identity convention</strong> using two paired tag prefixes.</p>

<div class="feature-block">
<h3>Two Kinds of Tags</h3>
<h3>Three Kinds of Tags</h3>
<table>
<thead>
<tr><th>Type</th><th>Examples</th><th>Purpose</th></tr>
</thead>
<tbody>
<tr>
<td><span class="tag tag-link">Linking</span></td>
<td><code>@FC-42</code>, <code>@regression</code></td>
<td>Shared with test results. Any matching tag links the test to the scenario.</td>
<td><span class="tag tag-link">Identity</span></td>
<td><code>@id:FC-42</code></td>
<td>Placed on scenarios in <code>.feature</code> files. Declares the scenario's stable identity — the canonical ID.</td>
</tr>
<tr>
<td><span class="tag tag-link">Reference</span></td>
<td><code>@scenario:FC-42</code></td>
<td>Placed on test results (via framework markers, test names, or properties). Declares which scenario this test satisfies.</td>
</tr>
<tr>
<td><span class="tag tag-require">Requirement</span></td>
Expand All @@ -260,6 +265,8 @@ <h3>Two Kinds of Tags</h3>
</table>
</div>

<p>Linking matches <strong>only</strong> <code>@scenario:VALUE</code> from results against <code>@id:VALUE</code> from scenarios. Other tags (like <code>@smoke</code> or <code>@regression</code>) never participate in linking — they are purely for organization or test-runner filtering.</p>

<h3>Default Behavior</h3>
<p>Scenarios without any <code>@require-*</code> tags default to <code>@require-e2e</code> &mdash; because if you wrote a Gherkin scenario but didn't declare requirements, it should at least have E2E coverage.</p>

Expand All @@ -284,7 +291,7 @@ <h3>How It Works</h3>
}
}</code></pre>
<p>In your feature file:</p>
<pre><code><span class="hl-tag">@FC-42</span> <span class="hl-tag">@require-unit:auth</span> <span class="hl-tag">@require-integration:billing</span> <span class="hl-tag">@require-e2e:checkout</span>
<pre><code><span class="hl-tag">@id:FC-42</span> <span class="hl-tag">@scenario:FC-42</span> <span class="hl-tag">@require-unit:auth</span> <span class="hl-tag">@require-integration:billing</span> <span class="hl-tag">@require-e2e:checkout</span>
<span class="hl-section">Scenario:</span> Authenticated billing checkout</code></pre>
<p>Each module-scoped requirement is <strong>only</strong> satisfied by a linked result registered under that exact module key. Unscoped results (config key <code>""</code>) never satisfy a module-scoped requirement. A bare <code>@require-unit</code> / <code>@require-e2e</code> (no module) is satisfied by any linked result for that layer.</p>
<p>In the HTML report, satisfied modules show as chips like <code>e2e (checkout) OK</code>; missing ones show <code>Missing</code>.</p>
Expand All @@ -300,8 +307,8 @@ <h2 id="edge-cases">Edge Case Behavior</h2>
<tr><td>Missing <code>features</code> or <code>output</code></td><td>Errors out — both are required</td></tr>
<tr><td>Empty/missing test result path</td><td>Silently ignored (zero tests for that layer)</td></tr>
<tr><td>Malformed XML or JSON</td><td>Aborts with a clear error message</td></tr>
<tr><td>Test matches no scenario (has tags)</td><td>Listed in "Unlinked Tests"</td></tr>
<tr><td>Test matches no scenario (no tags)</td><td>Silently excluded — tagless results can never link</td></tr>
<tr><td>Test with <code>@scenario:</code> matches no <code>@id:</code></td><td>Listed in "Unlinked Tests"</td></tr>
<tr><td>Test has no <code>@scenario:</code> tag</td><td>Silently excluded — cannot participate in identity-based linking</td></tr>
<tr><td>Scenario matches no test</td><td>Shown as "incomplete"</td></tr>
<tr><td><code>@require-*</code> with no matching test</td><td>Layer flagged as missing</td></tr>
<tr><td>Feature-level tags</td><td>Not inherited by scenarios</td></tr>
Expand Down
Loading
Loading