Skip to content

[Bug]: skillforge test returns exit code 1 despite successful tests in CI environment #6

@hertznsk

Description

@hertznsk

Description

When running skillforge test --format junit in CI/Docker environments, the command returns exit code 1 even when all tests pass successfully. This behavior differs from local development environments where the same command returns exit code 0.

The issue appears to be caused by a regression between the v1.0.0 release (from PyPI) and the latest commits on the main branch. Local development uses an editable install of the latest HEAD (v1.0.0-17-g821a01d) which includes fixes that resolve this exit code issue.

Steps to Reproduce

1. Install v1.0.0 from PyPI and run test (REPRODUCES BUG - exit code 1)

docker run --rm -v /home/gblinov/projects/g.blinov/skills:/skills -w /skills python:3.11-slim bash -c "pip install ai-skillforge==1.0.0 && skillforge test skills/dev-example/ --format junit -o /tmp/test.xml; echo Exit code: \$?"

2. Install latest HEAD from GitHub and run test (FIXED - exit code 0)

docker run --rm -v /home/gblinov/projects/g.blinov/skills:/skills -w /skills python:3.11-slim bash -c "apt-get update && apt-get install -y git && pip install git+https://github.com/lhassa8/skillforge.git && skillforge test skills/dev-example/ --format junit -o /tmp/test.xml; echo Exit code: \$?"

3. Compare pip version vs HEAD

# v1.0.0 from PyPI (buggy)
pip install ai-skillforge==1.0.0
skillforge test skills/dev-example/ --format junit -o /tmp/test.xml
echo $?  # Exit code: 1

# Latest from GitHub (fixed)
pip install git+https://github.com/lhassa8/skillforge.git
skillforge test skills/dev-example/ --format junit -o /tmp/test.xml
echo $?  # Exit code: 0

Expected Behavior

Return exit code 0

Actual Behavior

Return exit code 1

SKILL.md Content (if applicable)

Error Output / Logs

### Test Output (same in both environments)

Testing skill: dev-example
Mode: mock | Tests found: 1 file(s)

JUnit XML written to: /tmp/test.xml


### JUnit XML Output (same in both environments)

<?xml version="1.0" ?>
<testsuite name="dev-example" tests="1" failures="0" errors="0" skipped="0" time="0.000">
  <testcase name="basic_test" classname="dev-example.basic_test" time="0.000"/>
</testsuite>

SkillForge Version

1.0.0

Python Version

3.11

Operating System

Linux

Additional Context

Expected vs Actual Behavior

Environment skillforge Version Exit Code Status
Local (editable) v1.0.0-17-g821a01d 0 ✓ Works
Docker (pip install) v1.0.0 1 ✗ Fails

Root Cause

The issue is in the skillforge package version:

  1. v1.0.0 (PyPI): Contains a bug where the CLI exits with code 1 even when tests pass successfully
  2. v1.0.0-17-g821a01d (HEAD): Includes fixes from commits 821a01d, 879c686 that resolve the exit code issue

The fix appears to be related to how the CLI handles the exit code after test execution, specifically in the cli.py:test() function in the tester module.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions