Skip to content

Comparison: expected-side error is re-raised instead of producing an assertion failure #12

Description

@jcouball

Summary

When the expected expression evaluates to a StandardError (e.g. #=> raise ZeroDivisionError, "divided by 0") but the actual expression returns a normal value, compare_values re-raises the expected-side error via only_one_is_error?. Minitest reports this as an Error, making it look like the user's code crashed — when in fact the code returned a value and the comparison simply didn't match. The actual return value is never shown to the user.

Location

lib/yard_example_test/example/comparison.rbcompare_values and only_one_is_error?

def compare_values(expected, actual)
  if both_are_errors?(expected, actual)
    assert_equal(...)
  elsif (error = only_one_is_error?(expected, actual))
    raise error
  ...
end

only_one_is_error? is symmetric — it returns whichever side is a StandardError — but the two cases have different semantics:

Expected Actual Current behavior Correct behavior
Error Value Re-raises expected error → shows as Error Should produce an assertion failure comparing the two
Value Error Re-raises actual error → shows as Error Correct — the user's code raised unexpectedly

Concrete example

# @example
#   divide(1, 0) #=> raise ZeroDivisionError, "divided by 0"
def divide(a, b)
  0  # bug: returns 0 instead of raising
end

Current output

1) Error:
#divide#test_0001_:
ZeroDivisionError: divided by 0

The user sees a ZeroDivisionError and thinks divide raised — but divide returned 0.

Expected output

An assertion failure showing something like:

1) Failure:
#divide#test_0001_:
Expected: #<ZeroDivisionError: divided by 0>
  Actual: 0

Proposed solution

Split the only_one_is_error? branch into two distinct cases:

def compare_values(expected, actual)
  if both_are_errors?(expected, actual)
    assert_equal("#<#{expected.class}: #{expected}>", "#<#{actual.class}: #{actual}>")
  elsif expected.is_a?(StandardError) && !actual.is_a?(StandardError)
    # Expected an error but got a value — assertion failure, not re-raise
    assert_equal("#<#{expected.class}: #{expected}>", actual.inspect)
  elsif !expected.is_a?(StandardError) && actual.is_a?(StandardError)
    # Actual raised unexpectedly — re-raise so Minitest reports as Error
    raise actual
  elsif expected.nil?
    assert_nil(actual)
  else
    assert expected === actual, diff(expected, actual)
  end
end

Acceptance criteria

  • When expected evaluates to an error but actual returns a value, Minitest reports a Failure (not an Error) with a meaningful diff
  • When actual raises unexpectedly, the error is still surfaced as an Error
  • When both sides are errors, they are still compared via assert_equal
  • All existing cucumber scenarios continue to pass

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions