Skip to content

Some potential errors in the benchmark #68

@Vexoben

Description

@Vexoben

Hi,

While testing on the benchmark, I noticed several potential issues in some of the specifications. They mainly fall into two categories:

  • Some specifications appear to be weaker than intended and may allow unintended solutions.
  • Some specifications seem to describe behavior that differs from what is stated in the accompanying documentation.

I have documented the issues I found in a separate report. Unfortunately, I do not currently have the cycles to prepare a PR to fix them. I am therefore sharing the report here in case it is useful for maintainers or other contributors.

clever_feedback.pdf

Problem ID Potential Issue
Problem 7 Underspecified Post-condition
Problem 9 Implementation Issue
Problem 24 Underspecified Post-condition
Problem 27 Underspecified Post-condition
Problem 29 Underspecified Post-condition
Problem 30 Underspecified Post-condition
Problem 44 Underspecified Post-condition
Problem 49 Underspecified Post-condition; Possible Test Case Discrepancy
Problem 68 Underspecified Post-condition
Problem 69 Underspecified Post-condition
Problem 79 Underspecified Post-condition
Problem 84 Underspecified Post-condition
Problem 88 Underspecified Post-condition
Problem 89 Underspecified Post-condition
Problem 94 Possible Incorrect Specification
Problem 96 Underspecified Post-condition
Problem 156 Underspecified Post-condition
Problem 161 Underspecified Post-condition

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions