Hi,
While testing on the benchmark, I noticed several potential issues in some of the specifications. They mainly fall into two categories:
- Some specifications appear to be weaker than intended and may allow unintended solutions.
- Some specifications seem to describe behavior that differs from what is stated in the accompanying documentation.
I have documented the issues I found in a separate report. Unfortunately, I do not currently have the cycles to prepare a PR to fix them. I am therefore sharing the report here in case it is useful for maintainers or other contributors.
clever_feedback.pdf
| Problem ID |
Potential Issue |
| Problem 7 |
Underspecified Post-condition |
| Problem 9 |
Implementation Issue |
| Problem 24 |
Underspecified Post-condition |
| Problem 27 |
Underspecified Post-condition |
| Problem 29 |
Underspecified Post-condition |
| Problem 30 |
Underspecified Post-condition |
| Problem 44 |
Underspecified Post-condition |
| Problem 49 |
Underspecified Post-condition; Possible Test Case Discrepancy |
| Problem 68 |
Underspecified Post-condition |
| Problem 69 |
Underspecified Post-condition |
| Problem 79 |
Underspecified Post-condition |
| Problem 84 |
Underspecified Post-condition |
| Problem 88 |
Underspecified Post-condition |
| Problem 89 |
Underspecified Post-condition |
| Problem 94 |
Possible Incorrect Specification |
| Problem 96 |
Underspecified Post-condition |
| Problem 156 |
Underspecified Post-condition |
| Problem 161 |
Underspecified Post-condition |
Hi,
While testing on the benchmark, I noticed several potential issues in some of the specifications. They mainly fall into two categories:
I have documented the issues I found in a separate report. Unfortunately, I do not currently have the cycles to prepare a PR to fix them. I am therefore sharing the report here in case it is useful for maintainers or other contributors.
clever_feedback.pdf