Improvements to autotest (run towards jar, other server, print outcome to console...)#486
Open
Improvements to autotest (run towards jar, other server, print outcome to console...)#486
Conversation
Contributor
Author
|
In this log, search for "Tests outcome:" https://debusine.debian.net/debian/developers/artifact/3376800/raw/log The test suite is run against:
As you can see, outcome is not consistent between the different servers and also differs wrt to cli for some tests. EDIT: I'm pleased to announce that is MR is ready since tests all pass in debian when invoked through CLI, or as a servlet in jetty 9, 12, tomcat 10 & 11. However:
Sample of test log for the debian package can be viewed https://debusine.debian.net/debian/developers/artifact/3380734/#autopkgtest. See tests having name ending with "testsuite". |
tenzap
force-pushed
the
autotest
branch
3 times, most recently
from
7759df9 to
3070d71
Compare
tenzap
changed the title
tenzap
changed the title
Permit to build autotest without running the tests. This add a 2nd argument to the AutoTest.main which is the engine to use for tests. It can be either 'servlet' (URL to css-validator servlet at jigsaw.w3.org) or 'jar' (the presently built css-validator.jar) or 'cli' (a command line wrapper script called css-validator'
Properly manage when the CLI errors out (instead of test failure) Write also the outcome of CLI tests to result.html file Display more information in the console, esp. the errors.
Compared to how parameters are passed to the servlet and their default values, there is a difference with CLI which was causing different results when running the test suite. This will consider the warning argument properly, so that the count of warning is correct depending on the value of the parameter This will also pass the 'medium' argument properly as well, because it should not be considered for CSS1 (like the servlet does)
Same default value as the javascript test processor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Permit to build autotest without running the tests. This add a 2nd argument to the AutoTest.main which is the engine to use for tests. It can be either 'servlet' (URL to css-validator servlet at jigsaw.w3.org) or 'jar' (the presently built css-validator.jar) or 'cli' (a command line wrapper script called css-validator'
May be useful for debian package so that we can run the tests towards our build instead of official css-validator
This changes slighly the output
Only the errors will be written to stderr. Each item and its result is written to stdout.
By the way, some tests in
testsuite/(latest master commit of that repo) are actually failing...