CucumberSyntax.md

Cucumber syntax

We use cucumber for behaviour driven development and testing, with gherkin-based tests like the below:

Feature: the name of my feature

  Background:
    Given the generation strategy is interesting
    And there is a field foo

  Scenario: Running the generator should emit the correct data
    Given foo is equal to 8
    Then the following data should be generated:
      | foo  |
      | 8    |
      | null |

More examples can be seen in the generator cucumber features

The framework supports setting configuration settings for the generator, defining the profile and describing the expected outcome. All of these are described below, all variable elements (e.g. {generationStrategy} are case insensitive), all fields and values are case sensitive.

Configuration options

• the generation strategy is {generationStrategy} see generation strategies - default: random
• the combination strategy is {combinationStrategy} see combination strategies - default: exhaustive
• the walker type is {walkerType} see walker types - default: reductive
• the data requested is {generationMode}, either violating or validating - default: validating
• the generator can generate at most {int} rows, ensures that the generator will only emit int rows, default: 1000
• we do not violate constraint {operator}, prevent this operator from being violated (see Operators section below), you can specify this step many times if required

Defining the profile

It is important to remember that constraints are built up of 3 components: a field, an operator and most commonly an operand. In the following example the operator is 'greaterThan' and the operand is 5.

foo is greaterThan 5

Operators are converted to English language equivalents for use in cucumber, so 'greaterThan' is expressed as 'greater than'.

• there is a field {field}, adds a field called field to the profile
• the following fields exist:, adds a set of fields to the profile (is followed by a single column set of strings, each represents a field name)
• {field} is null, adds a null constraint to the profile for the field field
• {field} is anything but null, adds a not(is null) constraint to the profile for field field
• {field} is {operator} {operand}, adds an operator constraint to the field field with the data operand, see operators and operands sections below
• {field} is anything but {operator} {operand}, adds a negated operator constraint to the field field with the data operand, see operators and operands sections below
• there is a constraint:, adds the given JSON block as a constraint as if it was read from the profile file itself. It should only be used where the constraint cannot otherwise be expressed, e.g. for anyOf, allOf and if.
• the maximum string length is {length}, sets the maximum length for strings to the max for the given scenario. The default is 200 (for performance reasons), however in production the limit is 1000.
• untyped fields are allowed, sets the --allow-untyped-fields flag to false - default: flag is true

Operators

See Predicate constraints, Grammatical Constraints and Presentational Constraints for details of the constraints.

Operands

When specifying the operator/s for a field, ensure to format the value as in the table below:

data type	example
string	"my value"
number	1.234
datetime	2001-02-03T04:05:06.000
null	null

datetimes must be expressed as above (i.e. yyyy-MM-ddTHH:mm:ss.fff)

Examples

• ofType → Given foo is of type "string"
• equalTo → Given foo is equal to 5
• inSet →

Given foo is in set: 
  | foo |
  | 1   |
  | 2   |
  | 3   |

• not(after 01/02/2003) → Given foo is anything but after 2003-02-01T00:00:00.00

In addition the following shows how the there is a constraint step can be used:

And there is a constraint:
  """
    {
      "if": { "field": "foo", "is": "equalTo", "value": "dddd" },
      "then": { "field": "bar", "is": "equalTo", "value": "4444" },
      "else": { "field": "bar", "is": "shorterThan", "value": 1 }
    }
  """

Describing the outcome

• the profile is invalid because "{reason}", executes the generator and asserts that an InvalidProfileException or JsonParseException was thrown with the message {reason}, reason is a regular expression*.
• no data is created, executes the generator and asserts that no data was emitted
• the following data should be generated:, executes the generator and asserts that no exceptions were thrown and the given data appears in the generated data, no additional data is permitted.
• the following data should be generated in order:, executes the generator and asserts that no exceptions were thrown and the given data appears in the same order in the generated data, no additional data is permitted.
• the following data should be included in what is generated:, executes the generator and asserts that no exceptions were thrown and the given data is present in the generated data (regardless of order)
• the following data should not be included in what is generated:, executes the generator and asserts that no exceptions were thrown and the given data is not present in the generated data (regardless of order)
• some data should be generated, executes the generator and asserts that at least one row of data was emitted
• {number} of rows of data are generated, executes the generator and asserts that exactly the given number of rows are generated

* Because {reason} is a regular expression, certain characters will need to be escaped, by including a \ before them, e.g. \(, \), \[, \], etc.

Validating the data in the output

DateTime

• {field} contains datetime data, executes the generator and asserts that field contains either null or datetimes (other types are allowed)
• {field} contains only datetime data, executes the generator and asserts that field contains only null or datetimes
• {field} contains anything but datetime data, executes the generator and asserts that field contains either null or data that is not a datetime.
• {field} contains datetimes between {min} and {max} inclusively, executes the generator and asserts that field contains either null or datetimes between {min} and {max}. Does so in an inclusive manner for both min and max.
• {field} contains datetimes outside {min} and {max}, executes the generator and asserts that field contains either null or datetimes outside {min} and {max}.
• {field} contains datetimes before or at {before}, executes the generator and asserts that field contains either null or datetimes at or before {before}
• {field} contains datetimes after or at {after}, executes the generator and asserts that field contains either null or datetimes at or after {after}

Numeric

Note these steps work for asserting both integer and decimal data. There are no current steps for asserting general granularity.

• {field} contains numeric data, executes the generator and asserts that field contains either null or numeric values (other types are allowed)
• {field} contains only numeric data, executes the generator and asserts that field contains only null or numeric values
• {field} contains anything but numeric data, executes the generator and asserts that field contains either null or data that is not numeric.
• {field} contains numeric values between {min} and {max} inclusively, executes the generator and asserts that field contains either null or numeric values between {min} and {max}. Does so in an inclusive manner for both min and max.
• {field} contains numeric values outside {min} and {max}, executes the generator and asserts that field contains either null or numeric values outside {min} and {max}.
• {field} contains numeric values less than or equal to {value}, executes the generator and asserts that field contains either null or numeric values less than or equal to {value}
• {field} contains numeric values greater than or equal to {value}, executes the generator and asserts that field contains either null or numeric values greater than or equal to {value}

String

• {field} contains string data, executes the generator and asserts that field contains either null or string values (other types are allowed)
• {field} contains only string data, executes the generator and asserts that field contains only null or string values
• {field} contains anything but string data, executes the generator and asserts that field contains either null or data that is not a string.
• {field} contains strings of length between {min} and {max} inclusively, executes the generator and asserts that field contains either null or strings with lengths between {min} and {max}. Does so in an inclusive manner for both min and max.
• {field} contains strings of length outside {min} and {max}, executes the generator and asserts that field contains either null or strings with lengths outside {min} and {max}.
• {field} contains strings matching /{regex}/, executes the generator and asserts that field contains either null or strings that match the given regular expression.
• {field} contains anything but strings matching /{regex}/, executes the generator and asserts that field contains either null or strings that do not match the given regular expression.
• {field} contains strings shorter than or equal to {length}, executes the generator and asserts that field contains either null or string values shorter than or equal to {length}
• {field} contains strings longer than or equal to {length}, executes the generator and asserts that field contains either null or string values longer than or equal to {length}

Null (absence/presence)

• {field} contains anything but null, executes the generator and asserts that field has a value in every row (i.e. no nulls)

Cucumber test style guide

• Each test should be specific to one requirement.
• Tests should specify definite expected results rather than using "should include".
• All tables should be padded to the width of the largest item.
• All block-level indentation should be 2 spaces, as below:

Feature: ...
  ...

  Background:
    Given ...

  Scenario: ...
    Given ...:
      | ... |
      | ... |
      | ... |
    And ...:
      """
      """
    When ...
    Then ... 
    And ...