[Bug] EQL Parsing Error on String where condition

[eric-forte-elastic]

Describe the bug

When we have queries that specify a double quotes string as the source of a where, the detection rules EQL parser throws an error. However, this is allowed in the stack.

Note

This used to be a bug issue in the Detection Rules repo and there is a draft PR highlighting why this should be resolved in the upstream EQL library instead of in downstream repos elastic/detection-rules#5999

Core addition from the PR is:

# Matches a quoted-string event type at the start of a (sub)query, e.g.
#   "ec2.amazonaws.com" where ...
#   [ "ec2.amazonaws.com" where ... ]
#   ![ "ec2.amazonaws.com" where ... ]
# The grammar in `eql==1.0.0` only allows a NAME token here, but Elasticsearch
# accepts string literals (since event categories may contain characters such
# as "." that are not valid in an EQL identifier). See
# https://github.com/elastic/detection-rules/issues/5236.
_STRING_EVENT_TYPE_RE = re.compile(
    r'(?P<prefix>\A|(?<=\[)|(?<=!\[))'
    r'(?P<lead_ws>\s*)'
    r'"(?P<value>(?:\\.|[^"\\\r\n])*)"'
    r'(?P<post>\s+where\b)'
)

To Reproduce

"ec2.amazonaws.com" where event.dataset == "aws.cloudtrail"  
  and event.action == "ModifySnapshotAttribute"
  and event.outcome == "success"
  and stringContains (aws.cloudtrail.request_parameters, "attributeType=CREATE_VOLUME_PERMISSION")
  and stringContains (aws.cloudtrail.request_parameters, "add={items")

Will result in an error whereas:

any where event.dataset == "aws.cloudtrail"  
  and event.action == "ModifySnapshotAttribute"
  and event.outcome == "success"
  and stringContains (aws.cloudtrail.request_parameters, "attributeType=CREATE_VOLUME_PERMISSION")
  and stringContains (aws.cloudtrail.request_parameters, "add={items")

Will not. See elastic/detection-rules#5229 for examples, or using the following rule and attempt to load it via view rule (or other means)

1. python -m detection_rules view-rule rules/integrations/aws/exfiltration_ec2_ebs_snapshot_shared_with_another_account.toml

Will result in:

detection-rules on  tune_aws_ebs_snapshot_rules [!?] is  v1.5.0 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules view-rule rules/integrations/aws/exfiltration_ec2_ebs_snapshot_shared_with_another_account.toml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Error loading rule in /tmp/detection-rules/rules/integrations/aws/exfiltration_ec2_ebs_snapshot_shared_with_another_account.toml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/detection-rules/detection_rules/__main__.py", line 31, in <module>
    main()
  File "/tmp/detection-rules/detection_rules/__main__.py", line 28, in main
    root(prog_name="detection_rules")
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/main.py", line 462, in view_rule
    rule = RuleCollection().load_file(rule_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/rule_loader.py", line 507, in load_file
    return self.load_dict(obj, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/rule_loader.py", line 483, in load_dict
    contents = TOMLRuleContents.from_dict(obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/mixins.py", line 144, in from_dict
    return schema.load(obj)
           ^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 973, in load
    all_loaded = super().load(data, many=many, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 792, in load
    return self._do_load(
           ^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 974, in _do_load
    self._invoke_schema_validators(
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 1267, in _invoke_schema_validators
    self._run_validator(
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 845, in _run_validator
    validator_func(output, partial=partial, many=many)
  File "/tmp/detection-rules/detection_rules/rule.py", line 1550, in post_conversion_validation
    data.validate_query(metadata)
  File "/tmp/detection-rules/detection_rules/rule.py", line 760, in validate_query
    validator.validate(self, meta)
  File "/tmp/detection-rules/detection_rules/rule_validators.py", line 599, in validate
    all_targets = self.build_validation_plan(data, meta)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/rule_validators.py", line 401, in build_validation_plan
    is_sequence = getattr(data, "is_sequence", False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/functools.py", line 998, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/rule.py", line 923, in is_sequence
    return eql.utils.get_query_type(self.ast) == "sequence"  # type: ignore[reportUnknownMemberType]
                                    ^^^^^^^^
  File "/usr/lib/python3.12/functools.py", line 998, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/rule.py", line 766, in ast
    return validator.ast
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/functools.py", line 998, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/rule_validators.py", line 362, in ast
    return eql.parse_query(self.query)  # type: ignore[reportUnknownVariableType]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/eql/parser.py", line 1415, in parse_query
    return _parse(text,  "piped_query", implied_any=implied_any, implied_base=implied_base, preprocessor=preprocessor,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/detection_rules/rule_validators.py", line 139, in wrapper
    result = func(query, start=start, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/eql/parser.py", line 1367, in _parse
    raise exc
eql.errors.EqlSyntaxError: Error at line:1,column:19
Invalid syntax

^^^^^

Expected behavior

The query should parse as intended.

Screenshots

Verbose error messages via debugger:

Details

 File "/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 126, in feed_token
    action, arg = states[state][token.type]
                  ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'NAME'

During handling of the above exception, another exception occurred: