appsec: add smoke tests for apm standalone

[florentinl]

Motivation

Test that AppSec is correctly enabled and functional in APM Standalone mode (with infra disabled at the agent level).

To do so, I setup a testing class that performs AAP smoke tests using only data intercepted at the agent level. This testing class can then be used through inheritance to create different test suites linked to different features and manifest entries.

Changes

• Add appsec helpers in interfaces.agent to perform assertions on traces intercepted after the agent
• Add smoke tests to check:
  • Threat detection
  • RASP
  • Remote configuration of rules
  • Telemetry emission
  • API Security
    => This ensures that all communication protocols used by AAP (traces, trace stats, telemetry and RC) keep getting proxied by the agent.
• Add two scenarios and a feature for APM Standalone and ASM Standalone + APM Standalone

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/appsec/test_agent_level_smoke_tests.py                            @DataDog/asm-libraries @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
docs/edit/agent-interface-validation-methods.md                         @DataDog/system-tests-core
manifests/agent.yml                                                     @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/interfaces/_agent.py                                              @DataDog/system-tests-core

[datadog-official]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

❄️ 3 New flaky tests detected

tests.appsec.test_agent_level_smoke_tests.Test_AppSecAPMStandalone.test_api_security_smoke[nginx] from system_tests_suite     (Fix with Cursor)

assert False
 +  where False = any(<generator object AgentLevelSmokeTests.test_api_security_smoke.<locals>.<genexpr> at 0x7f150abe74c0>)

self = <tests.appsec.test_agent_level_smoke_tests.Test_AppSecAPMStandalone object at 0x7f153460b260>

    def test_api_security_smoke(self) -> None:
>       assert any(
            any(key.startswith("_dd.appsec.s.") for key in span.meta) for _, span in interfaces.agent.get_spans(self.r)
        )
E       assert False
...

tests.appsec.test_agent_level_smoke_tests.Test_AppSecAPMStandalone.test_api_security_smoke[spring-boot-wildfly] from system_tests_suite     (Fix with Cursor)

assert False
 +  where False = any(<generator object AgentLevelSmokeTests.test_api_security_smoke.<locals>.<genexpr> at 0x7f9ce46c2500>)

self = <tests.appsec.test_agent_level_smoke_tests.Test_AppSecAPMStandalone object at 0x7f9cf508e330>

    def test_api_security_smoke(self) -> None:
>       assert any(
            any(key.startswith("_dd.appsec.s.") for key in span.meta) for _, span in interfaces.agent.get_spans(self.r)
        )
E       assert False
...

tests.appsec.test_agent_level_smoke_tests.Test_AppSecAPMStandalone.test_lfi_smoke[spring-boot-wildfly] from system_tests_suite     (Fix with Cursor)

AssertionError: No RASP attack found for rule rasp-930-100

self = <tests.appsec.test_agent_level_smoke_tests.Test_AppSecAPMStandalone object at 0x7f9cf508e270>

    def test_lfi_smoke(self) -> None:
        assert self.r.status_code == 200
    
        expected_rule = "rasp-930-100"
        expected_params = {
            "resource": {"address": "server.io.fs.file", "value": "../etc/passwd"},
...

🧪 1 Test failed

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_mix_snapshot[uds] from system_tests_suite     (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7f4434c2b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7f4434c2b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f4426aa2000>.setup_failures

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7f4426aa2000>

    def test_mix_snapshot(self):
>       self._assert()

...

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4042a19 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03d3a84078

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03d3a84078

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 733fcc0d42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4042a1951a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cbeauchesne]

Could you move this deserialization inside the proxy? ping me if you need help (it may not be obvious to do).