Prepare AI Guard tag probability coverage

[smola]

Add a system test for tag probabilities in AI Guard responses and meta-structs.

APPSEC-61900

Motivation

Changes

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

docs/understand/scenarios/ai_guard.md                                   @DataDog/k9-ai-guard @DataDog/system-tests-core
docs/understand/weblogs/end-to-end_weblog.md                            @DataDog/system-tests-core
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/ai_guard/test_ai_guard_sdk.py                                     @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/python/flask/app.py                                  @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/ruby/rails52/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails61/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails72/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails80/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core

[datadog-prod-us1-5]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 6 Tests failed

tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets.test_log_line_budgets[apache-mod-7.0-zts] from system_tests_suite     (Fix with Cursor)

ValueError: Snapshot logb83bf-a085-4951-8ddd-b0abd131258e was not received.

self = <tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets object at 0x7f5cb8a63500>

    def test_log_line_budgets(self):
        self._assert()
>       self._validate_snapshots()

tests/debugger/test_debugger_probe_budgets.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...

tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets.test_log_line_budgets[apache-mod-8.2] from system_tests_suite     (Fix with Cursor)

ValueError: Snapshot log63e63-46fa-4039-9f23-1d95bcf2a62b was not received.

self = <tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets object at 0x7f80344b8ec0>

    def test_log_line_budgets(self):
        self._assert()
>       self._validate_snapshots()

tests/debugger/test_debugger_probe_budgets.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM.test_log_method_snapshot[poc] from system_tests_suite     (Fix with Cursor)

AssertionError: assert 'Snapshot was not received' is None
 +  where 'Snapshot was not received' = <built-in method join of str object at 0x7fbce2c2b720>(['Snapshot was not received'])
 +    where <built-in method join of str object at 0x7fbce2c2b720> = '\n'.join
 +    and   ['Snapshot was not received'] = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7fbcd8ba6c90>.setup_failures

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots_With_SCM object at 0x7fbcd8ba6c90>

    def test_log_method_snapshot(self):
>       self._assert()

...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fadbd2b | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}