Skip to content

Add size constraints to CTIM collection fields (XDR-46972)#483

Open
ereteog wants to merge 1 commit intothreatgrid:masterfrom
ereteog:add-collection-size-limits
Open

Add size constraints to CTIM collection fields (XDR-46972)#483
ereteog wants to merge 1 commit intothreatgrid:masterfrom
ereteog:add-collection-size-limits

Conversation

@ereteog
Copy link
Contributor

@ereteog ereteog commented Mar 12, 2026

Epic XDR-46972
Related flanders#58

Add pred/max-len constraints to collection fields in sighting, incident, and common schemas to prevent oversized documents causing ES bulk write failures (HTTP 413), connection resets, and memory pressure.

Changes

  • Bump flanders to 1.1.1-SNAPSHOT (adds :spec support on SequenceOfType and SetOfType)
  • Add default-collection-max-len (500) constant in common.cljc for shared collection fields
  • Constrain external_ids and external_references in base entity entries
  • Add specific limits for sighting fields:
    • columns: 100, rows: 10,000
    • targets: 1,000, observables: 2,000, relations: 10,000
  • Constrain incident fields (categories, assignees, tactics, techniques) to 500

§ QA

No QA is needed. All 138 existing tests (including generative tests) pass.

§ Release Notes

intern: Add size constraints to CTIM collection fields.

§ Squashed Commits

🤖 Generated with Claude Code

Add pred/max-len constraints to seq-of/set-of collection fields in
sighting, incident, and common schemas to prevent oversized documents
causing ES bulk write failures.

- Bump flanders to 1.1.1-SNAPSHOT (adds :spec support on collection types)
- Add default-collection-max-len (500) for common collection fields
- Add specific limits for sighting fields (observables, relations, targets, data tables)
- Constrain external_ids and external_references in base entity entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Contributor

@yogsototh yogsototh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have only two concerns:

  • why these numbers, to me they are a bit like "magic numbers" and it will eventually breaks in PROD refusing the creation of many entities suddenly without a warning which may create issues with already working integrations. Personally, I think it may be a bit too much to constrain statically here instead of directly into the CTIA application where we could apply different size constraints depending on the source for example. Which will give the time for an integration to adapt to the new constraints. And perhaps instead of a hard reject, we may also integrate a mechanism to just cut the strings/arrays to keep the first one.

  • We should check if adding these constraints does not affect Swagger and Swagger UI. From memory plumatic/schemas and swagger do not support very well certain kind of external constraints like this one. So while the API will work, Swagger UI may be broken, or not show the correct schemas anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants