Add size constraints to CTIM collection fields (XDR-46972)#483
Add size constraints to CTIM collection fields (XDR-46972)#483ereteog wants to merge 1 commit intothreatgrid:masterfrom
Conversation
Add pred/max-len constraints to seq-of/set-of collection fields in sighting, incident, and common schemas to prevent oversized documents causing ES bulk write failures. - Bump flanders to 1.1.1-SNAPSHOT (adds :spec support on collection types) - Add default-collection-max-len (500) for common collection fields - Add specific limits for sighting fields (observables, relations, targets, data tables) - Constrain external_ids and external_references in base entity entries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
yogsototh
left a comment
There was a problem hiding this comment.
I have only two concerns:
-
why these numbers, to me they are a bit like "magic numbers" and it will eventually breaks in PROD refusing the creation of many entities suddenly without a warning which may create issues with already working integrations. Personally, I think it may be a bit too much to constrain statically here instead of directly into the CTIA application where we could apply different size constraints depending on the source for example. Which will give the time for an integration to adapt to the new constraints. And perhaps instead of a hard reject, we may also integrate a mechanism to just cut the strings/arrays to keep the first one.
-
We should check if adding these constraints does not affect Swagger and Swagger UI. From memory plumatic/schemas and swagger do not support very well certain kind of external constraints like this one. So while the API will work, Swagger UI may be broken, or not show the correct schemas anymore.
Add
pred/max-lenconstraints to collection fields in sighting, incident, and common schemas to prevent oversized documents causing ES bulk write failures (HTTP 413), connection resets, and memory pressure.Changes
1.1.1-SNAPSHOT(adds:specsupport onSequenceOfTypeandSetOfType)default-collection-max-len(500) constant incommon.cljcfor shared collection fieldsexternal_idsandexternal_referencesin base entity entriescolumns: 100,rows: 10,000targets: 1,000,observables: 2,000,relations: 10,000categories,assignees,tactics,techniques) to 500§ QA
No QA is needed. All 138 existing tests (including generative tests) pass.
§ Release Notes
§ Squashed Commits
🤖 Generated with Claude Code