forked from nlohmann/json
-
Notifications
You must be signed in to change notification settings - Fork 6
Add assisting files for closed misbehaviours and runtime analysis #26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
124 commits
Select commit
Hold shift + click to select a range
18e1f0d
Halnasri resolve tt confidence feedback (#21)
Erikhu1 caefeae
Resolve TT-CONSTRUCTION Feedback (#23)
Erikhu1 28c4ccc
Erikhu1 add missing links (#25)
Erikhu1 37c1a07
Reference corrections (#19)
Erikhu1 748b55a
Resolve TT-PROVENANCE Feedback (#14)
LucaFgr df946d2
halnasri-Revisit TT-RESULTS (#17)
6ec3d20
added TA-Releases -> JLS-53 link (#27)
LucaFue dd3bf5b
bump urllib3 version from 2.5.0 to 2.6.0 (#26)
Erikhu1 ad3d3e9
Erikhu1 sync with prod (#31)
Erikhu1 ba5915b
Halnasri fix statements (#30)
2ddacb3
Removed multiple validators from statements by splitting them up (#35)
LucaFue 849f855
Erikhu1 sync with prod (#38)
Erikhu1 692baeb
Erikhu1 fix code scanning alerts (#40)
Erikhu1 ca899f0
Erikhu1 matrix specification (#43)
Erikhu1 941c9ff
add reference to check amalgamation in JLS-14 (#44)
Erikhu1 8f0d8cd
Resolve TA-Analysis comments (#39)
ee29f12
remove TSF/MemoryEfficientTest ResultData.db from stash (#46)
fefa936
fix branch checkout structure in publish_test_data_* workflows (#49)
17175a3
Halnasri fix ci build documentation (#50)
afccbfb
started with TA-Misbehaviours
LucaFgr 3ba9098
Filled out Checklist for TA-Behaviours
LucaFgr 8c9508f
filled out checklist for TA-Misbehaviours
LucaFgr df823a3
added additional context information
LucaFgr b6a6227
added checklist answers for TA-Behaviours and TA-Constraints
LucaFgr c11a978
cosmetic changes to TA-BEHAVIOURS
LucaFgr 693adb1
small updates to TA-Misbehaviours
LucaFgr 544f6d0
created JLS-54 to JLS-60
LucaFgr 20ff2b7
cosmetic change to JLS-54
LucaFgr 379c542
update
LucaFgr 65bc51c
improved checklist and evidence for TA-Behaviour
LucaFgr e181ee7
worked on TA-Misbehaviours
LucaFgr bf67eec
added JLS-70 and JLS-71
LucaFgr 1baed11
added JLS-72
LucaFgr b5895e7
Worked on Context file of TA-Constraints
LucaFgr 603a563
worked on misbehaviours context file
LucaFgr 225e46a
halnasri-Revisit_TT_INDICATORS (#28)
03d7107
remove references to parent workflow for JLS-54 and JLS-55
LucaFgr a5a6349
Added TODOs/Comments to TA-Behaviours and TA-Constraints Context files
LucaFgr c633504
created JLS-56
LucaFgr 4d0ded3
resolved TODOs in TA-Behaviours
LucaFgr 3ce7f40
edited JLS 71
70cfb5f
JLS-70 and JLS-72
c19d526
documented misunderstandings
14dc8ad
added https validator to JLS 70 and 71
e153c07
added JLS-73
08ae742
small fix
ff1df9e
added references to JLS 24 and 31
1282cee
edited zthe checklist of TA-Misbehaviour
025d035
added https validator to JLS-24
1695462
typos
00a8c81
fix pr count gate and coverage gate and add check_artifact_exists evi…
e9f9109
edited artifact output for the coverage gate
ee4b7a7
version control for PR and coverage gate workflow
b919c7d
fix file naming in workflows
af24784
fix answers to the checklist of TA-CONSTRAINTS
4f34209
edited JLS 72
b0faaaa
edited references for JLS 56
fea0fd7
TA-constraints edited checklist
f8e2657
set pr count gate to 15 open PRs
38dd6fa
small fixes
ef31e3d
exception handling tests evidence
LucaFgr c82acbc
TA-Misbehaviours stress tests
LucaFgr fb306b2
TA-Misbehaviours list of misbehaviours
LucaFgr f78941e
edited answer about risk analyses in TA-INDICATORS
b8c9aa8
added JLS-76 and comments to TT-Misbehaviours
LucaFgr 17cf293
removed JLS-69, its link and replace context answer with JLS-11.
LucaFgr 5650e92
replaced JLS-69 by JLS-11 in TA-Indicators context file
LucaFgr 40bb11a
added link of TA-Misbehaviours to JLS-11
LucaFgr d635790
added explanation of why there are no incentives to manipulate inform…
LucaFgr 6b0a4f1
added answer for undiscovered expectations
LucaFgr b0bedd9
answered new expectation identification question
LucaFgr f3ba14f
added test data to answer
LucaFgr cbc4546
test repo renaming effects
LucaFgr 2cdda56
filled out remaining answer in TA-Constraints
LucaFgr 02a2ca6
answered to TA-Behaviours
LucaFgr 995183d
answered result evaluation question in TA-Misbehaviours
LucaFgr 2ff3f10
added answer to fault induction test misbehaviour
LucaFgr c2fd098
answered ta-misbehaviour question, added evidence to JLS-76
LucaFgr 80f4108
add risk analysis
daa774b
small fixes
438687e
rewrite steps overview
1e8025f
replace - with comma in TA-Behaviours context file
LucaFgr 68e8252
Change phrasing of JLS-24
LucaFgr 4c1ff69
changed risk analysis AOU-07 formulation
LucaFgr f4e3fec
delete unnecessary section in risk analysis
LucaFgr 22d926b
deleted unnecessary section in risk analysis
LucaFgr f202d6b
changed naming of first risk analysis step
LucaFgr 840152b
changed referrals to risk_analysis in ta-indicators and ta-misbehaviours
ThomasClausnitzer 67900e5
changed links from legacy gitlab TSF documentation to new ecplise TSF…
ThomasClausnitzer 440a1ea
Added a control structure diagram both as drawio/png and embedded it …
ThomasClausnitzer f5a0021
Change risk_analysis headers and structure to fit expected steps of R…
ThomasClausnitzer 398a02c
Remade part 4 Unsafe Control Actions to fit review.
ThomasClausnitzer 8fa7394
Add step 5 Controller Constraints.
ThomasClausnitzer 802ec9d
Add step 6 Control loops to risk_analysis
ThomasClausnitzer 8975d49
Add step 8 Causal Scenario Constrains to risk analysis
ThomasClausnitzer 59f4e69
Add step 10 Review of STPA results to risk analysis.
ThomasClausnitzer 780b797
Reworked step 9 Misbehaviours and Expectations in risk analysis to b…
ThomasClausnitzer 0a820ce
Update markdown tables of risk_analysis to follow column names and or…
ThomasClausnitzer bb554ea
Add additional Controller Functional Constraints under 5) Device Cotr…
ThomasClausnitzer 38fc8fa
Updated step 7) Causal Scenarios to follow review guidelines in risk_…
ThomasClausnitzer ce03617
TA-Constraints: add AOU-31 resource/time budget assumption
ThomasClausnitzer c7d94f3
Removed typos and irregularities for review changes.
ThomasClausnitzer 5c0eb10
Replace legacy UCA IDs with current combined IDs as in review.
ThomasClausnitzer 48d149d
Changed minor remarks from review.
ThomasClausnitzer 1bd7fa3
Changed failure description in 6) Control Loops and Sequences in risk…
ThomasClausnitzer dfc804d
Changed minor errors in STPA diagram.
ThomasClausnitzer 5461aae
Sync up of latest changes from prod/main into dev/main (#51)
ThomasClausnitzer ff442f8
fix errors in dotstop file (#53)
LucaFue e3d0ef2
changing workflows so publish documentation runs (#54)
LucaFue 135d849
Fix publishing error into repo. (#56)
ThomasClausnitzer 89f727e
Add runtime and memory consumption analysis in TSF docs. (#57)
ThomasClausnitzer d3acc81
Fixed bug that skipped Coverage Gate for pushing into main, as this w…
ThomasClausnitzer bbf2dc7
Added a sampled list of closed upstream issues/misbehaviours and how …
ThomasClausnitzer a66c994
Updated closed misbehaviours list by splitting information in more co…
ThomasClausnitzer 71d6d0d
Fixed type in nlohmann closed misbehaviour file.
ThomasClausnitzer e2130c7
Merge remote-tracking branch 'upstream/main' into ThomasClausnitzer-h…
ThomasClausnitzer 1fe37e0
Minor review change.
ThomasClausnitzer 812c2ae
Merge pull request #59 from score-json/ThomasClausnitzer-history_sync…
ThomasClausnitzer e5259b0
Embedd additional documents in the TSF/docs folder (#61)
ThomasClausnitzer 7d8224e
Add tailored STPA risk analysis (#25)
aschemmel-tech 8992adb
Delete old first version risk_analysis.md as it was never merged upst…
ThomasClausnitzer ba01d69
Merge remote-tracking branch 'upstream/main' into ThomasClausnitzer-s…
ThomasClausnitzer 1912eeb
Merge pull request #63 from score-json/ThomasClausnitzer-sync-with-prod
ThomasClausnitzer b739695
Add changes due to review comments of upstream PR for runtime analysi…
ThomasClausnitzer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Sample of closed upstream misbehaviours (pre-3.12.0) | ||
|
|
||
| This file collects and comments a **sample** of *20 closed* upstream bug issues from the original [`nlohmann/json`](https://github.com/nlohmann/json) repository. | ||
|
|
||
| Selection criteria: | ||
| - Issue is **closed** and labeled **`kind: bug`** upstream. | ||
| - Issue was **closed before** the upstream release of **v3.12.0** (published at `2025-04-11`). | ||
| - A **random sample of 20** issues was taken from the upstream query: [closed `kind: bug` before v3.12.0](https://github.com/nlohmann/json/issues?q=is%3Aissue+is%3Aclosed+label%3A%22kind%3A+bug%22+closed%3A%3C2025-04-11) | ||
|
|
||
| ## sampled closed misbehaviours | ||
|
|
||
| issue-id | applies to S-CORE | problem | resolution | closed after | ||
| ---------|-------------------|---------|------------|------------- | ||
| [2147](https://github.com/nlohmann/json/issues/2147) | No | Compilation error report for very old GCC (4.8.5) with `std::bind`/`std::function`. | Was labeled as a compiler bug; closed as stale after no further comment from author after trial of workaround. | 2.5 months | ||
| [2574](https://github.com/nlohmann/json/issues/2574) | No | `get<std::array<T,N>>()` fails for non-default-constructible element types. | Was fixed in https://github.com/nlohmann/json/pull/2576 and ticket closed after release in 3.10. | 3.5 months | ||
| [2655](https://github.com/nlohmann/json/issues/2655) | No | `std::pair` serialization expectation mismatch, object serialized instead of arrays. | Documentation updated to better explain behavior. | 5 months | ||
| [2676](https://github.com/nlohmann/json/issues/2676) | No | NVCC warnings when instantiating templates under CUDA. | Was fixed in https://github.com/nlohmann/json/pull/3227 after release in 3.10.5. | 10 months | ||
| [2725](https://github.com/nlohmann/json/issues/2725) | No | Build with `-fno-exceptions` (and/or `JSON_NOEXCEPTIONS`) still hits `throw`. | Was already fixed in https://github.com/nlohmann/json/pull/2347. | the same day | ||
| [3025](https://github.com/nlohmann/json/issues/3025) | No | Brace-initialization in function calls was reported to select a copy path (`func({j});` prints `10` instead of `[10]`). | Was a duplicate of https://github.com/nlohmann/json/issues/2311 and that issue was declared not a bug and solved by referencing to Docu. | next day | ||
| [3156](https://github.com/nlohmann/json/issues/3156) | No | `std::filesystem::path` conversions unavailable for older macOS deployment targets even with C++17. | Was already solved in https://github.com/nlohmann/json/pull/3101. | the same day | ||
| [3542](https://github.com/nlohmann/json/issues/3542) | No | AddressSanitizer container-overflow report in diagnostics code path on Windows/MSVC test configuration. | Was declared bug of MSSTL outside of nlohmann/json. | 2.5 years | ||
| [3704](https://github.com/nlohmann/json/issues/3704) | No | `operator[]`/`.at()` to `std::string const&` binds a temporary; use `get_ref` for a real reference. | Was not a bug but expected behaviour. | the same day | ||
| [4019](https://github.com/nlohmann/json/issues/4019) | No | `flatten()` fails to compile when using an alternative string type. | Issue was confirmed and fixed upstream in https://github.com/nlohmann/json/pull/4613. | 21 months | ||
| [4066](https://github.com/nlohmann/json/issues/4066) | No | Parsing freeze reported on a constrained platform (Nintendo Switch) with a larger JSON input. | Author identified problem in own code. | within 1 day | ||
| [4116](https://github.com/nlohmann/json/issues/4116) | No | GCC `-Wodr` warning about potential ODR violations in the single-header build. | Author identified it as issue of other library tinygltf. | the same day | ||
| [4163](https://github.com/nlohmann/json/issues/4163) | No | Deprecation warning about `std::char_traits<unsigned char>` when parsing binary formats from `std::vector<std::uint8_t>`. | Was fixed in https://github.com/nlohmann/json/pull/4179. | 2 months | ||
| [4193](https://github.com/nlohmann/json/issues/4193) | No | Exception SIGSEGV reported while parsing a valid JSON file from a stream. | Was declared issue of compiler. | the same day | ||
| [4241](https://github.com/nlohmann/json/issues/4241) | No | `#include <version>` was reported to be shadowed by a project-provided header named `version` on the include path. | Declared issue of the calling library not adhering to standard. | the same day | ||
| [4284](https://github.com/nlohmann/json/issues/4284) | No | `#pragma pack(push, 1)` was reported to break compilation units and lead to a segfault if packing is not restored. | Was declared not a bug but user error. | within 1 week | ||
| [4299](https://github.com/nlohmann/json/issues/4299) | No | Single-element `std::initializer_list` construction was reported to yield an array for strings (e.g., `json{"hello"}`). | Was declared not a bug but user error. | the same day | ||
| [4332](https://github.com/nlohmann/json/issues/4332) | No | Hex-like git ref string reported as number overflow. | Author realised themselves that it was user error. | the same day | ||
| [4427](https://github.com/nlohmann/json/issues/4427) | No | Misfiled "CVE" report describing a Python `jaraco/zipp` issue for possible DoS Attack. | Was declared not a bug. | the same day | ||
| [4467](https://github.com/nlohmann/json/issues/4467) | No | Error C2440 report with little context given. | Author realised themselves that it was user error. | within 1 week | ||
|
|
||
| ## Conclusion | ||
|
|
||
| The analysis of a random sample of 20 closed upstream misbehaviours showed that misbehaviours were addressed appropriately and closed in a reasonable timeframe. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Runtime and memory consumption analysis | ||
|
|
||
| This file gives an overview about performance and memory usage of the nlohmann/json library and analyses railguards at extreme memory usage and timeout. | ||
|
|
||
| ## External parsing benchmark snapshot | ||
|
|
||
| The upstream benchmark project [miloyip/nativejson-benchmark](https://github.com/miloyip/nativejson-benchmark#parsing-time) publishes a parsing-time comparison for a wide set of C and C++ JSON libraries. Its sample result includes `nlohmann/json` and provides an external point of reference for parsing speed. | ||
|
|
||
| .png) | ||
|
|
||
| In the parsing-time benchmark, nlohmann/json has a parsing time of 72 ms and ranks 17th out of 40 compared JSON parsers, placing it in the faster half of the sample. The average parsing time across all parsers in that sample is 145.675 ms. | ||
|
|
||
| The same benchmark also publishes a parsing-memory comparison for the same sample environment. | ||
|
|
||
| .png) | ||
|
|
||
| In the parsing-memory benchmark, the rank-1 parser "Qt(C++)" is excluded because the benchmark did not correctly hook its memory allocations and therefore reports an incorrect result (see https://github.com/miloyip/nativejson-benchmark?tab=readme-ov-file#parsing-memory). Of the remaining 39 parsers, nlohmann/json ranks 10th with a memory usage of 9,897,872 bytes, placing it in the lowest-memory quarter of the sample. The average memory usage across the remaining parsers is 28,922,224 bytes. | ||
|
|
||
| Note: The figures and data above are externally maintained benchmark snapshots from the `nativejson-benchmark` repository and are included here as contextual evidence only. They were not generated by the CI of this repository and were last updated on 2016-09-09. This benchmark library was chosen due to the wide range of tested JSON libraries and its popularity, as it is also referenced in the original `nlohmann/json` repository. Newer comparisons such as https://github.com/stephenberry/json_performance show a similar broad trend for libraries that appear in both comparisons, but the exact numbers are environment-dependent and should not be treated as repository-specific guarantees. | ||
|
|
||
| ## Benchmark conclusion | ||
|
|
||
| The benchmark snapshot does not show nlohmann/json to be a best-in-class parser for raw performance. In the sampled environment it is materially slower (~9x) and more memory-intensive (~2x) than the popular JSON parser RapidJSON, while still performing better than the sample average and remaining within the same general range as many other widely used JSON libraries. | ||
|
|
||
| For the TSF process, the relevant conclusion is therefore not that nlohmann/json is the fastest parser, but that its observed runtime and memory characteristics are consistent with the stated scope and claims of this repository. This repository does not make a hard real-time, low-latency, or minimum-memory claim. Instead, it documents the library's expected trade-off: usability, maturity, and broad ecosystem adoption are prioritised over maximum parsing speed. As such, no disproportionate runtime or memory concern is indicated for the intended repository scope. | ||
|
|
||
| ## Scaling with runtime and memory usage | ||
|
|
||
| For `nlohmann/json`, parsing complexity is input-dependent. [The nlohmann/json documentation](https://json.nlohmann.me/api/basic_json/parse/#complexity) states that parsing is linear in the length of the input, and may be higher depending on callback complexity and input access characteristics. [Memory usage](https://json.nlohmann.me/api/basic_json/) also depends on the stored JSON structure and value representation, as the library uses default container/value types such as std::string, std::map, and std::vector. | ||
|
|
||
| ## Maximum runtime and timeout criterion | ||
|
|
||
| There is currently no repository-level timeout criterion defined for the runtime of `nlohmann/json` in deployed use. Instead, this is explicitly treated as an integrator responsibility in AOU-31. | ||
|
|
||
| More specifically: | ||
|
|
||
|
aschemmel-tech marked this conversation as resolved.
|
||
| - This repository defines CI- and process-based indicators, such as coverage and pull-request count gates. | ||
| - This repository does not define a numeric runtime limit, latency budget, or automatic timeout threshold for arbitrary JSON processing in production use. | ||
| - AOU-31 requires the integrator to define resource and time budgets for JSON parsing and validation, including maximum input size, maximum nesting or structure complexity, and maximum processing time. | ||
| - AOU-31 also requires suitable enforcement mechanisms at the integration boundary to prevent hangs or resource exhaustion when processing untrusted or extreme inputs. | ||
|
|
||
| This is consistent with the existing TSF scope: If a consuming system needs a maximum runtime criterion or timeout threshold, that threshold must be defined and enforced by the integrator in the target environment rather than by this repository alone. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.