fix: Avoid overflows when reading json inputs by whoisj · Pull Request #8676 · triton-inference-server/server

whoisj · 2026-02-24T23:21:06Z

Check input from HTTP+JSON sizes before allocating memory for them.

Protect against integer overflows while we're at it.

mudit-eng · 2026-02-26T19:53:38Z

LGTM

mattwittwer · 2026-02-27T18:51:31Z

One edge case:
max_input_size_ is checked per-input rather than cumulatively, allowing up to a 4x memory limit bypass.
This PR checks byte_size > max_input_size_ to prevents excessive memory allocation. However, because this check is evaluated on individual inputs rather than the cumulative total of the entire HTTP request, a single request can still cause the server to allocate up to 4x the memory limit.

The scenario:
The upstream EVBufferToJson limits the physical size of the incoming HTTP JSON payload to max_input_size_.
However, the most compact JSON representation of an element is 0, (2 bytes), while an FP64 element allocates 8 bytes in memory.
With a limit set to --http-max-input-size=128000000 (128 MB), a single 128 MB JSON payload formatted like this can allocate 512MB:

{
  "parameters": {
    "in_1": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]},
    "in_2": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]},
    "in_3": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]},
    "in_4": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]}
  }
}

What happens:
The upstream check passes because the total HTTP payload is exactly 128 MB.
ExactMappingInput parses in_1. The calculated byte_size is 128 MB. The check if (byte_size > max_input_size_) passes. Triton allocates 128 MB. This repeats for in_2, in_3, and in_4. Resulting in the server allocating 512 MB of memory for a single request, bypassing the 128 MB max_input_size_ limit.

yinggeh

LGTM. Though I think it's a bit of over-engineering here.

whoisj · 2026-02-27T20:12:32Z

max_input_size_ is checked per-input rather than cumulatively, allowing up to a 4x memory limit bypass.

Agreed about exceeding the overall limit. I'll create a follow up to address that separately. Thanks for pointing it out.

src/http_server.cc

yinggeh · 2026-03-04T10:56:56Z

PR title isn't following the convention.

yinggeh · 2026-03-04T10:59:43Z

I think test is required for such code change. Whether verify its functionality with existing ones or by creating a new test.

qa/common/gen.ONNXRuntime.gen_qa_model_repository.docker.v2.sh

whoisj · 2026-03-06T01:43:45Z

I think test is required for such code change. Whether verify its functionality with existing ones or by creating a new test.

Added 2 new tests. Let me know if they look good to you.

yinggeh · 2026-03-06T16:57:43Z

qa/L0_http/http_input_size_limit_test.py

+        )
+        error_msg = response.content.decode()
+        self.assertEqual(
+            '{"error":"Request JSON size of 89478576 bytes exceeds the maximum allowed value of 67108864 bytes. Use --http-max-input-size to increase the limit."}',


Would like to see tests that passing with size 67108864 but failed with 67108864+1.

changed the format.

yinggeh · 2026-03-06T16:58:51Z

qa/L0_http/test.sh

 fi
+
+# Run test to verify that large inputs fail with default limit
+python http_input_size_limit_test.py InferSizeLimitTest.test_type_size_explosion >> $CLIENT_LOG 2>&1


I am confused with the naming here. Can you use a more descriptive name?

do you have a suggestion?

qa/L0_http/generate_endpoint_test.py

whoisj · 2026-03-19T22:03:30Z

Converted to a draft after hitting another SEGFAULT bug. Will reopen once the tests are clean.

mudit-eng · 2026-03-24T06:09:50Z

src/http_server.cc

          std::vector<char>& serialized = infer_req->serialized_data_.back();
          serialized.resize(byte_size);
-
+          char* serialized_base = &serialized[0];


&serialized[0] is undefined behavior if vector is empty. Safer to do something like:

char* serialized_base = serialized.data();

or

char* serialized_base = serialized.empty() ? nullptr : serialized.data();

sure. I was just using the existing code, but I'll fix up regardless.

mudit-eng · 2026-03-24T06:17:57Z

src/http_server.h

    const MappingSchema* request_schema_{nullptr};
    const MappingSchema* response_schema_{nullptr};
    const bool streaming_{false};
+    const size_t max_input_size_{0};


HTTPAPIServer already has protected member:size_t max_input_size_;. Can this create confusion?

I don't see how, they're both private fields.

mudit-eng · 2026-03-24T06:21:28Z

src/http_server.cc

-         " bytes. Use --http-max-input-size to increase the limit.")
+        ("Request JSON size of " + std::to_string(length) + " + " +
+         std::to_string(overrun) +
+         " bytes exceeds the maximum allowed input size. "


Not sure if this message is intuitive. This does not convey what is the maximum input size.

the updated format was a request from @yinggeh. Can you two decide between you which you prefer?

mudit-eng · 2026-03-24T06:23:28Z

src/http_server.cc

    const std::string& name,
    triton::common::TritonJson::Value& generate_request,
-    std::map<std::string, triton::common::TritonJson::Value>& input_metadata)
+    std::map<std::string, triton::common::TritonJson::Value>& input_metadata,


Can we use unordered_map here?

no idea, but that would be changing existing code and I'd rather not because I feel that is outside the scope of this PR.

mudit-eng · 2026-03-24T06:33:27Z

src/http_server.h

    const MappingSchema* RequestSchema() { return request_schema_; }
    const MappingSchema* ResponseSchema() { return response_schema_; }

+    size_t MaxInputSize() { return max_input_size_; }


this is not being called anywhere

I'll remove.

mudit-eng · 2026-03-24T06:37:42Z

src/http_server.cc

+    // allowed input size.
+    if (byte_size + consumed_input_byte_size > max_input_size_ ||
+        byte_size + consumed_input_byte_size < consumed_input_byte_size) {
+      auto overrun = byte_size + consumed_input_byte_size - max_input_size_;


Can byte_size + consumed_input_byte_size result in an overflow?

no because of the check immediately previous.

Yes, you checked for overflow here: byte_size + consumed_input_byte_size < consumed_input_byte_size) but can't there be an overflow here: auto overrun = byte_size + consumed_input_byte_size - max_input_size_

whoisj requested review from mattwittwer, pskiran1 and yinggeh February 24, 2026 23:21

whoisj added the PR: fix A bug fix label Feb 24, 2026

yinggeh approved these changes Feb 27, 2026

View reviewed changes

yinggeh requested changes Feb 27, 2026

View reviewed changes

src/http_server.cc Show resolved Hide resolved

whoisj requested a review from yinggeh March 2, 2026 22:09

whoisj force-pushed the jwyman/tri-673 branch 2 times, most recently from c603e3d to 75c3c95 Compare March 3, 2026 00:09

yinggeh reviewed Mar 4, 2026

View reviewed changes

src/http_server.cc Show resolved Hide resolved

src/http_server.cc Show resolved Hide resolved

whoisj requested a review from yinggeh March 4, 2026 20:03

yinggeh reviewed Mar 4, 2026

View reviewed changes

qa/common/gen.ONNXRuntime.gen_qa_model_repository.docker.v2.sh Outdated Show resolved Hide resolved

whoisj changed the title ~~Avoid overflows when reading json inputs~~ fix: Avoid overflows when reading json inputs Mar 5, 2026

whoisj force-pushed the jwyman/tri-673 branch from 2586e7c to a48058d Compare March 6, 2026 01:44

whoisj requested a review from yinggeh March 6, 2026 01:48

yinggeh reviewed Mar 6, 2026

View reviewed changes

whoisj force-pushed the jwyman/tri-673 branch 4 times, most recently from 3f045a5 to 52f656f Compare March 9, 2026 15:49

github-advanced-security bot found potential problems Mar 9, 2026

View reviewed changes

qa/L0_http/generate_endpoint_test.py Fixed Show fixed Hide fixed

whoisj force-pushed the jwyman/tri-673 branch 2 times, most recently from db78671 to a67598d Compare March 11, 2026 20:39

whoisj force-pushed the jwyman/tri-673 branch from b2d0437 to a7994f5 Compare March 13, 2026 15:34

whoisj added 3 commits March 16, 2026 17:32

Merge remote-tracking branch 'src/main' into jwyman/tri-673

8deadbf

Merge branch 'main' into jwyman/tri-673

b94b60d

catch & handle null pointer conditions

dcac291

whoisj marked this pull request as draft March 19, 2026 22:02

whoisj force-pushed the jwyman/tri-673 branch 5 times, most recently from e793a58 to 77b3f26 Compare March 20, 2026 20:12

avoid adding zero sized buffer to TRITONSERVER_InferenceRequestAddInput

edb8aef

whoisj force-pushed the jwyman/tri-673 branch from 77b3f26 to edb8aef Compare March 20, 2026 22:29

whoisj marked this pull request as ready for review March 23, 2026 16:08

whoisj force-pushed the jwyman/tri-673 branch from 68df00c to 6df85d5 Compare March 23, 2026 16:09

whoisj requested a review from yinggeh March 23, 2026 16:09

fix test

98eabf9

whoisj force-pushed the jwyman/tri-673 branch 2 times, most recently from d864e63 to fcaabcf Compare March 23, 2026 20:08

print error when error

3d21dc1

whoisj force-pushed the jwyman/tri-673 branch from fcaabcf to 3d21dc1 Compare March 23, 2026 20:17

revert some error message changes

defd24a

whoisj force-pushed the jwyman/tri-673 branch from cec7b6e to b183ae7 Compare March 23, 2026 21:29

fix test match

5efc58b

whoisj force-pushed the jwyman/tri-673 branch from b183ae7 to 5efc58b Compare March 23, 2026 21:39

make pre-commit happy

8428658

whoisj requested a review from mudit-eng March 23, 2026 21:43

mudit-eng reviewed Mar 24, 2026

View reviewed changes

react to @mudit-eng comments

4b99ac0

whoisj requested a review from mudit-eng March 25, 2026 16:45

Conversation

whoisj commented Feb 24, 2026

Uh oh!

mudit-eng commented Feb 26, 2026

Uh oh!

mattwittwer commented Feb 27, 2026

Uh oh!

yinggeh left a comment

Choose a reason for hiding this comment

Uh oh!

whoisj commented Feb 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yinggeh commented Mar 4, 2026

Uh oh!

yinggeh commented Mar 4, 2026

Uh oh!

Uh oh!

whoisj commented Mar 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

whoisj commented Mar 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants