Reading end-of-statement-OK packets with info throws `trilogy_read_row: TRILOGY_TRUNCATED_PACKET`.

[jdelStrother]

Hi there - I'm interested in using Trilogy to talk to manticore, which publishes a mysql-compatible interface.

It works ok for trivial queries (SELECT 1), but longer result-sets throw trilogy_read_row: TRILOGY_TRUNCATED_PACKET.

From what I can tell (heavily aided by Claude, so take this with a grain of salt), it's because Manticore's end-of-statement OK packet includes szMeta in the info field, which is a human-readable string containing, eg, "3 out of >=24 results in 0ms". This is > 8 bytes, so trilogy_read_row doesn't think it's an EOF marker.

With regular mysql-server, end-of-statement OK packets are sent with nullptr in the info field, which is why trilogy handles them fine.

Claude thinks this should be fixed in manticore (by changing SendMysqlEofPacket to send NULL info in place of szMeta), I'm not so sure. The regular mysql client handles this fine, and the mysql docs seem to imply that any OK packets might include an info field with arbitrary length, even if they're being used for an end-of-statement marker.

WDYT? I can file this in manticore if you think it's a bug on their end.

Claude's writeup, with some rambling, packet captures, and possible fixes for manticore and/or trilogy

Bug: Manticore sends oversized EOF packet, breaking trilogy

Symptom

trilogy_read_row: TRILOGY_TRUNCATED_PACKET (Trilogy::QueryError)

Raised by trilogy when reading query results from Manticore Search (port 9306). The same query succeeds with mysql2.

Root cause

The bug is in Manticore. Both clients receive identical bytes; mysql2 tolerates the protocol violation, trilogy does not.

Background: CLIENT_DEPRECATE_EOF

Both Manticore and trilogy advertise and negotiate CLIENT_DEPRECATE_EOF (capability flag 0x01000000) during the handshake. Under this protocol:

• No EOF packet is sent between the column definitions and the row data.
• The final end-of-rows marker still starts with 0xFE — the header byte does not change. What changes is the body format: OK layout (affected_rows, last_insert_id, status flags, warnings) rather than legacy EOF layout (warnings, status flags only). MySQL server achieves this by calling net_send_ok(..., eof_identifier=true), which selects 0xFE as the header even though it's sending an OK-format body (mysql-server/sql/protocol_classic.cc lines 887–893, 1323–1327).
• Since 0xFE is also the length-encoded prefix for an 8-byte integer in a result row, the packet must stay under 9 bytes total so clients can tell it apart from row data.

What Manticore actually sends

Manticore correctly omits the intermediate EOF between columns and rows, but then appends a query-stats info string to the final 0xFE end-of-rows packet, inflating it well past the 9-byte limit:

Packet seq=16, len=44:
fe 00 00 02 00 00 00  24 2d 2d 2d 20 33 20 6f 75 74 20 6f 66 ...
│  ├─────┤ ├─────┤   └─ "$--- 3 out of >=24 results in 0ms ---" (39 bytes)
│  warn=0  status=2
└─ 0xFE (EOF marker)

A valid EOF packet (with CLIENT_PROTOCOL_41) is exactly 5 bytes. Manticore's is 44 bytes.

Why trilogy fails

The MySQL protocol allows result rows to start with 0xFE — it means a length-encoded 8-byte integer follows. The MySQL server source documents this directly:

2^24 | 2^64 | 0xFE + 8-byte integer

Warning: If the first byte of a packet is a length-encoded integer and its byte value is 0xFE, you must check the length of the packet to verify that it has enough space for a 8-byte integer. If not, it may be an EOF_Packet instead.

— mysql-server/sql/protocol_classic.cc lines 98, 103–107

To distinguish a real EOF packet from such a row, trilogy (like most MySQL clients) uses the heuristic:

// trilogy/src/client.c, trilogy_read_row()
if (current_packet_type(conn) == TRILOGY_PACKET_EOF && conn->packet_buffer.len < 9) {
    // treat as EOF / end of results
}

A genuine EOF packet is at most 5 bytes, so < 9 is a reliable test. This is both documented and implemented in the MySQL server source:

These rules distinguish whether the packet represents OK or EOF:

• OK: header = 0 and length of packet > 7
• EOF: header = 0xfe and length of packet < 9

— mysql-server/sql/protocol_classic.cc lines 717–719

Warning: The EOF_Packet packet may appear in places where a Protocol::LengthEncodedInteger may appear. You must check whether the packet length is less than 9 to make sure that it is a EOF_Packet packet.

— mysql-server/sql/protocol_classic.cc lines 993–997

The 5-byte maximum comes from the EOF packet's fixed structure (CLIENT_PROTOCOL_41): 1 header byte + 2 warnings + 2 status flags = 5 bytes, implemented in MySQL server as a literal uchar buff[5]:

// mysql-server/sql/protocol_classic.cc, write_eof_packet(), lines 1084–1106
uchar buff[5];
buff[0] = 254;           // 0xFE header
int2store(buff + 1, tmp);         // 2-byte warning count
int2store(buff + 3, server_status); // 2-byte status flags
error = my_net_write(net, buff, 5);

Manticore's packet is 44 bytes, so trilogy falls through and tries to parse it as a row:

1. Reads byte 0xFE → length-encoded integer, next 8 bytes are the length
   (trilogy's lenenc parser: trilogy/src/reader.c lines 176–177: case 0xfe: return trilogy_reader_get_uint64(reader, out))
2. Reads 00 00 02 00 00 00 00 24 as a uint64 → 3,252,724,830,868,471,808
3. That length vastly exceeds the remaining bytes → TRILOGY_TRUNCATED_PACKET

Why mysql2 succeeds

mysql2 is backed by libmysqlclient, which uses a completely different heuristic split by whether CLIENT_DEPRECATE_EOF is negotiated (sql-common/client.cc, cli_safe_read, lines 1279–1293):

// DEPRECATE_EOF path: 0xFE is treated as an OK/EOF marker unless the packet
// exceeds MAX_PACKET_LENGTH (16MB), which would mean it's a genuine multi-packet row
if ((mysql->server_capabilities & CLIENT_DEPRECATE_EOF) &&
    (net->read_pos[0] == 254)) {
    if (len > MAX_PACKET_LENGTH) return len;  // huge data packet, keep reading
    if (is_data_packet) *is_data_packet = false;  // it's an EOF/OK marker
    if (parse_ok) read_ok_ex(mysql, len);
    return len;
}

// Legacy path (no DEPRECATE_EOF): 0xFE is EOF only if < 8 bytes
if (!(mysql->server_capabilities & CLIENT_DEPRECATE_EOF) &&
    (net->read_pos[0] == 254) && (len < 8)) {
    if (is_data_packet) *is_data_packet = false;
}

When is_data_packet is set to false, read_one_row_complete (line 3115) exits with end-of-data and parses the packet body as OK:

if (net->read_pos[0] != 0x00 && !is_data_packet) {
    if (mysql->server_capabilities & CLIENT_DEPRECATE_EOF)
        read_ok_ex(mysql, pkt_len);  // parse 0xFE packet body as OK
    return 1;  /* End of data */
}

With CLIENT_DEPRECATE_EOF active, libmysqlclient treats any 0xFE packet under 16MB as the end-of-rows marker — so Manticore's 44-byte packet is handled without error. Trilogy applies the stricter < 9 check from the MySQL spec regardless of whether CLIENT_DEPRECATE_EOF is set.

Client	CLIENT_DEPRECATE_EOF threshold	Legacy threshold
libmysqlclient / mysql2	len ≤ MAX_PACKET_LENGTH (16MB)	len < 8
trilogy	len < 9	len < 9

Evidence

Packets captured via TCP proxy (trilogy and mysql2 receive identical bytes from Manticore):

Seq	Len	Type	Notes
0	92	Handshake	Server version "0.0.0", capabilities 0x011a8228 incl. CLIENT_DEPRECATE_EOF
1	60	Auth (client→server)	Client capabilities 0x018aa200 incl. CLIENT_DEPRECATE_EOF
2	7	OK	Auth success
1	1	Result	Column count = 11 (0x0b)
2–12	35–59	Column defs	11 column definition packets
(no intermediate EOF)			Correct per CLIENT_DEPRECATE_EOF
13	99	Row	Row 1
14	98	Row	Row 2
15	102	Row	Row 3
16	44	0xFE	Correct marker byte, but 44 bytes due to appended stats string — must be < 9

Fix

In Manticore (src/netreceive_ql.cpp, SendMysqlEofPacket): when CLIENT_DEPRECATE_EOF is negotiated, the end-of-rows packet body must use OK format but must not include the meta/info string, so the total stays under 9 bytes (7 bytes: 0xFE + affected_rows + last_insert_id + status + warnings). The stats string --- N out of M results in Xms --- must be dropped from this packet.

MySQL server's own Protocol_classic::send_eof confirms this: when CLIENT_DEPRECATE_EOF is set it calls net_send_ok(..., nullptr, true) — explicitly passing nullptr for the message so the info field is never written (mysql-server/sql/protocol_classic.cc lines 1323–1327). The info field in the OK packet spec is intended for DML responses (INSERT/UPDATE/DELETE), not for end-of-rows markers.

Without the meta string the final packet is 7 bytes — under the 9-byte threshold — and clients recognise it correctly as end-of-rows:

fe 00 00 02 00 00 00
│  │     │
│  │     └─ status_flags (2 bytes)
│  └─ affected_rows=0, last_insert_id=0 (1 byte each, lenenc)
└─ 0xFE marker

Possible workaround in trilogy: replace the < 9 threshold with <= TRILOGY_MAX_PACKET_LEN (i.e. <= 0xFFFFFF = 16,777,215 bytes). This matches libmysqlclient's CLIENT_DEPRECATE_EOF behaviour and tolerates Manticore's non-standard 44-byte packet without introducing any false positives.

Why <= TRILOGY_MAX_PACKET_LEN is safe

The concern with raising the threshold is accidentally treating a genuine 0xFE-prefixed row as an EOF. But this cannot happen:

A text-protocol result row whose first column begins with 0xFE is encoding a length-encoded integer for a value ≥ 2²⁴ (= 16,777,216). The encoded form is 0xFE (1 byte) + 8-byte uint64. The column value itself must therefore be at least 2²⁴ bytes long. Minimum total row size:

1 byte (0xFE prefix) + 8 bytes (length) + 2²⁴ bytes (value) = 16,777,225 bytes

TRILOGY_MAX_PACKET_LEN is 0xFFFFFF = 16,777,215. A single MySQL wire packet cannot exceed this — larger payloads are split across multiple fragment packets and reassembled. After trilogy's fragment reassembly (packet_parser.c, on_packet_begin is skipped for continuation packets), conn->packet_buffer.len for such a row is the reassembled length of at least 16,777,225 bytes, which always exceeds TRILOGY_MAX_PACKET_LEN.

Therefore:

packet_buffer.len	Meaning
≤ TRILOGY_MAX_PACKET_LEN (16,777,215)	Cannot be a 0xFE-prefixed row — safe to treat as EOF
> TRILOGY_MAX_PACKET_LEN	Must be a large data row

The three call sites in trilogy/src/client.c (trilogy_read_row line 985, trilogy_drain_results line 1020, trilogy_stmt_read_row line 1262) would change from:

if (current_packet_type(conn) == TRILOGY_PACKET_EOF && conn->packet_buffer.len < 9) {

to:

if (current_packet_type(conn) == TRILOGY_PACKET_EOF && conn->packet_buffer.len <= TRILOGY_MAX_PACKET_LEN) {

This is exactly how libmysqlclient arrives at the same guarantee from the other direction: it checks the raw (pre-reassembly) packet length against MAX_PACKET_LENGTH (16MB) before any fragment reassembly occurs. Both approaches exploit the same mathematical constraint — a 0xFE-prefixed row can never fit in a single 16MB packet.

[byroot]

Yeah, sounds like something trilogy should handle. The annoying part would be having test coverage for it.

[composerinteralia]

From the MySQL docs:

These rules distinguish whether the packet represents OK or EOF:

OK: header = 0 and length of packet >= 7
EOF: header = 0xfe and length of packet < 8

[composerinteralia]

There's some potential for mixing up an EOF with a large length-encoded integer if you don't check the length:

If the first byte of a packet is a length-encoded integer and its byte value is 0xFE, you must check the length of the packet to verify that it has enough space for a 8-byte integer. If not, it may be an EOF_Packet instead.

See #179, for example

[borama]

Hello, we also hit this issue and for us it is actually a regression since the 2.10.0 Trilogy version. I suspect it is related to #179.

We have Manticore search integrated into Rails using the Trilogy connector (and a custom-made Rails connection adapter) and all was good up to Trilogy 2.9. Trivial responses tend to work OK even on later versions, but other queries end up with a TRILOGY_TRUNCATED_PACKET error.

I made a minimalistic script to prove the regression with latest Manticore Search (17.5.1):

# trilogy_test.rb

require "trilogy"

config = { host: "127.0.0.1", port: 9306, username: "", password: "", ssl: false }
client = Trilogy.new config

p client.query("DROP TABLE IF EXISTS test")
p client.query("CREATE TABLE test (some_column TEXT)")
p client.query("INSERT INTO test (some_column) VALUES ('test value')")
client.query("SELECT * FROM test").each { |row| p row }

In Trilogy 2.9, all works OK:

$ bundle exec gem list | grep trilogy
trilogy (2.9.0)

$ bundle exec ruby trilogy_test.rb
#<Trilogy::Result:0x00000001212c4328 @fields=[], @rows=[], @query_time=0.000869, @last_insert_id=0, @affected_rows=0>
#<Trilogy::Result:0x0000000121a4caa0 @fields=[], @rows=[], @query_time=0.001046, @last_insert_id=0, @affected_rows=0>
#<Trilogy::Result:0x0000000121a4ca00 @fields=[], @rows=[], @query_time=0.000626, @last_insert_id=1228634930633373830, @affected_rows=1>
[1228634930633373830, "test value"]

In Trilogy 2.10 (and later versions) however, an error is raised while grabbing the SELECT result:

$ bundle exec gem list | grep trilogy
trilogy (2.10.0)

$ bundle exec ruby trilogy_test.rb
#<Trilogy::Result:0x0000000123772a80 @fields=[], @rows=[], @query_time=0.000931, @last_insert_id=0, @affected_rows=0>
#<Trilogy::Result:0x0000000123f1cc18 @fields=[], @rows=[], @query_time=0.001508, @last_insert_id=0, @affected_rows=0>
#<Trilogy::Result:0x0000000123f1cb78 @fields=[], @rows=[], @query_time=0.000547, @last_insert_id=1228634930633373831, @affected_rows=1>
trilogy_test.rb:9:in 'Trilogy#query': trilogy_read_row: TRILOGY_TRUNCATED_PACKET (Trilogy::QueryError)
        from trilogy_test.rb:9:in '<main>'

For a comparison, I made the same test with latest mysql2 gem and all seems to work OK:

# mysql_test.rb

require "mysql2"

config = { host: "127.0.0.1", port: 9306, username: "", password: "", ssl: false }
client = Mysql2::Client.new config

p client.query("DROP TABLE IF EXISTS test")
p client.query("CREATE TABLE test (some_column TEXT)")
p client.query("INSERT INTO test (some_column) VALUES ('test value')")
client.query("SELECT * FROM test").each { |row| p row }

$ bundle exec gem list | grep mysql2
mysql2 (0.5.7)

$ bundle exec ruby mysql_test.rb
nil
nil
nil
{"id" => 1228634930633373832, "some_column" => "test value"}

If anybody has some pointers how to dig deeper, I will be glad to test/debug them.

[byroot]

I would say the main blocker is testing / test coverage. If we could integrate manticore in the CI, it would help ensure whatever change we make to fix support of it is covered by test and won't regress.

Other than that the fix isn't too hard I think.

[borama]

Thanks, @byroot. As adding Manticore to the CI seemed very tedious to me at first glance, I spent some time trying to trick MySQL into replicating the issue instead. And although I failed, I believe it got me a bit closer to the core of it.

The communication seems to error out when both of the following conditions apply: 1) the OK packet contains the info human-readable status information and 2) the OK packet follows row data.

While MySQL does use the info field for status information in some DML queries such as INSERT INTO ... SELECT ..., it seems to never do so when returning row data (i.e. in responses to SELECT queries, for example).

On the contrary, Manticore returns the info message in every SELECT response, after sending row data:

mysql> select * from test;
+---------------------+-------------+
| id                  | some_column |
+---------------------+-------------+
| 1228634930633373836 | test value  |
+---------------------+-------------+
1 row in set (0.006 sec)
--- 1 out of 1 results in 0ms ---      <---- THIS is the info message

The above result inspected in Wireshark:

Now, when Trilogy processes row data, it recognizes an OK packet only when its length is less than 9 Bytes which is not the case when the info message is present. This condition is, in practice, fine for MySQL (and MariaDB) as these servers seem to never send info after row data. But for the Manticore server, I believe it would need to be a bit more elaborate.

Coincidentally, I found this fix in the sqlx rust library which I think handles precisely the same situation as this issue in Trilogy.

Now, with this info, do you still think adding Manticore to the CI would be necessary to proceed further? I was thinking maybe some unit tests with mock up response data might work too but I know nothing about the standards here and understand the risks of such approach. Still, maybe the question is whether the test should rather be a test of the coverage of a detail in the protocol specification, or a test of successful communication with a real-world, but lesser-known, specific server.

[byroot]

Now, with this info, do you still think adding Manticore to the CI would be necessary to proceed further?

I know nothing of manticore, If it's really too much trouble, a mocked test will do.

[jdelStrother]

We could spin up a manticore docker instance in CI, if that isn't viewed as too heavyweight for what must be a fairly niche use-case.

[byroot]

Whichever is simpler.

[borama]

Manticore is a fast search database, somewhat similar to Elasticsearch in principle but preferring an SQL interface over REST API. It implements the MySQL protocol so that any MySQL client can be used. But otherwise the SQL dialect is quite different from MySQL and implements only a subset of commands and functions available in MySQL. Manticore also for example has no concept of "databases" (supports just one) or user access control, has only a subset of data types, etc.

I think we would quickly get into a lot of trouble if we tried to run all the tests here against Manticore as the interface is just too different from MySQL/MariaDB, even though all three speak the MySQL protocol.

I'll try to have a look at the code and the unit tests to see if I can make trilogy read Manticore results properly.

[jdelStrother]

I think we would quickly get into a lot of trouble if we tried to run all the tests here against Manticore

Oh for sure - I was envisaging just a few extra tests that checked some basic functionality if, eg, ENV["MANTICORE_HOST"] was set.

I don't have a strong opinion on doing that vs just checking the handling of some mocked responses though

[borama]

In #281 I suggest a little more precise logic for differentiating between EOF packets and huge row data packets. What do you think? I am also still looking for ways to test this. Thanks.

[composerinteralia]

Looks good to me. No strong opinions on testing from me—testing the behavior of is_eof_packet directly might be sufficient.