Complement `TestMessagesOverFederation/Backfill_from_nearby_backward...` flakes

[gamesguru]

Description

Seems relatively uncommon, like 1 in 20 or 1 in 50 runs?

Complement (workers, Postgres)

https://github.com/gamesguru/synapse/actions/runs/28653363967/job/84976804375

##[group]  ❌ TestMessagesOverFederation [0;37m (1.14s) 
    client.go:845: [CSAPI] POST hs1/_matrix/client/v3/register => 200 OK (73.188942ms)
    client.go:845: [CSAPI] POST hs2/_matrix/client/v3/register => 200 OK (46.100422ms)
    client.go:845: [CSAPI] POST hs1/_matrix/client/v3/createRoom => 200 OK (282.429961ms)
    client.go:845: [CSAPI] PUT hs1/_matrix/client/v3/rooms/!DNMWjflWBcLhfRDdlp:hs1/send/m.room.message/1 => 200 OK (81.551654ms)
    client.go:845: [CSAPI] GET hs1/_matrix/client/v3/sync => 200 OK (75.180093ms)
    client.go:845: [CSAPI] PUT hs1/_matrix/client/v3/rooms/!DNMWjflWBcLhfRDdlp:hs1/send/m.room.message/2 => 200 OK (41.251558ms)
    client.go:845: [CSAPI] GET hs1/_matrix/client/v3/sync => 200 OK (31.586532ms)
    client.go:845: [CSAPI] PUT hs1/_matrix/client/v3/rooms/!DNMWjflWBcLhfRDdlp:hs1/send/m.room.message/3 => 200 OK (38.6272ms)
    client.go:845: [CSAPI] GET hs1/_matrix/client/v3/sync => 200 OK (353.036166ms)
    client.go:845: [CSAPI] POST hs2/_matrix/client/v3/join/!DNMWjflWBcLhfRDdlp:hs1 => 502 Bad Gateway (24.256145ms)
##[endgroup]
##[group]  ❌ TestMessagesOverFederation/Backfill_from_nearby_backward_extremities_past_token [0;37m (930ms) 
    room_messages_test.go:108: SendEventSynced waiting for event ID $25kj8hYM6-KCda4-T98-zNBtPb3AdYL6xbV-3eJo9mI
    room_messages_test.go:108: SendEventSynced waiting for event ID $ntaMvAVN5ta_33FvoaXHtxvZPf-v1JnR3Np-SBuC0_c
    room_messages_test.go:108: SendEventSynced waiting for event ID $kXZxbVizLVdJi22YvrRrF5MnZqXALkRjsdugzkYiN7I
    room_messages_test.go:111:   CSAPI.Must: POST http://127.0.0.1:32953/_matrix/client/v3/join/%21DNMWjflWBcLhfRDdlp:hs1?server_name=hs1 returned non-2xx code: 502 Bad Gateway - body: {"errcode":"M_UNKNOWN","error":"Failed to make_join via any server"}
2026/07/03 10:34:29 ============================================

---

Also affects TestJoinViaRoomIDAndServerName, see: https://github.com/gamesguru/complement/actions/runs/28673516291/job/85041912970?pr=4

❌ TestJoinViaRoomIDAndServerName (3.27s)
      client.go:860: [CSAPI] POST hs1/_matrix/client/v3/register => 200 OK (24.633702ms)
      server.go:552: ListenFederationServer: ServeTLS failed: tls: failed to find any PEM data in certificate input
      client.go:860: [CSAPI] GET hs1/_matrix/client/v3/capabilities => 200 OK (4.895969ms)
      server.go:210: Creating room !0-Jxe3ns4IVdgSrwJ155:host.docker.internal:34409 with version 10
      client.go:860: [CSAPI] POST hs1/_matrix/client/v3/join/!0-Jxe3ns4IVdgSrwJ155:host.docker.internal:34409 => 502 Bad Gateway (313.660229ms)
      federation_room_join_test.go:76: CSAPI.Must: POST http://127.0.0.1:32823/_matrix/client/v3/join/%210-Jxe3ns4IVdgSrwJ155:host.docker.internal:34409?server_name=host.docker.internal%3A34409 returned non-2xx code: 502 Bad Gateway - body: {"errcode":"M_UNKNOWN","error":"Failed to make_join via any server"}
...
$qxxaZk6XrhdyTSsYUnZKeQC7RekcetZRDGSEgGTr170 - {POST-O-40} [host.docker.internal:34747] Completed request: 200 OK in 0.00 secs, got 779 bytes - POST matrix-federation://host.docker.internal:34747/_matrix/federation/v1/get_missing_events/%210-ZN5pf5GgVny6qMs7ou%3Ahost.docker.internal%3A34747
2026-07-03T16:56:25.5918895Z synapse_main | 2026-07-03 16:53:33,227 - synapse.federation.federation_server - 1405 - ERROR - _process_incoming_pdus_in_room_inner-4 - Failed to handle PDU $qxxaZk6XrhdyTSsYUnZKeQC7RekcetZRDGSEgGTr170
2026-07-03T16:56:25.5919034Z synapse_main | Traceback (most recent call last):
2026-07-03T16:56:25.5919441Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/federation_event.py", line 327, in on_receive_pdu
2026-07-03T16:56:25.5919600Z synapse_main |     await self._get_missing_events_for_pdu(
2026-07-03T16:56:25.5919739Z synapse_main |         origin, pdu, prevs, min_depth
2026-07-03T16:56:25.5919832Z synapse_main |     )
2026-07-03T16:56:25.5920203Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
2026-07-03T16:56:25.5920347Z synapse_main |     return await func(*args, **kwargs)
2026-07-03T16:56:25.5920472Z synapse_main |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-03T16:56:25.5920925Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/federation_event.py", line 833, in _get_missing_events_for_pdu
2026-07-03T16:56:25.5921163Z synapse_main |     missing_events = await self._federation_client.get_missing_events(
2026-07-03T16:56:25.5921331Z synapse_main |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-03T16:56:25.5921431Z synapse_main |     ...<7 lines>...
2026-07-03T16:56:25.5921523Z synapse_main |     )
2026-07-03T16:56:25.5921608Z synapse_main |     ^
2026-07-03T16:56:25.5922045Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/federation/federation_client.py", line 1575, in get_missing_events
2026-07-03T16:56:25.5922414Z synapse_main |     events = parse_events_from_pdu_json(
2026-07-03T16:56:25.5922550Z synapse_main |         content.get("events", []),
2026-07-03T16:56:25.5922654Z synapse_main |         room_version,
2026-07-03T16:56:25.5922786Z synapse_main |         received_time=received_time,
2026-07-03T16:56:25.5922875Z synapse_main |     )
2026-07-03T16:56:25.5923442Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/federation/federation_base.py", line 315, in parse_events_from_pdu_json
2026-07-03T16:56:25.5923703Z synapse_main |     event_from_pdu_json(pdu_json, room_version, received_time=received_time)
2026-07-03T16:56:25.5923893Z synapse_main |     ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-03T16:56:25.5924324Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/federation/federation_base.py", line 373, in event_from_pdu_json
2026-07-03T16:56:25.5924463Z synapse_main |     validate_canonicaljson(pdu_json)
2026-07-03T16:56:25.5924587Z synapse_main |     ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
2026-07-03T16:56:25.5924978Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/events/utils.py", line 818, in validate_canonicaljson
2026-07-03T16:56:25.5925104Z synapse_main |     validate_canonicaljson(v)
2026-07-03T16:56:25.5925208Z synapse_main |     ~~~~~~~~~~~~~~~~~~~~~~^^^
2026-07-03T16:56:25.5925589Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/events/utils.py", line 818, in validate_canonicaljson
2026-07-03T16:56:25.5925704Z synapse_main |     validate_canonicaljson(v)
2026-07-03T16:56:25.5925811Z synapse_main |     ~~~~~~~~~~~~~~~~~~~~~~^^^
2026-07-03T16:56:25.5926187Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/events/utils.py", line 814, in validate_canonicaljson
2026-07-03T16:56:25.5926417Z synapse_main |     raise SynapseError(400, "Bad JSON value: float", Codes.BAD_JSON)
2026-07-03T16:56:25.5926645Z synapse_main | synapse.api.errors.SynapseError: 400: Bad JSON value: float
2026-07-03T16:56:25.5926736Z synapse_main | 
2026-07-03T16:56:25.5926974Z synapse_main | The above exception was the direct cause of the following exception:
2026-07-03T16:56:25.5927064Z synapse_main | 
2026-07-03T16:56:25.5927196Z synapse_main | Traceback (most recent call last):
2026-07-03T16:56:25.5927700Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/federation/federation_server.py", line 1395, in _process_incoming_pdus_in_room_inner
2026-07-03T16:56:25.5927894Z synapse_main |     await self._federation_event_handler.on_receive_pdu(
2026-07-03T16:56:25.5927999Z synapse_main |         origin, event
2026-07-03T16:56:25.5928087Z synapse_main |     )
2026-07-03T16:56:25.5928487Z synapse_main |   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/federation_event.py", line 331, in on_receive_pdu
2026-07-03T16:56:25.5928597Z synapse_main |     raise Exception(
2026-07-03T16:56:25.5928696Z synapse_main |     ...<2 lines>...
2026-07-03T16:56:25.5928797Z synapse_main |     ) from e
2026-07-03T16:56:25.5929262Z synapse_main | Exception: Error fetching missing prev_events for $qxxaZk6XrhdyTSsYUnZKeQC7RekcetZRDGSEgGTr170: 400: Bad JSON value: float
2026-07-03T16:56:25.5929839Z synapse_main | 2026-07-03 16:53:33,235 - synapse.federation.transport.server.federation - 111 - INFO - PUT-140 - Received txn t2 from host.docker.internal:34747. (PDUs: 1, EDUs: 0)
2026-07-03T16:56:25.5930853Z synapse_main | 2026-07-03 16:53:33,242 - synapse.federation.federation_server - 1383 - INFO - _process_incoming_pdus_in_room_inner-5 - handling received PDU in room !0-ZN5pf5GgVny6qMs7ou:host.docker.internal:34747: <Event event_id=$YlnXM-GqGXn_XvPD2socPqK3rr8aj8-bMmluTCmdIWE, type=m.room.message>

Steps to reproduce

Run a lot of workflows on GitHub.

Homeserver

complement run

Synapse Version

ab277b3

Workers

Multiple workers

Platform

Linux