Discrepancy in fts `SNIPPET` text normalization

[Zarux]

There seems to be a fundamental difference in how the production spanner service and the spanner emulator normalizes whitespace within the SNIPPET function.

Production Spanner normalizes text by collapsing multiple consecutive spaces and newlines into a single space, recalculating the begin and end offsets relative to this normalized snippet string. The emulator, however, leaves the original whitespace and newlines completely intact, causing both the raw text and the highlight position offsets to mismatch.

SELECT SNIPPET("test 123\nabc", "123")

Returns this from production

{
  "snippets": [
    {
      "highlights": [{"begin": 6, "end": 9}],
      "snippet": "test 123 abc",
      "source_begin": 1,
      "source_end": 13
    }
  ]
}

and this from emulator (ignore the schema difference due to #355)

{
  "snippets": [
    {
      "highlights": [{"start_position": 6, "end_position": 9}],
      "snippet": "test 123\nabc"
    }
  ]
}

Same offsets.

But with this query:

SELECT SNIPPET("test       123\n abc", "123")

Production returns

{
  "snippets": [
    {
      "highlights": [{"begin": 6, "end": 9}],
      "snippet": "test 123 abc",
      "source_begin": 1,
      "source_end": 20
    }
  ]
}

Notice the highlight offsets are the same and the snippet spaces are collapsed.

Emulator returns

{
  "snippets": [
    {
      "highlights": [{"start_position": 12, "end_position": 15}],
      "snippet": "test       123\n abc"
    }
  ]
}

Where the higlight offsets and snippet text is keeping the whitespace intact.

Running Spanner Emulator Version: 1.5.54