FTS5_TOKEN_COLOCATED

[andersjo]

The FTS tokenizer API has the concept of "colocated" tokens where multiple tokens can occupy the same position in a sentence. The main use of this functionality is to implement synonyms (See Sec 7.1.1).

Is there any way to mark a token as colocated through the Python API?

[bernard-crnkovic]

I believe the author thought of it, however, I haven't tested it.

xToken(pCtx, 0, "i",                      1,  0,  1);
xToken(pCtx, 0, "won",                    3,  2,  5);
xToken(pCtx, 0, "first",                  5,  6, 11);
xToken(pCtx, FTS5_TOKEN_COLOCATED, "1st", 3,  6, 11);
xToken(pCtx, 0, "place",                  5, 12, 17);

sqlite-fts-python/sqlitefts/fts5.py

Line 244 in 2808e91

r = xToken(

It should be possible, or even if it isn't yet, shouldn't be hard to implement. Will test it to see if it works and make PR if it doesn't.

EDIT: Remove docs link. sorry 😅, you already linked relevant section on SQLite docs website.