Singleton dropping preprocessing in solve()

[schroedk]

Motivation

Long-tailed FE designs have most factor levels appearing in one observation,
contributing no identifying information. On directors.csv (525K obs), iteratively
dropping singletons reduces the solve from ~1.4s to ~50ms (≥20×). pyfixest does
this automatically inside feols(); within currently has no equivalent.

Goal

Implement iterative singleton dropping as default preprocessing inside solve(),
with an opt-out. Return β̂ aligned with original factor-level indexing.

[leostimpfle]

I'd be interested in working on this @schroedk

[schroedk]

I'd be interested in working on this @schroedk

Hey @leostimpfle, nice! A bit of background: I wanted to make within more open to other estimators than pyfixest. Right now the singletons dropping happens in pyfixest and I guess before any demean backend logic happens. The challenge here is to create a good ergonomic API. Would love to have a fresh pair of eyes on this! The pure logic exists already in pyfixest.

[leostimpfle]

Sounds good! I'll spend some time to think about the API design before jumping to the implementation of the singleton detection.

As background, I was keen to understand within better prompted by your recent addition of varying slopes where we still have to implement the formula plumbing on the pyfixest side. So thinking carefully about the API will be helpful for me personally, too.