Make the encode pool opt-out behind a parallel feature

[ArthurZucker]

parallel joins the default feature set, so nothing changes unless you ask for it. With --no-default-features the private rayon ThreadPool is never built, no worker threads are spawned, and the job-splitting machinery is not compiled — encode takes the serial path a small batch already took.

66,272 bytes on the stripped binsize_pipeline example: 2,124,096 → 2,057,824. Measured with progressbar held on, so the number is the pool rather than indicatif.

What this is not

It does not make rayon optional, and the title deliberately doesn't claim it does. rayon stays a hard dependency with the feature off, because ptr_hash declares it unconditionally to build the MPHF — and rayon-cond and rdst pull it as well:

rayon v1.12.0
├── ptr_hash v2.0.2   -> tk-encode
├── rayon-cond v0.4.0 -> tk-encode
└── rdst v0.20.14

cargo tree still shows rayon either way. What the flag buys is a build with no thread pool and no parallel encode path, which is what matters for embedded and single-threaded hosts. It is not a dependency reduction, and a reviewer should not expect one.

Making rayon genuinely optional would mean serial fallbacks at all 19 maybe_par_* call sites (7 in tk-encode, 12 in tk-train) plus tk-train opting in — and it still wouldn't remove rayon, for the reason above. That work also lands squarely in tokenizer/mod.rs, encoding.rs and padding.rs, which the pipeline-only refactor replaces. Not worth doing unless ptr_hash changes, and if it does it should be re-derived from scratch.

Implementation notes

• The parallel batch body moves out of encode into a gated encode_parallel. It takes the owned input storage rather than a slice because JobCore ends up owning it, which is also why refs is rebuilt inside.
• With the feature off, the striding and segment-planning helpers have no caller. They're left compiled-but-unused under a scoped allow(dead_code) rather than cfg'd one by one: they reference each other densely and LTO drops all of it from the binary regardless, so cfg-ing each would be churn in files this branch's successor replaces.
• Nested if let rather than a let-chain in the chunk-commit scatter — this crate is edition 2018.

Verification

	default	--no-default-features
build	clean, 0 warnings	clean, 0 warnings
lib tests	291 passed	290 passed¹
parallel oracle	15 passed	15 passed
doc tests	20 passed	20 passed

¹ the pool's own test is gated with the module.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.