Use transformer-backed spaCy pipeline, remove Optuna tuning, and add notebook

[Rafaildavar]

Motivation

• Replace the previous ru_core_news_lg + tok2vec setup with a BERT-backed transformer pipeline to leverage contextual embeddings for NER.\n- Remove the optional Optuna hyperparameter tuning scaffolding and its runtime dependency to simplify the code and runtime surface.\n- Provide an executable notebook version of the pipeline for exploration and demonstration.\n

Description

• Switched pipeline construction to use spacy.blank("ru") and added a transformer component configured with DeepPavlov/rubert-base-cased, and moved the regex_pii_matcher to run after the transformer.\n- Replaced tok2vec-centric NER wiring with a transformer-aware NER head by adding _build_transformer_ner_config and updated train_pipeline to select_pipes for transformer and ner.\n- Removed Optuna-related code and flags including the optuna import, _build_pipeline_with_params, _train_for_trial, run_hyperparameter_tuning, train_with_best_params, and command-line options --tune, --tune-and-train, --n-trials, --tuning-epochs, and --best-params.\n- Added a notebook spacy_pii_pipeline.ipynb generated from spacy_pii_pipeline.py for interactive use and documentation.\n

Testing

• No automated tests were added or executed as part of this change.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3005a4174a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Remove script-style entrypoint from notebook cell

Executing the notebook’s single code cell in Jupyter will immediately call main(), and argparse.parse_args() will try to parse kernel flags like -f, causing a SystemExit error before any interactive exploration can happen. Since this file was added specifically as an executable notebook, keeping the if __name__ == "__main__" block here breaks the primary notebook workflow.

Useful? React with 👍 / 👎.