Skip to content

Add ETL standardization pipeline for source-agnostic bibliographic data (Base Level)#29

Open
kunalg2909-debug wants to merge 1 commit into
PRAISELab-PicusLab:mainfrom
kunalg2909-debug:etl-standardization-pipeline
Open

Add ETL standardization pipeline for source-agnostic bibliographic data (Base Level)#29
kunalg2909-debug wants to merge 1 commit into
PRAISELab-PicusLab:mainfrom
kunalg2909-debug:etl-standardization-pipeline

Conversation

@kunalg2909-debug

Copy link
Copy Markdown

Overview

This PR implements a robust ETL (Extract → Transform → Validate → Load) pipeline
that makes Bibliometrix-Python source-agnostic, addressing the fragility described
in the project brief where analytical functions only worked reliably with
Web of Science data.

Architecture

  • Extract: Reuses the existing parsers.py (WoS/.txt, PubMed/.txt, Cochrane/.txt)
    and pandas readers already present in the codebase — no duplication.
  • Transform: Reuses format_functions.py's existing per-source, per-column
    mapping logic (the "Dispatcher"), which already converts each source's
    proprietary column names into the standard WoS field-tag schema.
  • Validate (www/services/validation.py, new): Enforces the Type Contract
    required by the spec — multi-value fields must be list[str], scalar fields
    must be str, TC must be int, and no NaN/None may remain in the final
    output. Includes coerce_to_contract(), a non-destructive auto-fix pass that
    repairs common representation issues (e.g. a stray string where a list was
    expected) before re-validating.
  • Load (www/services/get_data.py, new): The single entry-point function
    that was referenced in app.py's "Start" button handler but never defined
    anywhere in the codebase. Orchestrates Extract → Transform → Validate →
    Coerce → SR generation → df.set().
  • Calculated Fields (SR): Reuses the existing metaTagExtraction(df, Field="SR")
    function from metatagextraction.py rather than reimplementing SR-generation
    logic, per the project requirements.

Bugs Found and Patched

  1. format_functions.py / process_single_file() — deletes AU or AF
    entirely based on the user's author-format selection, instead of leaving an
    empty list, violating the always-present-column contract. Patched
    non-destructively in get_data.py by restoring the dropped column as an
    empty list[str] column after parsing.

  2. functions/get_maininformations.py — assumed PY (Publication Year) was
    numeric (data["PY"].max() - data["PY"].min()), but the standardized schema
    stores PY as str per spec. Patched to cast PY to int locally, on a
    copy of the working DataFrame, without mutating the shared reactive state.

  3. functions/get_maininformations.py — reassigned data = df.get() after
    calling metaTagExtraction(), discarding all previously computed columns in
    the same function (Min_Year, Max_Year, CAGR, the PY cast). Patched to merge
    only the newly derived AU_CO column into the existing local DataFrame.

  4. Various format_xx_column() functions — inconsistent typing across
    sources (TC as string instead of int for WoS; VL not fully unwrapped;
    PubMed leaves OA/SC in non-list form). Handled defensively in the Load
    phase via coerce_to_contract() rather than patched at the source, per the
    Phase 5 validation design — documented as a known upstream inconsistency.

Testing / Execution Evidence

See execution_evidence.ipynb for a full, restart-and-run-all-clean notebook
demonstrating:

  • Standalone pipeline execution (Extract → Validate → Coerce → SR) against a
    real Web of Science .txt export and a real PubMed .txt export (10 records).
  • Screenshots of the live Shiny dashboard successfully running Main
    Information
    (WoS), Most Frequent Words (PubMed), and Co-occurrence
    Network
    (PubMed) — proving the standardized data is fully usable by
    existing analysis functions, not just correctly shaped in isolation.

Files Changed

  • www/services/get_data.py (new) — pipeline entry point
  • www/services/validation.py (new) — validation + coercion module
  • www/services/__init__.py — exposes the two new modules
  • functions/get_maininformations.py — 2 patches (see above)
  • execution_evidence.ipynb (new) — test evidence notebook
  • sample_wos.txt, pubmed-bibliometr-set.txt (new) — sample data used in testing
  • screenshots/ (new) — dashboard screenshots referenced in the notebook

…gnostic support for WoS and PubMed

Implements Extract-Transform-Validate-Load pipeline: reuses existing parsers.py and format_functions.py, adds new validation module enforcing the WoS type contract (list[str]/str/int, no NaN/None), and wires get_data() as the missing entry point invoked from app.py's Start button. Reuses existing metaTagExtraction() for SR generation per project requirements rather than reimplementing it. Includes 3 patches to functions/get_maininformations.py (PY type handling, metaTagExtraction call overwriting prior calculations) and 1 patch to format_functions.py's AU/AF handling (via get_data.py), documented in execution_evidence.ipynb along with full test evidence against real WoS and PubMed exports.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant