Add ETL standardization pipeline for source-agnostic bibliographic data (Base Level)#29
Open
kunalg2909-debug wants to merge 1 commit into
Conversation
…gnostic support for WoS and PubMed Implements Extract-Transform-Validate-Load pipeline: reuses existing parsers.py and format_functions.py, adds new validation module enforcing the WoS type contract (list[str]/str/int, no NaN/None), and wires get_data() as the missing entry point invoked from app.py's Start button. Reuses existing metaTagExtraction() for SR generation per project requirements rather than reimplementing it. Includes 3 patches to functions/get_maininformations.py (PY type handling, metaTagExtraction call overwriting prior calculations) and 1 patch to format_functions.py's AU/AF handling (via get_data.py), documented in execution_evidence.ipynb along with full test evidence against real WoS and PubMed exports.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR implements a robust ETL (Extract → Transform → Validate → Load) pipeline
that makes Bibliometrix-Python source-agnostic, addressing the fragility described
in the project brief where analytical functions only worked reliably with
Web of Science data.
Architecture
parsers.py(WoS/.txt, PubMed/.txt, Cochrane/.txt)and pandas readers already present in the codebase — no duplication.
format_functions.py's existing per-source, per-columnmapping logic (the "Dispatcher"), which already converts each source's
proprietary column names into the standard WoS field-tag schema.
www/services/validation.py, new): Enforces the Type Contractrequired by the spec — multi-value fields must be
list[str], scalar fieldsmust be
str,TCmust beint, and noNaN/Nonemay remain in the finaloutput. Includes
coerce_to_contract(), a non-destructive auto-fix pass thatrepairs common representation issues (e.g. a stray string where a list was
expected) before re-validating.
www/services/get_data.py, new): The single entry-point functionthat was referenced in
app.py's "Start" button handler but never definedanywhere in the codebase. Orchestrates Extract → Transform → Validate →
Coerce → SR generation →
df.set().metaTagExtraction(df, Field="SR")function from
metatagextraction.pyrather than reimplementing SR-generationlogic, per the project requirements.
Bugs Found and Patched
format_functions.py/process_single_file()— deletesAUorAFentirely based on the user's author-format selection, instead of leaving an
empty list, violating the always-present-column contract. Patched
non-destructively in
get_data.pyby restoring the dropped column as anempty
list[str]column after parsing.functions/get_maininformations.py— assumedPY(Publication Year) wasnumeric (
data["PY"].max() - data["PY"].min()), but the standardized schemastores
PYasstrper spec. Patched to castPYtointlocally, on acopy of the working DataFrame, without mutating the shared reactive state.
functions/get_maininformations.py— reassigneddata = df.get()aftercalling
metaTagExtraction(), discarding all previously computed columns inthe same function (Min_Year, Max_Year, CAGR, the PY cast). Patched to merge
only the newly derived
AU_COcolumn into the existing local DataFrame.Various
format_xx_column()functions — inconsistent typing acrosssources (
TCas string instead of int for WoS;VLnot fully unwrapped;PubMed leaves
OA/SCin non-list form). Handled defensively in the Loadphase via
coerce_to_contract()rather than patched at the source, per thePhase 5 validation design — documented as a known upstream inconsistency.
Testing / Execution Evidence
See
execution_evidence.ipynbfor a full, restart-and-run-all-clean notebookdemonstrating:
real Web of Science
.txtexport and a real PubMed.txtexport (10 records).Information (WoS), Most Frequent Words (PubMed), and Co-occurrence
Network (PubMed) — proving the standardized data is fully usable by
existing analysis functions, not just correctly shaped in isolation.
Files Changed
www/services/get_data.py(new) — pipeline entry pointwww/services/validation.py(new) — validation + coercion modulewww/services/__init__.py— exposes the two new modulesfunctions/get_maininformations.py— 2 patches (see above)execution_evidence.ipynb(new) — test evidence notebooksample_wos.txt,pubmed-bibliometr-set.txt(new) — sample data used in testingscreenshots/(new) — dashboard screenshots referenced in the notebook