Add ArtiFinder discovered-artifact integration (stage 4b)#17
Open
vahldiek wants to merge 11 commits into
Open
Conversation
Ingest DistriNet/ArtiFinder-Data and match discovered artifact links to AE papers by normalised title + author list. ArtiFinder links carry no badges and are excluded from all scores; the only reuse is repository statistics for GitHub repos matched to AE papers. - models: Artifact.artifinder_urls, SearchEntry.artifinder_urls, new ArtiFinderEntry model; SCHEMA_VERSION 0.2.0 -> 0.3.0 - src/scrapers/artifinder.py: loader (GitHub or local checkout), configurable min_year (default 2017) and local_dir - src/generators/artifinder/generate_artifinder.py: matching, artifacts.json back-patch, artifinder.json, Jekyll _data aggregates, repo-stats hand-off - stages: new optional 'artifinder' stage; search_data + repo_stats consume it - config: artifinder_min_year, artifinder_local_dir (+ env vars) - tests: loader, generator, search_data, repo_stats, models (676 pass) - README + schema registry updated
Drop the redundant assets/data/artifinder.json (it just republished the upstream ArtiFinder-Data repo) and the _build/artifinder_matched_urls.json hand-off. The only persisted artifact of discovered links is now the artifinder_urls back-patch on artifacts.json, plus the small Jekyll _data aggregates for the discovery page. - generate_artifinder: match_entries returns records only; no JSON dumps - repo_stats: read matched GitHub links directly from artifacts.json (_inject_artifinder_urls), no hand-off file - remove unused ArtiFinderEntry model + schema registry entry - update stages outputs/inputs, README, and tests (664 pass)
The 'for art in artifacts' loop bound 'art' to dict, clashing with the later 'art = artifact_index.get(...)' (dict | None). Rename to 'art_row'.
Discovered papers that never went through AE are now emitted as marked search rows (no badges, no scores) so they appear in the search index. The artifinder stage writes _build/artifinder_search_entries.json; generate_search_data appends them (de-duplicated against AE rows). Tests added; 672 pass.
New assets/data/artifinder_authors.json maps a normalised author key to the non-AE discovered papers for that author, so profiles can list them (marked, distinct colour) without affecting any score. Adds _author_key/_build_author_index + tests (674 pass).
- loader captures source_updated (latest ArtiFinder-Data commit date: git locally, commits API remotely) and returns it on ArtiFinderData - summary gains data_updated (ArtiFinder data date) + generated_at (pipeline run time), replacing the ambiguous last_updated - artifinder stage now declares inputs + a 1-day TTL so the orchestrator can skip it when AE inputs are unchanged (incremental builds)
The model was dropped during output slimming; a filesystem sync re-added it. It is not imported anywhere and not in the schema registry.
_author_key -> _af_author_key, mirroring the website's afAuthorKey(); both carry a shared test vector + mutual cross-reference so the byte-identical normalisation can't drift. Algorithm unchanged (no data change).
Search rows now carry source='ae' (AE papers, may also have artifinder_urls) or 'artifinder' (discovered-only, no badges/scores), so consumers don't infer provenance from empty badges/artifact_urls. SearchEntry model + both generators + tests updated (674 pass).
Recovers same-paper links whose AE title differs by Unicode/LaTeX scraping artifacts or minor wording (~21 in current data). Same conf+year, SequenceMatcher >= 0.94, guarded by author overlap (rejects different papers), and logged. Tests added; 677 pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new ArtiFinder data source to the pipeline. ArtiFinder scrapes conference papers directly and discovers links to their artifacts; the published data lives in DistriNet/ArtiFinder-Data.
A new optional
artifinderstage (4b) ingests that data and matches each discovered link to an existing artifact-evaluation (AE) paper by normalised title + author list (same conference + year; ArtiFinder uses DBLP author names, as we do).Policy (important)
Changes
Artifact.artifinder_urls,SearchEntry.artifinder_urls, newArtiFinderEntrymodel.SCHEMA_VERSIONbumped 0.2.0 → 0.3.0 and registered in the schema export.src/scrapers/artifinder.py: reads from GitHub or a local checkout; configurablemin_year(default 2017, the AE era) andlocal_dir.src/generators/artifinder/generate_artifinder.py: matching +artifacts.jsonback-patch (artifinder_urls) + small Jekyll_data/artifinder_*aggregates. No redundant copy of the source data is written;repo_statsreads matched GitHub links directly fromartifacts.json.artifinderstage;search_dataandrepo_statsnow depend on it.artifinder_min_year,artifinder_local_dir(+PIPELINE_*env vars).Configuration
artifinder_min_yearPIPELINE_ARTIFINDER_MIN_YEAR2017artifinder_local_dirPIPELINE_ARTIFINDER_LOCAL_DIR(orREPRODB_ARTIFINDER_DIR)Companion PRs
artifinder_urls, bundle v0.3.0)Website changes (search/profile markers + a new ArtiFinder statistics page): Add ArtiFinder discovered-artifact integration (website) reprodb.github.io#20
Preview of the website changes: https://vahldiek.github.io/reprodb.github.io/artifinder.html
Notes
artifinderstage is optional and skips cleanly when the source is unreachable (offline CI).