Add tool input formats extraction#701
Open
paulzierep wants to merge 24 commits into
Open
Conversation
- clone_repositories() clones/pulls repos locally instead of PyGithub API - Proper XML macro expansion via galaxy.util.xml_macros - Supports non-GitHub URLs (GitLab, self-hosted) - Shallow clones (--depth 1) by default for CI efficiency - Repo URL deduplication prevents wasted clones - --workers N for parallel parsing - No --api flag or GITHUB_API_KEY required - Fixes: 74 previously missed tools found, 36 conda packages gained
This reverts commit 41fb91c.
This reverts commit b2094c2.
- Suite parsed folder: use GitHub repo URL + tree/master + relative path instead of local filesystem path - Suite version: resolve combined macro tokens (e.g. @TOOL_VERSION@) using values from macro XML files before stripping +galaxy suffix - Suite first commit date: deepen shallow clones (git fetch --deepen 1000) when no commit history is found for a tool folder
- New get_tool_inputs() function that recursively collects format attributes from <param type='data'> and <param type='data_collection'> - Handles nested params in conditionals, sections, repeats - Adds Tool input formats field to metadata schema and all export formats - 11 unit tests covering edge cases
- _normalize_repo_url: trailing slash, .git, whitespace - _repo_name_from_url: org-repo name extraction - get_first_commit_for_local_folder: shallow clone deepening, empty output
- Install scholarly (already in requirements.txt) instead of try/except hack - Fix test_extract_galaxy_workflows.py import path to work without PYTHONPATH
- Replace synthetic test fixtures with real wrappers from CI test repo - Test get_tool_metadata_from_local with fastp (macros) and 2d_auto_threshold (no macros) - Add tests for 22 previously untested functions - Fix lint: ruff, black, isort, mypy all passing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding input formats to the tools, but this is based on #697, so we should merge that one first, then this one :)