Merge develop into main - #205
Open
jeanbez wants to merge 245 commits into
Open
Conversation
…on partial metric support
…multiple uploads.
…front end support.
Kaveen/redis
license update
Co-authored-by: Dylan_Ryan <ryan.1989@osu.edu> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>
* pre-commit-hooks check * readme updated * doc update
Bumps [psutil](https://github.com/giampaolo/psutil) from 5.9.8 to 7.2.2. - [Changelog](https://github.com/giampaolo/psutil/blob/master/docs/changelog.rst) - [Commits](giampaolo/psutil@v5.9.8...v7.2.2) --- updated-dependencies: - dependency-name: psutil dependency-version: 7.2.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pytz](https://github.com/stub42/pytz) from 2024.1 to 2026.3.post1. - [Release notes](https://github.com/stub42/pytz/releases) - [Commits](stub42/pytz@release_2024.1...release_2026.3.post1) --- updated-dependencies: - dependency-name: pytz dependency-version: 2026.3.post1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Update Zenodo DOI
Bumps [cryptography](https://github.com/pyca/cryptography) from 48.0.1 to 50.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@48.0.1...50.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-version: 50.0.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
A pandas-written HDF5 file keeps its frame behind pandas' private block layout: axis0/block0_values for `fixed`, a compound `table` dataset plus an _i_table index for `table`. The reader walked those datasets as if they were independent arrays, so the row index and the column-name arrays were concatenated into the data itself. Both bundled samples are pandas stores and both were affected: adult.h5 (65155, 7) columns '0'..'6' -> (32561, 16) real names employees.h5 (31, 4) columns '0'..'3' -> (8, 7) real names adult.h5 now round-trips exactly equal to the adult.csv it was derived from. Categorical columns such as income, race and sex live in the object block and were unreachable before, so fairness and privacy metrics could not run on these files at all. Detect an HDFStore and let pandas decode it, falling back to the existing traversal if decoding fails, so no currently-readable file changes behaviour. A store may hold several frames; an explicit key is honoured in either form (`first` or `/first`) and defaulting to the first frame is logged rather than silently hiding the rest. Layout classification is unchanged: these files stay `legacy`, so the multi-dataset picker correctly does not appear for them. Also point two tests at examples/sample_data instead of the pre-reorg web/static/datasets path. Both had been silently skipping, which is why this went unnoticed. Co-authored-by: Kaveen Hiniduma <hiniduma.1@buckeyemail.osu.edu>
Bump cryptography from 48.0.1 to 50.0.0
Bump aiohttp from 3.14.1 to 3.14.3
…d149aa1f Bump the lock-maintenance group with 3 updates
…195) * docs: align theme with the app, and reconcile the docs with the code Squashed restoration of a 16-commit branch whose git objects were lost when the working directory was reaped. Only the final file state survived, so the history could not be replayed; the original messages are preserved alongside the backup. Theme - Port the light-mode design tokens from web/static/css/theme.css so the docs and the app read as one product: page and sidebar surfaces, links, nav text, code blocks, table stripes and borders, and admonitions, which the theme left on its own blue and orange. - Replace the theme's depth-stacked neutral grays in the sidebar with a single accent-tinted open state, and stop the expander icon vanishing on hover. - Fix .admonition note, a descendant selector for a <note> element that never matched, so every note rendered a rounded title bar on a square body. - Let table cells wrap; the theme's white-space: nowrap pushed prose off-page. - Shrink the sidebar logo by a third. Correctness - Two documented pages did not exist: /upload-file is a legacy redirect and FAIR assessment is a panel, not a page. index.html#key-features never existed. aidrin.io only redirects at the root, so deep links 404; the app is served from demo.aidrin.org. - web_usage.rst taught TestPyPI at 0.9.7 against a 2026.x CalVer release. - pip install aidrin cannot run the web app: only aidrin* ships in the wheel. - "Output" was mis-levelled, nesting Data Structure, one of the six advertised dimensions, inside the FAIR report. - contributing.rst said "the infrastructure" where it meant "the web interface", in four places. - AGENTS.md gave underscore CLI names that argparse rejects, said the MCP server was not on develop when it is, and cited a setuptools pin that moved. Coverage - aidrin summarize and differential-privacy were undocumented; the latter writes noisy/noisy_data.csv into the working directory, documented nowhere. - 10 of 21 exported library functions were undocumented; all 21 now are. - The MCP server registers 11 tools; 5 were documented. - testing.rst listed 7 of 25 unit test files, and sent contributors to stand up Redis, Celery and Flask for integration tests that need none of them. Structure - Globus and LLM setup move from the contributor guide to the user guide. - The Python library gets its own page; the install page drops the PyPI option that could not work and folds Docker in as a peer path. - New metric_names.rst maps every metric across the web interface, CLI, library, Globus and MCP, with a test guarding it against drift. * pin docs/requirements.txt sphinx to 6.2.1 to match pyproject.toml and uv.lock * minor updates to be conisistent with the capabilities --------- Co-authored-by: Kaveen Hiniduma <hiniduma.1@buckeyemail.osu.edu> Co-authored-by: Kaveen Hiniduma <123603111+kaveenh@users.noreply.github.com>
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.2.1 to 8.1.3. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/v8.1.3/CHANGES.rst) - [Commits](sphinx-doc/sphinx@v6.2.1...v8.1.3) --- updated-dependencies: - dependency-name: sphinx dependency-version: 8.1.3 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bump psutil from 5.9.8 to 7.2.2
Bump pytz from 2024.1 to 2026.3.post1
Bump sphinx from 6.2.1 to 8.1.3
Nothing installs this file. The only `pip install -r` in the repo is docs.yml:23, which installs docs/requirements.txt; no workflow, neither Dockerfile, and not .readthedocs.yaml reference the root one. Its only remaining mention was a comment in dependabot.yml, updated here. It was also wrong in three ways, measured against pyproject.toml: - Missing five core runtime dependencies: flower, numba, pyarrow, pyyaml and tables. pyarrow and tables back the Parquet and HDF5 readers, so anything installed from this file could not read two of the six supported formats. - Unsatisfiable alongside the package. It pinned dython==0.7.5 against dython>=0.7.11, and pandas==2.2.1 against pandas>=2.2.2. The dython pin is the version pyproject.toml explicitly moved off, because 0.7.11 dropped pkg_resources. - 20 of its 34 entries were transitive pins already covered by uv.lock, which is why dependabot kept opening pull requests against it. #190, #192 and #193 all wrote pins into a file no install path reads. pyproject.toml plus uv.lock already cover every case it was meant to, and unlike this file they know about the globus, llm, telemetry, mcp and agentic extras. Verified after removal: uv sync resolves, pyarrow/tables/pyyaml/numba import, unit suite 444 passed, integration suite 130 passed, and the docs build clean under -W. Co-authored-by: Kaveen Hiniduma <123603111+kaveenh@users.noreply.github.com>
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 8.1.3 to 9.1.0. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst) - [Commits](sphinx-doc/sphinx@v8.1.3...v9.1.0) --- updated-dependencies: - dependency-name: sphinx dependency-version: 9.1.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the lock-maintenance group with 1 update: [breathe](https://github.com/breathe-doc/breathe). Updates `breathe` from 4.35.0 to 4.36.0 - [Release notes](https://github.com/breathe-doc/breathe/releases) - [Changelog](https://github.com/breathe-doc/breathe/blob/main/CHANGELOG.rst) - [Commits](breathe-doc/breathe@v4.35.0...v4.36.0) --- updated-dependencies: - dependency-name: breathe dependency-version: 4.36.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: lock-maintenance ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pyarrow](https://github.com/apache/arrow) from 24.0.0 to 25.0.0. - [Release notes](https://github.com/apache/arrow/releases) - [Commits](apache/arrow@apache-arrow-24.0.0...apache-arrow-25.0.0) --- updated-dependencies: - dependency-name: pyarrow dependency-version: 25.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…nd MCP server (#187) * docs: add design spec and implementation plan for Globus remote CLI * feat(compute): add headless remote entry point for Globus Compute * feat(compute): add named endpoint profiles for headless remote runs * fix(compute): correct default-source provenance, dangling-default fallback, and atomic 0600 writes in profiles * feat(compute): add headless Globus Compute client * fix(compute): tighten check() pending heuristic to match check_task Drop the "running" substring match, which was accidentally lifted from a different function (_run_probe_sync) and caused genuine failures to be misclassified as still-processing, discarding the real error until poll() timed out. * feat(compute): add RemoteExecutor duck-typed against the headless api * fix(compute): enforce image policy in run_batch_metrics, fix run_metric positional parity run_batch_metrics forwarded config verbatim, letting the endpoint default save_images to true and write files locally there; it also defaulted strip_visualizations to false, opposite the remote default. Apply the same image policy run_metric already had: force save_images=false on the wire, strip_visualizations=true unless images were requested, and save returned per-metric visualizations on the client. run_metric also diverged from api.run_metric's parameter order, so a positional call past file_type raised TypeError against the executor while succeeding against the module. Declare the intervening parameters explicitly in the same order to restore positional compatibility. * feat(cli): add 'aidrin remote' for Globus Compute execution * fix(cli): honest interrupt reporting, remote help, and Ctrl-C tests * feat(mcp): accept endpoint/profile for remote execution * fix(mcp): correct endpoint docstrings for run_batch and summarize_dataset * docs: document remote execution for the CLI, MCP, and skill * test: add end-to-end checks for headless remote execution * fix(cli): fail remote runs that come back as an error result remote_headless_runner returns a worker exception as data, so the Globus task succeeds and the CLI's `except Exception` never fires: a failed `aidrin remote run` printed the error dict and exited 0, and `aidrin remote data-quality` rendered it as a clean-looking all-N/A summary, making a failed assessment indistinguishable from a benign one. Every remote dispatch in main() -- the metric branches, summarize, data-quality and batch -- now routes its result through _fail_on_remote_error(), which writes a top-level {"Error"/"ErrorType"} result to stderr and exits 1, matching the local commands. Only the top level counts: a per-metric error inside a batch mapping is one metric failing, exactly as it is locally. The guard no-ops for local runs, so that path is untouched. Also in the same surface: - fail fast when globus-compute-sdk is missing, using the previously dead client.is_available(), so the CLI no longer announces a remote run and only then reports that the SDK is absent (exit 2 with the pip install 'aidrin[globus]' message, now a single shared constant). - guard conn.logout(), which was an unverified SDK call: a missing method now yields a message pointing at globus-compute-endpoint and ~/.globus_compute/ instead of an AttributeError traceback. - honour --timeout in `aidrin remote check`, keeping 60s as the default. * chore(remote): drop the client image_dir, ignore .aidrin.json, extend the docs - RemoteExecutor.run_batch_metrics no longer forwards the client-local image_dir inside the submitted config, where it is inert because save_images is forced False on the endpoint; run_metric already dropped it. - .gitignore: .aidrin.json, written by `aidrin remote configure --local`. - test_mcp_server: isolate the profile listing test the way the other profile tests do, so it stops reading the developer's real ~/.aidrin/config.json and ./.aidrin.json. - docs/source/remote.rst: state that exit codes match the local commands; document the one output-shape divergence (a batch config with save_images: true keeps the visualization keys as written file paths remotely, where a local run strips them); and add the parts of `aidrin remote --help` the page was missing -- task --cancel, --timeout, list, remove, check, login, status, logout. * fix(cli): only treat a remote result as failed when the worker raised The error guard added with the previous commit keyed on "Error" or "ErrorType", but many built-in metrics return a legitimate {"Error": "No numerical features found in the dataset."} for input they cannot score, and remote_headless_runner wraps only *raised* exceptions, returning everything else verbatim. So an ordinary metric result became a failed run remotely while the same input exits 0 locally -- the exact parity this feature exists to preserve. Require ErrorType, which only the runner's except wrapper sets and no metric's validation return ever carries. It is the clean line between "the worker raised" and "the metric reported a problem in its result". `aidrin remote task <id> --wait` now runs its recovered result through the same guard: recovering a failed task is still a failed run. Tests cover both halves of the predicate, and the two bare-Error cases assert against the local command's own stdout and exit code, so they state the parity rather than transcribe it. * Fix remote task cancellation falsely reporting success globus_compute_sdk.Client has no cancel_task method in the currently supported SDK (verified against 4.15.0), so client.cancel() was hitting an AttributeError that its bare except swallowed at debug level. Every caller then assumed the cancellation worked: `aidrin remote task <id> --cancel` printed "Cancelled" and exited 0, and Ctrl-C during a blocking remote run printed "cancelled remote task" -- both while the task kept running on the endpoint. cancel() now returns a bool (via getattr so older SDKs that do expose cancel_task still work), and both the --cancel command and the Ctrl-C path in the CLI report the real outcome, including the --wait recovery hint when cancellation isn't possible. Docs updated to match. * fix(remote): sweep frame cache sidecar on worker, catch Ctrl-C in management dispatch remote_headless_runner left a .aidrin.feather cache sidecar on the endpoint's filesystem with nothing to clean it up, since the client-side cleanup in cli.py's finally is deliberately skipped for remote runs. Sweep it on the worker instead, in a finally that derives the dataset path per command and never affects the returned result. Ctrl-C during a blocking `aidrin remote task <id> --wait`, `check`, or `configure` dumped a bare traceback because the management dispatch only caught `Exception`, which KeyboardInterrupt doesn't inherit from. Catch it explicitly and exit 130 with wording consistent with the existing interrupt handler. * fix(tests): skip POSIX-only file mode assertions on Windows The two profile-config permission tests asserted st_mode == 0o600, which fails on Windows: os.chmod there only toggles the read-only flag and st_mode reports 0o666, so CI failed on every Windows/Python combination while Linux and macOS passed. The write path itself is portable (mkstemp + os.replace); only the assertions were over-specified. Gate them on os.name == 'posix' and document in _write() that the 0600 is POSIX-only, noting that the file holds endpoint UUIDs rather than credentials. * docs: state plainly that remote runs produce no images by default The skill said 'Visualizations come back in the result and are written on your machine' as an unconditional description of remote runs, and advised rerunning 'without image output' on a size failure. Both read as though image output were on. It is not: every remote path sends save_images=False and strip_visualizations=True, so nothing is produced unless a batch config sets save_images: true. An agent following the old wording would have gone looking for PNGs that were never written. Also corrects a stale pre-existing claim in the same section: 'aidrin run' does not write PNGs (verified: no /tmp/aidrin_images is created), while 'aidrin batch' does unless the config disables it. * Pin mcp<2.0.0 to keep aidrin.mcp.server importable (2.0.0 dropped fastmcp) --------- Co-authored-by: Kaveen Hiniduma <hiniduma.1@buckeyemail.osu.edu>
Bumps [h2](https://github.com/python-hyper/h2) from 4.4.0 to 4.4.1. - [Changelog](https://github.com/python-hyper/h2/blob/master/CHANGELOG.rst) - [Commits](python-hyper/h2@v4.4.0...v4.4.1) --- updated-dependencies: - dependency-name: h2 dependency-version: 4.4.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bump sphinx from 8.1.3 to 9.1.0 in /docs
…c210e9fb Bump breathe from 4.35.0 to 4.36.0 in the lock-maintenance group
Bump pyarrow from 24.0.0 to 25.0.0
Bump h2 from 4.4.0 to 4.4.1
Bumps the lock-maintenance group with 4 updates in the / directory: [openai](https://github.com/openai/openai-python), [langchain-openai](https://github.com/langchain-ai/langchain), [faiss-cpu](https://github.com/facebookresearch/faiss) and [pymupdf](https://github.com/pymupdf/pymupdf). Updates `openai` from 2.52.0 to 2.53.0 - [Release notes](https://github.com/openai/openai-python/releases) - [Changelog](https://github.com/openai/openai-python/blob/main/CHANGELOG.md) - [Commits](openai/openai-python@v2.52.0...v2.53.0) Updates `langchain-openai` from 1.4.1 to 1.4.2 - [Release notes](https://github.com/langchain-ai/langchain/releases) - [Commits](langchain-ai/langchain@langchain-openai==1.4.1...langchain-openai==1.4.2) Updates `faiss-cpu` from 1.14.3 to 1.15.0 - [Release notes](https://github.com/facebookresearch/faiss/releases) - [Changelog](https://github.com/facebookresearch/faiss/blob/main/CHANGELOG.md) - [Commits](facebookresearch/faiss@v1.14.3...v1.15.0) Updates `pymupdf` from 1.28.0 to 1.28.2 - [Release notes](https://github.com/pymupdf/pymupdf/releases) - [Changelog](https://github.com/pymupdf/PyMuPDF/blob/main/changes.txt) - [Commits](pymupdf/PyMuPDF@1.28.0...1.28.2) --- updated-dependencies: - dependency-name: openai dependency-version: 2.53.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: lock-maintenance - dependency-name: langchain-openai dependency-version: 1.4.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: lock-maintenance - dependency-name: faiss-cpu dependency-version: 1.15.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: lock-maintenance - dependency-name: pymupdf dependency-version: 1.28.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: lock-maintenance ... Signed-off-by: dependabot[bot] <support@github.com>
…2746341d Bump the lock-maintenance group across 1 directory with 4 updates
main has been on a parallel track since 182685f (2025-07-16). Its only commit not in develop is 35caf44 "prepare for first release (#16)", which develop has entirely superseded. A content merge produces 69 conflicts (add/add, content, rename/rename, and 15 modify/delete on files the restructure in 2fb9cb2 removed). Every one resolves in develop's favour, so this merge takes develop's tree wholesale via -s ours + read-tree rather than reconciling line by line. The resulting tree is identical to develop at 2b315fe; 35caf44 is retained in ancestry as the first parent.
Collaborator
Author
|
To avoid issues in the future, we need to merge with merge-commit method. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
mainup to date withdevelop(2b315fe).Why this isn't a normal merge
mainhas been on a parallel track since182685f(2025-07-16). Its only commit not indevelopis35caf44"prepare for first release (#16)" —mainis exactly merge-base + 1 commit, and that single commit is the sole cause of the divergence.A content merge produces 69 conflicts:
Every one of them resolves in
develop's favour — there is no real content decision in any of them. Themodify/deletegroup is the risky part: those are files the restructure in2fb9cb2deliberately removed (aidrin/main.py,aidrin/static/,aidrin/templates/), and git's default onmodify/deleteis to keepmain's copy. Resolving them by hand invites resurrecting the old monolithic app alongside the currentweb//worker/structure.So this merge takes
develop's tree wholesale instead of reconciling line by line:Verification
git diff HEAD origin/develop→ empty; the tree is identical todevelopat2b315fe35caf44(old main) and2b315fe(develop) — a real merge, not a reset35caf44remains inmain's ancestry; no force-push, nothing discardedaidrin/{main.py,static/,templates/}in the result — the removed monolith did not come backNote for reviewers
This records
developas merged while discardingmain's side of the content, so git will not surface those differences again. That is the intent here, given35caf44is superseded release prep.