Save Apps in their own database table#373
Conversation
Three comments in apps/core.py claimed the poll loop dispatches `bjobs -u all` through a user's worker to fetch all jobs. The actual implementation passes an explicit list of cluster_job_ids and lets py-cluster-api run `bjobs` for just those IDs — cross-user visibility comes from LSF allowing ID-based queries across users, not from -u all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
npm 10 warns "Unknown project config 'min-release-age'" because the key was renamed in npm 11. Using the new name silences the warning and is forward-compatible; npm 10 silently ignores it (the supply-chain floor only kicks in once we're on npm 11+). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This reverts commit dda7045.
| try: | ||
| await refresh_cached_manifest(username, app_url, manifest_path) | ||
| except Exception as e: | ||
| logger.warning(f"Failed to refresh cached manifest after pull: {e}") |
There was a problem hiding this comment.
I'm finding the added code/overall flow of submit_job to be a little confusing. The YAML manifest is read from cache (fallback to disk) at the very start of this function, but then the repo code itself is confirmed to be cached (and re-pulled if the user selects the pull_latest option, setting it to true) later, at which point it's checked again whether the manifest was updated? It seems that maybe the repo cache should also be checked at the beginning, and the manifest updated if pull_latest=true at that point. I'm also finding the logic of the if/else statement under the #clone/pull repo into the user's cache comment hard to follow - I'm not clear why one branch sets the cd_suffix to one value and also how the value of pulled_manifest_repo is determined.
Move user apps from the manifest table to a dedicated DB table. Always use the cached manifest in the DB, only syncing from disk when they change. The main benefit is that this approach makes the Apps page load much faster.
This PR also contains some minor misc fixes to unrelated stuff like the devcontainer, and code comments.
Summary
user_preferencesto a dedicateduser_appstable, with each row caching the resolvedAppManifestso the hot path is a single SELECT instead of disk I/O on every request.get_or_load_manifest(cache-first read) andrefresh_cached_manifest(re-read from disk after clone/pull) so the DB cache stays in lockstep with the on-disk YAML.~/.gitconfiginto the devcontainer so git operations inside the container use the user's identity.bjobs -u all.Details
user_appstableNew Alembic migration
c4e8a7d92b15_add_user_apps_tableadds:user_apps(id, username, url, manifest_path, name, description, branch, manifest JSON, added_at, updated_at)(username, url, manifest_path)usernameCRUD helpers added to
database.py:list_user_apps,get_user_app,upsert_user_app,delete_user_app. The old JSON-blob-in-preferences path is removed from the apps endpoints.Manifest cache
get_or_load_manifestreads the cachedmanifestJSON from the row and validates it. On NULL or schema-validation failure, it falls back tofetch_app_manifest(disk) and writes the fresh value back. If no row exists (preview for not-yet-installed apps), it just reads from disk without inserting.refresh_cached_manifestis called after operations that mutate the on-disk YAML (e.g.submit_jobwithpull_latest=Trueagainst the manifest's own repo), keeping the cache consistent.GET /api/appsno longer hits disk per app — it returns cached manifests and lazily backfills any rows whose stored manifest is missing/stale.POST /api/apps/fetch-manifest,POST /api/apps,POST /api/apps/update, andDELETE /api/appsall go through the new table.Devcontainer
.devcontainer/devcontainer.jsonmounts${localEnv:HOME}/.gitconfigread-only into the container so commits/pulls inside the container use the host git identity.Polling comments
Updated comments in
apps/core.pyto accurately describe how the poll loop dispatches apollaction with explicitcluster_job_idsthrough one user's worker — LSF allows cross-user lookup by ID, so a single worker call covers all users' jobs. No behavior change.Other
FgButton.stories.tsx.minimum-release-agenpm config rename.@StephanPreibisch @allison-truhlar