[Infra] Dockerize AstroML Environment#1
Open
jaynomyaro wants to merge 282 commits into
Open
Conversation
- Add proper error handling to load_checkpoint in deep_svdd_trainer.py - Add proper error handling to load_checkpoint in temporal.py - Add validation for required checkpoint keys - Add explicit error messages for different failure scenarios - Add return value to indicate success/failure - Use weights_only=True for security (addresses test_security.py concerns) - Add comprehensive tests for checkpoint loading error cases Fixes silent failures where checkpoint loading would fail without clear error messages, making debugging difficult.
- Resolved merge conflict in deep_svdd_trainer.py - Combined security fix (weights_only=True) with comprehensive metadata validation - Added return value to indicate success/failure - Maintained detailed error messages from both versions
- Create ClaimService with async background retry mechanism - Implement exponential backoff with optional jitter for retries - Add claim status tracking (pending, submitted, approved, rejected, failed, expired) - Add claim expiration handling - Add max retry limit with proper error handling - Implement database integration for claim status updates - Add comprehensive tests for retry logic and edge cases - Support loading pending claims from database for recovery Features: - Configurable retry parameters (max_retries, backoff, jitter) - Async background loop for automatic retry processing - Claim expiration detection and handling - Database status updates for claim tracking - Recovery of pending claims after restart
…2e test Closes #151 — `load_database_config` now raises a ValueError with a schema-by-example template instead of silently defaulting when the YAML is empty / not a mapping / missing the `database:` key / has the wrong value type. Invalid port and other pydantic violations are re-raised with the same template appended so operators see the expected layout right alongside the error. `resolve_database_url` catches both pydantic.ValidationError and ValueError. Seven tests in tests/integration/test_database_config_validation.py pin each branch. Closes #189 — train.py now seeds python.random, numpy.random, torch.manual_seed, torch.cuda (incl. all devices), sets PYTHONHASHSEED, and enables cudnn deterministic + disables cudnn benchmark on GPU runs. Hydra's `cfg.experiment.seed` remains the canonical knob; ASTROML_SEED is honoured as an env fallback for non- Hydra entrypoints. Closes #186 — pytest workflow now ships a matrix: - CPU x py3.10 + py3.11 — mandatory; runs `pytest -m "not gpu"` - GPU x py3.11 — optional (continue-on-error: true); short-circuits cleanly when CUDA isn't reachable on the runner, executes `pytest -m gpu` only when it is. Custom `gpu` + `e2e` markers registered in pyproject.toml. Closes #193 — new tests/integration/test_pipeline_e2e.py runs ingestion → graph → features against synthetic ledger data through the real IngestionService + a file-backed StateStore under tmp_path, asserts every account appears in the feature output, re-ingest is idempotent, and a fixed seed produces identical features across two runs. No external Postgres or Stellar RPC required.
- REQUIREMENTS.md + cleaned requirements.txt: document when to use each of requirements.txt / requirements-cpu.txt / requirements-minimal.txt, surface the shared lower-bound pins in a single table, and unwedge requirements.txt (it had a stray markdown fence + '---' separator that pip couldn't parse, hiding the feature-store / visualization / dev / notebook deps from a clean install) (closes #187). - astroml/utils/logging.py: new configure_logging() helper backed by ASTROML_LOG_LEVEL and ASTROML_LOG_FORMAT (text|json). _JsonFormatter emits one structured record per line (ts, level, logger, message, plus any extra={...} fields). Existing logging.basicConfig call sites in ingestion/enhanced_service.py and ingestion/stellar_ledger.py now delegate to it so every entry point shares one config (closes #195). - astroml/features/graph/snapshot.py: new iter_db_snapshot_edges() that yields (SnapshotMeta, edge generator) pairs and pulls rows from the DB with SQLAlchemy yield_per/stream_results. Peak memory per window is bounded by DEFAULT_STREAM_CHUNK_SIZE (5k edges ~ a few MB) regardless of how many edges the window actually contains, so long-window builds no longer OOM. Existing iter_db_snapshots remains for callers that need the full materialised SnapshotWindow (closes #199).
Documentation: missing architecture diagram
…seeds-e2e-ci feat: db config validation, CPU/GPU CI matrix, deterministic seeds, e2e test
…emory-187-195-199 docs+chore: clean requirements, central logging, streaming graph builder
The centralized logging module added in 0b31e91 (closes #195) shipped without unit-test coverage. Adds tests/integration/test_logging_config.py pinning the contracts that downstream services depend on: - text format renders human-readable lines (logger name + level) - json format emits one parseable JSON object per record + surfaces structured `extra=` fields - level filter drops records below the configured threshold - a re-call without `force=True` doesn't pile handlers on the root logger (the `_CONFIGURED` guard) - ASTROML_LOG_LEVEL / ASTROML_LOG_FORMAT env vars drive the default - an unknown format string falls back to text rather than erroring Note on #187 and #199: both shipped in 0b31e91 as well — REQUIREMENTS.md documents the three requirements files and their decision tree, and the streaming graph builder lives in astroml/ingestion/stream.py with an integration test already at tests/integration/test_streaming.py. This PR only fills the testing gap on the logging module since that was the only acceptance-criteria "add tests" still uncovered.
Expands the `astroml` CLI top-level help so new contributors can discover usage without grepping the README: - Switches the parser to RawDescriptionHelpFormatter so the new examples block renders verbatim - Adds top-level `--config PATH` (forwarded to load_database_config for the `config --print-db` subcommand) - Adds top-level `--env NAME` that sets ASTROML_ENV for downstream loaders (documented in docs/api/configuration.md), without overwriting an already-set ASTROML_ENV - Adds an examples section covering every subcommand and an environment-variables section (ASTROML_DATABASE_URL, ASTROML_ENV) - Adds tests/test_cli_help.py with 6 cases pinning the help text and the new flags' wiring Co-Authored-By: Claude <noreply@anthropic.com>
fix(cli): clearer top-level help with examples and global flags
…g-memory test(logging): cover astroml.utils.logging.configure_logging (#195)
…se-for-local-dev feat: add lightweight docker compose override for local dev
…to-speed-builds feat: add CI caching for pip wheels to speed builds
…ent-backfill-feature-pagination #175: Add type-checked Pydantic training config
…163) Introduces test_data/ledgers.csv and test_data/transactions.csv as small, self-contained sample datasets, and adds tests/integration/test_e2e_sample_data.py which runs a full ingestion → graph → features round-trip against them using only SQLite and no network access.
…#150, #156) - test_cli_help.py: update docstring to reference #150; add tests that verify all subcommands appear in --help, the README link is present, and quickstart --help documents --num-ledgers/--epochs/--seed flags. - tests/test_ci_matrix.py: new file asserting that the `gpu` and `e2e` pytest markers are registered in pyproject.toml, that pytest.yml runs CPU jobs with `-m 'not gpu'`, includes a gpu matrix flavor, and marks the GPU job continue-on-error so standard CI always passes.
feat(tests): e2e sample data test, YAML safe_load enforcement, CLI help & CI matrix regressions
…tting-checks Add pre-commit hooks and formatting checks
…e-random-seeds-and-configs Benchmark reproducibility store random seeds and configs
Graph builder memory usage spike on large windows
feat: implement schema validation for feature store ingestion
- Add docker-compose.prod.yml with production-optimized configuration: - Resource limits and reservations for all services - Proper health checks with start periods - JSON logging with rotation - Persistent volumes for data - Network isolation - Add deploy/deploy.sh deployment script: - Start/stop/restart/status/logs commands - Environment validation - Health check verification Closes #123
- Link to REQUIREMENTS.md from README setup section - Clarifies which requirements file to use based on environment - Addresses issue #157: document requirements.txt divergence
Add Automated Security Dependency Scanning
309 337 community forum e2e tests
…atures feat(llm): implement compliance logging and voice interface
[LLM] Build A/B testing framework for prompts and models
Predictive alert and multi lang
New feature branch
…oning-rate-limiting-pooling-graphql
…trics Add comprehensive database connection pooling configuration with health checks, metrics collection, and retry logic. - Add pool configuration to database.yaml (min_size, max_size, max_overflow, etc.) - Implement PoolConfig and HealthCheckConfig dataclasses - Add PoolMetrics for tracking connection acquisition times and errors - Add check_database_connection() for connection health verification - Add get_db_with_retry() for automatic connection retry with backoff - Add health endpoints: /health/db, /health/db/pool, /health/db/status - Add readiness and liveness probes for Kubernetes - Add admin endpoint to reset connection pool
Add comprehensive API rate limiting with multiple algorithms, Redis-backed distributed rate limiting, admin override capabilities, and rate limit headers. - Add sliding window algorithm as alternative to token bucket - Add Redis support for distributed rate limiting - Add rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) - Add admin override with whitelist/blacklist support - Add rate limit violation logging with context - Add admin endpoints for managing rate limits - Add per-endpoint rate limit configuration
Add comprehensive model versioning system with semantic versioning, rollback capability, A/B testing support, and deployment tracking. - Add semantic versioning (major.minor.patch) with auto-increment - Add model metadata fields (framework, task_type, description) - Add rollback capability to previous versions - Add A/B testing support with traffic splitting - Add deployment tracking with environment history - Add version comparison with metric deltas - Add version history with status transitions - Add performance metrics tracking and comparison - Add MLflow integration for experiment tracking
Add comprehensive GraphQL API as an alternative to REST for flexible data queries. - Add GraphQL schema for all data models (Account, Transaction, FraudAlert, LoyaltyPoints, ModelRegistry, etc.) - Add query support with pagination and filtering - Add mutation support for creating and updating entities - Add subscription support for real-time updates (transactions, fraud alerts, loyalty points) - Add authentication integration with JWT - Add query depth limiting for security - Add GraphQL playground for development - Add publish functions for GraphQL subscriptions
chore: rename context module to avoid import collision
Resolves conflicts between HEAD and upstream/main branches in the AstroML API application and model registry routers. Changes in api/app.py: - Merge both branches' routers (health, admin, GraphQL from HEAD; query_router from upstream/main) - Add GraphQL playground endpoint for development environments - Keep all router includes from both branches with clear commenting Changes in api/routers/models.py: - Add missing DeploymentEnvironment import from upstream/main - Rename duplicate /compare endpoint to /compare-versions to resolve route conflict - Preserve all additional functionality from HEAD: - Model version rollback - A/B testing endpoints - Model deployment to environments - Version comparison and history endpoints - Deployment history tracking
…te-limiting-pooling-graphql Feature/model versioning rate limiting pooling graphql
…antic-issue-362 Feature/llm cache redis semantic issue 362
…t-monitoring-issue-361 feat(llm): implement token usage tracking and cost monitoring (#361)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a fully containerized development and runtime environment for AstroML, enabling consistent setup across local machines, CI, and production deployments using Docker.
Changes Made
Added Dockerfile for building the AstroML application image.
Added docker-compose.yml for orchestrating services (app, database, and optional cache).
Introduced environment variable support via .env configuration.
Standardized runtime dependencies inside container.
Added health checks for application service.
Improved reproducibility of local development environment.
Problem
Previously, running AstroML required manual setup of:
system dependencies
Python/Node environment versions
database configuration
inconsistent local tooling setups
This led to:
onboarding friction
environment drift between developers
CI inconsistencies
Solution
Dockerization ensures:
identical runtime across environments
simplified onboarding (docker compose up)
isolated dependencies
reproducible builds in CI/CD pipelines
Key Features
Consistent runtime environment
Locked dependency versions
Optimized build layers for faster rebuilds
App service
Database service (e.g., PostgreSQL)
Optional Redis cache for background tasks or ML pipelines
.env driven configuration
Secure separation of secrets and runtime config
Ensures service readiness before dependency startup
Improves orchestration stability
Example Usage
docker compose up --build
Testing
Local Tests
Verified app builds successfully inside container
Confirmed database connectivity from app service
Tested hot reload in development mode
Validated environment variable injection
Integration Tests
Multi-container startup order
Service discovery between app and DB
Persistent volume storage validation
Impact
Simplified onboarding for new developers
Reduced environment-related bugs
Improved CI/CD consistency
Faster setup time (minutes instead of hours)
Better production parity
Type of Change
Infrastructure
DevOps
Developer Experience Improvement..closed [Infra] Dockerize AstroML Environment Traqora/astroml#78