diff --git a/README.md b/README.md index 7cf1ddd2..10f69e7f 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,7 @@ Most fine-tuning tools stop at "the model trained." ForgeLM produces the artefac - **EU AI Act** — auto-generated Annex IV technical documentation, Article 10 data governance, Article 14 human-oversight staging gate - **GDPR** — `forgelm purge` (Article 17 right-to-erasure) and `forgelm reverse-pii` (Article 15 right-of-access) +- **Model & log integrity** — a SHA-256 manifest per trained model (`forgelm verify-integrity`) and a tamper-evident audit chain (`forgelm verify-audit`) give you a one-command proof-of-integrity before you ship - **Append-only audit log** — HMAC-chained when `FORGELM_AUDIT_SECRET` is configured; every decision gate emits a structured event - **Supply-chain hardening** — CycloneDX 1.5 SBOM per release, nightly `pip-audit` + `bandit`, `gitleaks` pre-commit - **ISO 27001 / SOC 2 alignment** — software cannot be certified, but ForgeLM produces the change-management, data-lineage, and audit-trail evidence your deployer's auditor needs. See the [Deployer Audit Guide](docs/guides/iso_soc2_deployer_guide.md). diff --git a/docs/design/data_audit_cli_split.md b/docs/design/data_audit_cli_split.md index 60d63271..dd8dbbee 100644 --- a/docs/design/data_audit_cli_split.md +++ b/docs/design/data_audit_cli_split.md @@ -597,7 +597,7 @@ A PR in either series is "accepted" when ALL of the following hold: 10. **ruff format + ruff check clean.** 11. **No new top-level imports of torch / transformers / unsloth / deepspeed / lm_eval inside `forgelm/cli/**` or `forgelm/data_audit/**`.** 12. **Logger names unchanged.** -13. **Public exit codes preserved:** `EXIT_SUCCESS == 0`, `EXIT_CONFIG_ERROR == 1`, `EXIT_TRAINING_ERROR == 2`, `EXIT_EVAL_FAILURE == 3`, `EXIT_AWAITING_APPROVAL == 4`, `_PUBLIC_EXIT_CODES == frozenset({0,1,2,3,4})`. +13. **Public exit codes preserved:** `EXIT_SUCCESS == 0`, `EXIT_CONFIG_ERROR == 1`, `EXIT_TRAINING_ERROR == 2`, `EXIT_EVAL_FAILURE == 3`, `EXIT_AWAITING_APPROVAL == 4`, `EXIT_WIZARD_CANCELLED == 5`, `_PUBLIC_EXIT_CODES == frozenset({0,1,2,3,4,5})`. --- diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 309d2fce..ec097e6b 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -80,14 +80,14 @@ Two pre-existing surfaces overlap with this design and must be resolved up front **Conflict-resolution semantics (dual-set window v0.5.5–v0.6.x):** - - When only `evaluation.staging_ttl_days` is set → alias-forward to `retention.staging_ttl_days` and emit a single `DeprecationWarning` naming the new field + the v0.7.0 removal target. + - When only `evaluation.staging_ttl_days` is set → alias-forward to `retention.staging_ttl_days` and emit a single `DeprecationWarning` naming the new field + the v0.8.0 removal target. - When only `retention.staging_ttl_days` is set → no warning; canonical path. - In the case where **both** are set with **identical** values → emit `DeprecationWarning` for the deprecated field; the canonical `retention.staging_ttl_days` value is used; the operator's intent is unambiguous. - If **both** are set with **different** values → raise `ConfigError` at validation time naming both keys, both values, and instructing the operator to remove the deprecated entry. Silent winner = wrong winner; ambiguous configs are refused. Phase 21 ships `tests/test_config.py::test_staging_ttl_days_dual_set_with_different_values_refused` asserting the `ConfigError` and that the message mentions both keys + both values. - **Tracking issue (per `docs/standards/release.md:95`):** Phase 21 also files a tracking issue `'Remove EvaluationConfig.staging_ttl_days in v0.7.0'` and links it from both the `DeprecationWarning` message text and the v0.5.5 CHANGELOG `### Deprecated` entry. The v0.7.0 removal PR closes that issue. + **Tracking issue (per `docs/standards/release.md:95`):** Phase 21 also files a tracking issue `'Remove EvaluationConfig.staging_ttl_days in v0.8.0'` and links it from both the `DeprecationWarning` message text and the v0.5.5 CHANGELOG `### Deprecated` entry. The v0.8.0 removal PR closes that issue. - **`docs/usermanuals/en/compliance/gdpr.md` lines 51-64** + closure plan §15.5 row GH-023 reference an `ingestion.retention.raw_documents.ttl_days` shape. This design (§10 Q1) standardises on the top-level `retention.*` form because the policy covers more than ingestion artefacts (audit logs, staging dirs, ephemeral snapshots). Phase 21 updates the GDPR user-manual page in the same PR; closure plan §15.5 GH-023 entry is amended to "absorbed under top-level `retention.*`" rather than carrying the `ingestion.retention.*` shape forward. ### 3.2 Schema diff --git a/docs/design/iso27001_soc2_alignment.md b/docs/design/iso27001_soc2_alignment.md index 96a4668c..0972d41a 100644 --- a/docs/design/iso27001_soc2_alignment.md +++ b/docs/design/iso27001_soc2_alignment.md @@ -122,7 +122,8 @@ auditable evidence. Cited symbols are real (verified against | HTTP discipline | `safe_post` (Phase 7) | SSRF guard, TLS-only, no-redirect, header masking | | Data audit pipeline | `forgelm audit` (Phase 11) | `data_audit_report.json` (PII / secrets / dedup) | | Annex IV §8 governance | `_data_governance_block` (compliance.py) | `data_governance_report.json` Article 10 | -| Model integrity | `generate_model_integrity` (delegates to `_hash_file`) | `model_integrity.json` | +| Model integrity generation | `generate_model_integrity` (delegates to `_hash_file`) | `model_integrity.json` | +| Model-integrity manifest verification | `forgelm verify-integrity` (Wave 2b / Art. 15) | exit 0/1/2, JSON envelope | | SBOM | `tools/generate_sbom.py` (Wave 2 era) | CycloneDX 1.5 JSON per (OS, py-version), uploaded as release artefact | | Bilingual EN+TR docs | `tools/check_bilingual_parity.py` (Phase 24) | strict-mode diff exit code | @@ -251,7 +252,7 @@ ForgeLM contributes nothing here. | A.8.21 Security of network services | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | TLS 1.2+ enforcement; cert rotation | | A.8.22 Segregation of networks | OOS | — | VPC / subnet design | | A.8.23 Web filtering | OOS | — | Egress proxy | -| A.8.24 Use of cryptography | FL | SHA-256 + HMAC-SHA-256 (audit chain key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`, see `forgelm/compliance.py:104-114`); salted SHA-256 identifier hashing for purge / reverse-pii (`_purge._resolve_salt`, distinct concern) | KMS for `FORGELM_AUDIT_SECRET` | +| A.8.24 Use of cryptography | FL | SHA-256 + HMAC-SHA-256 (audit chain key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`, see `AuditLogger.__init__`, `forgelm/compliance.py:164`); salted SHA-256 identifier hashing for purge / reverse-pii (`_purge._resolve_salt`, distinct concern) | KMS for `FORGELM_AUDIT_SECRET` | | A.8.25 Secure development life cycle | FL-helps | `docs/standards/code-review.md` + `docs/standards/release.md` + CI gates (ruff, pytest, dry-run, parity, SBOM) | SDLC framework | | A.8.26 Application security requirements | FL-helps | F-compliance-110 strict gate; Pydantic config validation; ReDoS guard in `_reverse_pii` | App-level threat modelling | | A.8.27 Secure system architecture and engineering principles | FL-helps | Append-only audit log architecture (`flock`+`fsync` per line); audit chain HMAC; lazy import discipline; `safe_post` SSRF guard | Defence-in-depth design | diff --git a/docs/guides/alignment-tr.md b/docs/guides/alignment-tr.md index 44a3f067..1797a61b 100644 --- a/docs/guides/alignment-tr.md +++ b/docs/guides/alignment-tr.md @@ -61,8 +61,7 @@ Ya da modern `messages` formatı: ```yaml model: name_or_path: "meta-llama/Llama-3.1-8B-Instruct" - max_length: 4096 - backend: "transformers" + load_in_4bit: true lora: r: 16 @@ -266,10 +265,7 @@ forgelm --config configs/stage2_dpo.yaml forgelm --config configs/stage3_grpo.yaml ``` -> **Planlanan (Faz 14 — pipeline chains):** Tek bir YAML dosyasında -> çok-aşamalı eğitim zincirlerini tanımlayan bir `pipeline:` config -> anahtarı, aşamalar arası manuel config curating'i ortadan kaldıracak. -> Takip issue'su v0.6.0+ release penceresine bağlı. +> **v0.7.0'dan itibaren mevcut (Faz 14):** `pipeline:` config bloğu, tek bir YAML'da çok-aşamalı eğitimi (SFT → DPO → GRPO) zincirler. Tam kılavuz için [Çok Aşamalı Eğitim Pipeline'ları](pipeline-tr.md) sayfasına bakın. --- diff --git a/docs/guides/alignment.md b/docs/guides/alignment.md index 36096960..07e134f7 100644 --- a/docs/guides/alignment.md +++ b/docs/guides/alignment.md @@ -227,7 +227,7 @@ forgelm --config configs/stage2_dpo.yaml forgelm --config configs/stage3_grpo.yaml ``` -> **Planned (Phase 14 — pipeline chains):** A `pipeline:` config key will define multi-stage training chains in a single YAML file, eliminating manual config juggling between stages. Tracking issue links the v0.6.0+ release window. +> **Available since v0.7.0 (Phase 14):** The `pipeline:` config block chains multi-stage training (SFT → DPO → GRPO) in a single YAML. See [Multi-Stage Training Pipelines](pipeline.md) for the full guide. --- diff --git a/docs/guides/cicd_pipeline-tr.md b/docs/guides/cicd_pipeline-tr.md index 6970272f..0fe0afad 100644 --- a/docs/guides/cicd_pipeline-tr.md +++ b/docs/guides/cicd_pipeline-tr.md @@ -116,7 +116,6 @@ stages: - validate - train - evaluate - - deploy validate: stage: validate diff --git a/docs/guides/cicd_pipeline.md b/docs/guides/cicd_pipeline.md index b3f70df5..46156b04 100644 --- a/docs/guides/cicd_pipeline.md +++ b/docs/guides/cicd_pipeline.md @@ -8,7 +8,7 @@ ForgeLM is built for automation. This guide shows how to integrate fine-tuning i ForgeLM's CI/CD-native design provides: - **YAML-driven**: Entire training runs defined in version-controlled config files -- **Meaningful exit codes**: `0` success, `1` config error, `2` training error, `3` eval failure +- **Meaningful exit codes**: `0` success, `1` config error, `2` training error, `3` eval failure, `4` awaiting approval - **JSON output**: `--output-format json` for machine-readable results - **Dry-run validation**: `--dry-run` validates without GPU - **Webhook notifications**: Real-time Slack/Teams alerts on start/success/failure diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 9bc74def..5cc4981c 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -192,9 +192,10 @@ Faz 11.5 simhash backend'ini de değiştirilebilir hale getirdi: - **xxhash.xxh3_64** opsiyonel `xxhash` bağımlılığı (artık `forgelm[ingestion]`'ın parçası) yüklüyse per-token digest'i bu - sürüyor; kısa anahtarlarda BLAKE2b'ye göre Python katmanında ~%30 - hızlandırma sağlıyor (lru_cache devreye girince end-to-end kazanç - daha az; backend swap'i öncelikle ileriye dönük güvence). + sürüyor; ~1.3x ham digest başına, ~1.05x end-to-end (Apple Silicon / + Python 3.11 ölçümü) hızlandırma sağlıyor (`compute_simhash` içinde + `lru_cache` tekrar eden token'ları absorbe ettiğinden end-to-end + kazanç daha az; backend swap'i öncelikle ileriye dönük güvence). - **BLAKE2b** bare install için fallback olarak kalıyor. - Modül seviyesinde `lru_cache(maxsize=10_000)` digest'i token seviyesinde memoize ediyor — Zipfian token frekansı sayesinde küçük diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 7ba344fc..acac00dc 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -93,7 +93,7 @@ Wave 1 `evaluation.staging_ttl_days` field'ı (v0.5.5'te shipped) **deprecate** - İkisi de **aynı** değerlerle açıkça set'liyse `DeprecationWarning` yayılır (canonical block kazanır). - İkisi de **farklı** değerlerle açıkça set'liyse config-load zamanında `ConfigError` raise edilir. Sessiz kazanan = yanlış kazanan. -Deprecate edilen field **v0.7.0**'da kaldırılır. +Deprecate edilen field **v0.8.0**'da kaldırılır. ## Audit-event sözlüğü @@ -185,5 +185,5 @@ glob); `2` = runtime hatası (mid-scan I/O failure). JSON envelope ## Bkz. - `docs/qms/sop_data_management.md` — retention + erasure prosedürleri dahil tam data-lifecycle SOP. -- `docs/usermanuals/tr/compliance/safety_compliance.md` — buraya link veren operator-facing compliance overview. +- `docs/usermanuals/tr/compliance/overview.md` — buraya link veren operator-facing compliance overview. - [`docs/design/gdpr_erasure.md`](../design/gdpr_erasure.md) — bu implementation'ın gerçekleştirdiği Phase 20 design dokümanı. diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 1a770756..7998d6ee 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -93,7 +93,7 @@ The Wave 1 `evaluation.staging_ttl_days` field (shipped in v0.5.5) is **deprecat - Setting both explicitly with **identical** values emits a `DeprecationWarning` (the canonical block wins). - Setting both explicitly with **different** values raises `ConfigError` at config-load time. Silent winner = wrong winner. -The deprecated field is removed in **v0.7.0**. +The deprecated field is removed in **v0.8.0**. ## Audit-event vocabulary @@ -184,5 +184,5 @@ for the JSON envelope schema. ## See also - `docs/qms/sop_data_management.md` — the full data-lifecycle SOP including the retention + erasure procedures. -- `docs/usermanuals/en/compliance/safety_compliance.md` — the operator-facing compliance overview that links here. +- `docs/usermanuals/en/compliance/overview.md` — the operator-facing compliance overview that links here. - [`docs/design/gdpr_erasure.md`](../design/gdpr_erasure.md) — the Phase 20 design document that this implementation realises. diff --git a/docs/guides/library_api-tr.md b/docs/guides/library_api-tr.md index af1cf8e3..54bfc91c 100644 --- a/docs/guides/library_api-tr.md +++ b/docs/guides/library_api-tr.md @@ -12,7 +12,7 @@ ForgeLM **eşit derecede first-class iki giriş noktası** yayınlar: shell pipe **CLI**'yı seçin: -- Bir Bash / GitHub Actions / GitLab CI pipeline yayınlıyorsanız. Exit-code kontratı (0/1/2/3/4) entegrasyon yüzeyidir. +- Bir Bash / GitHub Actions / GitLab CI pipeline yayınlıyorsanız. Exit-code kontratı (0/1/2/3/4/5) entegrasyon yüzeyidir. - ForgeLM'in stdout'a kutudan çıkar çıkmaz yaydığı yapılandırılmış logging + JSON envelope'larını istiyorsanız. - Aşama başına bir process'in operasyonel birim olduğu altyapıda (çoğu CI runner, çoğu Argo pipeline) çalışıyorsanız. @@ -47,8 +47,8 @@ assert "torch" not in sys.modules # Ama public yüzey autocomplete için tam keşfedilebilir. print("ForgeTrainer" in dir(forgelm)) # True -print(forgelm.__version__) # ör. "0.5.5" -print(forgelm.__api_version__) # ör. "0.5" +print(forgelm.__version__) # ör. "0.7.0" +print(forgelm.__api_version__) # ör. "1.0.0" ``` Wheel ile birlikte yayınlanan çalıştırılabilir bir notebook (`notebooks/library_api_example.ipynb`) bu sayfanın kapsadığı aynı üç deseni yürür — provenance için tasarım dokümanı Faz 19 görev #13'e bakın. diff --git a/docs/guides/library_api.md b/docs/guides/library_api.md index cad4f768..26ca5659 100644 --- a/docs/guides/library_api.md +++ b/docs/guides/library_api.md @@ -12,7 +12,7 @@ ForgeLM ships **two equally first-class entry points**: the `forgelm` console sc Choose the **CLI** when: -- You're shipping a Bash / GitHub Actions / GitLab CI pipeline. The exit-code contract (0/1/2/3/4) is the integration surface. +- You're shipping a Bash / GitHub Actions / GitLab CI pipeline. The exit-code contract (0/1/2/3/4/5) is the integration surface. - You want the structured logging + JSON envelopes ForgeLM emits to stdout out of the box. - You're running on infrastructure where one process per stage is the operational unit (most CI runners, most Argo pipelines). @@ -47,8 +47,8 @@ assert "torch" not in sys.modules # But the public surface is fully discoverable for autocomplete. print("ForgeTrainer" in dir(forgelm)) # True -print(forgelm.__version__) # e.g. "0.5.5" -print(forgelm.__api_version__) # e.g. "0.5" +print(forgelm.__version__) # e.g. "0.7.0" +print(forgelm.__api_version__) # e.g. "1.0.0" ``` A runnable notebook (`notebooks/library_api_example.ipynb`) ships with the wheel and walks the same three patterns this page covers — see the design doc Phase 19 task #13 for its provenance. diff --git a/docs/guides/performance-tr.md b/docs/guides/performance-tr.md index 30c6346a..0b385a3f 100644 --- a/docs/guides/performance-tr.md +++ b/docs/guides/performance-tr.md @@ -192,4 +192,4 @@ GaLore belleği compute ile takas eder. Projeksiyon matris çarpımı gerçek ov - [`library_api-tr.md`](library_api-tr.md) — bu düğmeleri Python'dan çağırma. - [`../standards/coding.md`](../standards/coding.md) — ForgeLM'in dayattığı lazy-import standardı. - [`ingestion-tr.md`](ingestion-tr.md) — chunker kullanıcı-bakışlı derinlik. -- [`alignment.md`](alignment-tr.md) — GaLore + 4-bit + packing'in ne zaman uygun olduğu. +- [`alignment-tr.md`](alignment-tr.md) — GaLore + 4-bit + packing'in ne zaman uygun olduğu. diff --git a/docs/guides/pipeline-tr.md b/docs/guides/pipeline-tr.md index fcfdc65e..6e865a94 100644 --- a/docs/guides/pipeline-tr.md +++ b/docs/guides/pipeline-tr.md @@ -1,6 +1,6 @@ # Çok Aşamalı Eğitim Pipeline'ları -Faz 14 — `v0.7.0` ile geliyor (CHANGELOG kaydı 2026-05-14; PyPI tag bekleniyor). +Faz 14 — `v0.7.0` ile gönderildi (2026-05-14). ForgeLM'in `pipeline:` config bloğu, 2 veya daha fazla eğitim aşamasını (genellikle SFT → DPO → GRPO) tek bir config-tabanlı, dry-run-doğrulanabilir, diff --git a/docs/guides/pipeline.md b/docs/guides/pipeline.md index bd3cef3d..579dd7b0 100644 --- a/docs/guides/pipeline.md +++ b/docs/guides/pipeline.md @@ -1,6 +1,6 @@ # Multi-Stage Training Pipelines -Phase 14 — landing in `v0.7.0` (CHANGELOG entry dated 2026-05-14; PyPI tag pending). +Phase 14 — shipped in `v0.7.0` (2026-05-14). ForgeLM's `pipeline:` config block chains 2 or more training stages (typically SFT → DPO → GRPO) into one config-driven, dry-run-validatable, diff --git a/docs/guides/safety_compliance-tr.md b/docs/guides/safety_compliance-tr.md index 8972538a..da016bcb 100644 --- a/docs/guides/safety_compliance-tr.md +++ b/docs/guides/safety_compliance-tr.md @@ -350,7 +350,7 @@ require_hmac=False)`, bir `VerifyResult` dataclass'ı (`valid`, ```json { - "forgelm_version": "0.5.5", + "forgelm_version": "0.7.0", "generated_at": "2026-03-23T14:30:00+00:00", "model_lineage": { "base_model": "meta-llama/Llama-3.1-8B-Instruct", @@ -478,7 +478,7 @@ evaluation: **Ne olur:** 1. Eğitim tamamlanır, tüm otomatik değerlendirmeler geçer -2. Model nihai dizine kaydedilir +2. Model staging dizinine kaydedilir (`final_model.staging./`) 3. ForgeLM **kod 4** ("onay bekleniyor") ile çıkar 4. Bir insan değerlendirme sonuçlarını, model card'ı ve uyumluluk artefaktlarını inceler diff --git a/docs/guides/safety_compliance.md b/docs/guides/safety_compliance.md index c488dbcb..588d09ab 100644 --- a/docs/guides/safety_compliance.md +++ b/docs/guides/safety_compliance.md @@ -305,7 +305,7 @@ The library function `forgelm.compliance.verify_audit_log(path, *, hmac_secret=N ```json { - "forgelm_version": "0.5.5", + "forgelm_version": "0.7.0", "generated_at": "2026-03-23T14:30:00+00:00", "model_lineage": { "base_model": "meta-llama/Llama-3.1-8B-Instruct", @@ -431,7 +431,7 @@ evaluation: **What happens:** 1. Training completes, all automated evaluations pass -2. Model is saved to the final directory +2. Model is saved to the staging directory (`final_model.staging./`) 3. ForgeLM exits with **code 4** ("awaiting approval") 4. A human reviews the evaluation results, model card, and compliance artifacts 5. The human approves or rejects the model diff --git a/docs/guides/troubleshooting-tr.md b/docs/guides/troubleshooting-tr.md index da290656..dc113f7b 100644 --- a/docs/guides/troubleshooting-tr.md +++ b/docs/guides/troubleshooting-tr.md @@ -197,7 +197,7 @@ synthetic: max_new_tokens: 512 # asılı kalıyorsa teacher yanıt boyutunu sınırla ``` -`SyntheticConfig`, v0.5.5'te ayrı retry / batch ayarları yüzeylemiyor — +`SyntheticConfig`, v0.7.0'da ayrı retry / batch ayarları yüzeylemiyor — retry'lar HTTP-client katmanında ele alınır ve batch size API çağrısı başına bir prompt'a sabittir. Phase 28+ backlog'u açık retry-count ve batched-call parametreleri eklemeyi takip ediyor. @@ -414,6 +414,7 @@ docker run --gpus all --shm-size=16g ... | `2` | Eğitim hatası | GPU, bellek, bağımlılıkları kontrol edin | | `3` | Değerlendirme arızası | Model kalitesi eşiğin altında — eşikleri ayarlayın ya da veriyi iyileştirin | | `4` | Onay bekleniyor | İnsan incelemesi gerekli — staging dizinini incelemek için `forgelm approvals --show --output-dir ` çalıştırın, sonra promote için `forgelm approve --output-dir ` ya da kalıcı reject için `forgelm reject --output-dir `. Staging yolu `/final_model.staging./`'dir. | +| `5` | Sihirbaz iptal | Sihirbaz iptal edildi (Ctrl-C veya açık iptal) — config yazılmadı; `--wizard` ile yeniden çalıştırın ya da etkileşimsiz kullanım için `forgelm quickstart