🔴 INT-P08 (Dev.WeOwn.Tools) — Unexpected Instance Restart + Template Drift [REF: GTM_2026-W27_1024 + 1025]

[YonksTEAM]

---

title: "🔴 INT-P08 (Dev.WeOwn.Tools) — Unexpected Instance Restart + Template Drift [REF: GTM_2026-W27_1024 + 1025]"
labels: bug, instance-config, incident, int-p08, infra, p0
assignees: SHD, CTO

🔴 INCIDENT REPORT: INT-P08 UNEXPECTED RESTART + TEMPLATE DRIFT

Field	Value
Instance	INT-P08 (Dev.WeOwn.Tools — WeOwnLLM v1.14.1)
Incident REFs	GTM_2026-W27_1024 (Log) · GTM_2026-W27_1025 (ContextVolley)
Detected	00:52 MDT — 30 Jun 2026 (W27 D2) — MetaCouncil VSA agents all returned zero success
Diagnosed	01:04 MDT — @YonksTEAM:ADMIN confirmed incorrect template settings
Deep Dive	@ncimino 🏗️ — Root cause investigation completed
Resolution Status	⬜ PENDING — @mshahid538 🖥️ tasked with config restore
Priority	🔴 P0 — Blocking ALL MetaCouncil agents on INT-P08

---

❌ What Happened

INT-P08 experienced an unexpected restart during the W27 D1→D2 transition period. Upon restart, the instance loaded default WeOwnLLM templates instead of the custom configuration, causing three critical configuration drifts:

Setting	❌ DEFAULT (Loaded After Restart)	✅ CORRECT (Must Restore)
#LLMmodel	Anthropic: Claude Opus 4.5	DeepSeek V4 Flash
Stream Timeout	3000ms (AnythingLLM default)	10000ms
Embedding Provider	AnythingLLM Embedder → all-MiniLM-L6-v2	OpenRouter.ai → Perplexity 1V 4B

---

🔎 Root Cause Analysis

Factor	Detail
Trigger Event	INT-P08 WeOwnLLM v1.14.1 instance restarted unexpectedly — exact trigger under investigation
Root Cause	Instance restart caused loss of custom configuration — WeOwnLLM loaded factory default templates instead of saved custom config. Config persistence after restart appears unreliable in v1.14.1
Contributing Factor #1	No automated health check/alerting — restart was only detected when @YonksTEAM:ADMIN noticed MetaCouncil agents behaving incorrectly
Contributing Factor #2	No config backup or startup script — nothing re-applies custom settings after an unexpected restart
Detection Gap	~1+ hour gap between restart (late W27 D1) and detection (00:52 MDT W27 D2)

---

📋 Impact Assessment — Chain Reaction

INT-P08 RESTARTED
    ↓
Loaded DEFAULT template (not custom config)
    ↓
All agents switched from DeepSeek V4 Flash → Claude Opus 4.5
    ↓
Stream timeout cut from 10000ms → 3000ms
    ↓
Embedder changed from Perplexity 1V 4B → all-MiniLM-L6-v2
    ↓
RAG retrieval degraded (wrong embedder → wrong vector similarity)
    ↓
Agent responses inconsistent/failing
    ↓
MetaCouncil VSA attempts = ZERO SUCCESS across ALL agents

Component	Impact	Severity
🔬｜DRP｜Research	DRP.bot 🛡️	🔴 HIGH
🚀｜MAIT-Postiz	Newly initiated MAIT agent	🟡 MEDIUM
ALL AGENTS	Stream timeout too low → timeouts on long tool calls	🔴 HIGH
RAG Vector Store	Wrong embedder → retrieval failure	🔴 HIGH
🔴 MetaCouncil VSA	Failed — agents couldn't execute properly	🔴 HIGH

---

🛠️ Resolution Steps — Tasked to @mshahid538 🖥️

#	Action	Detail	Priority	Status
1	Diagnose restart cause	Check WeOwnLLM v1.14.1 logs on INT-P08 — why did it restart?	🔴 P0	⬜ PENDING
2	Restore #LLMmodel	Set back to DeepSeek V4 Flash for ALL workspaces	🔴 P0	⬜ PENDING
3	Restore Stream Timeout	Set to 10000ms	🔴 P0	⬜ PENDING
4	Restore Embedder	Set to OpenRouter.ai → Perplexity 1V 4B	🔴 P0	⬜ PENDING
5	Verify persistence	Ensure config survives next restart	🟠 P1	⬜ PENDING
6	RAG Re-sync	Re-sync MetaCouncil.md + recent docs to vector store	🟠 P1	⬜ PENDING
7	Smoke test	Verify agent responsiveness post-restore	🔴 P0	⬜ PENDING
8	Document root cause	Log findings in repo	🟡 P2	⬜ PENDING

---

Reference Configuration (Known Correct)

Setting	Value	Source
#LLMmodel	DeepSeek V4 Flash — 1M context	INT-P08 workspace prompt (v4.1.5.1-r1)
Stream Timeout	10000ms	INT-B001 proven stable across W26+W27
Embedder	OpenRouter.ai → Perplexity 1V 4B	Prior INT-P08 known-good config

---

🛡️ #BestPractices — Lessons for Prevention

Lesson	Detail	Status
Config Persistence	WeOwnLLM v1.14.1 lost custom config on restart — investigate if this is a bug or expected behavior	🔒 IMMUTABLE TRACKING
Health Check Monitoring	No automated alerting exists for INT-P08 restarts. Need heartbeat or uptime monitor.	🟡 PROPOSED
Startup Script	Consider a startup script that re-applies known-good config after any restart	🟡 PROPOSED
Backup Config	Export known-good config as JSON/YAML — store in this repo for rapid restore	🟡 PROPOSED
Detection Timeline	~1hr gap between restart and detection. Auto-notification on instance restart would close this to <1min.	🟡 PROPOSED

---

🔗 Related References

Reference	Description
[REF: GTM_2026-W27_1024]	Infrastructure issue log — INT-P08 template drift
[REF: GTM_2026-W27_1025]	ContextVolley → @mshahid538 (CC: @ncimino) — Resolution tasking
ContextVolley → @mshahid538 (01:13 MDT)	Full mission brief with 8-step resolution checklist
WeOwnLLM Repo — Issue #1	API key lifecycle management (parallel infra concern)

---

🏷️ Labels

bug instance-config incident int-p08 infra p0 template-drift config-restore

---

🔄 Status Updates

Leave updates as comments below. @mshahid538 🖥️ please comment when config restore is complete.

• Initial Report: 30 Jun 2026 — 11:37 MDT — @YonksTEAM 🎯

---

[mshahid538]

⁨📷⁩ ⁨[REF: SHD_2026-W27_5040] ⚙️ #Infra:Status | INT-P08 Configuration Restored & IaC Hardened

Hi @YonksTEAM

I have successfully resolved Issue #1 regarding the template drift on the INT-P08 (Dev.WeOwn.Tools) instance.

To prevent this configuration from disappearing on future container restarts, I transitioned the system metrics into our Git repository manifests as declarative code.

I have verified the environment state directly from the server container backend:

LLM Model: Pinned to deepseek/deepseek-v4-flash

Stream Timeout: Extended to 10000ms (OPENROUTER_TIMEOUT_MS + LLM_STREAM_TIMEOUT)

Embedding Model: Pinned to perplexity/perplexity-1v-4b via OpenRouter

Since I only hold a standard user profile on this instance, could you, with Admin dashboard rights, please perform a quick front-end validation check?

Re-sync MetaCouncil.md in the workspace to align the vector chunks with the new Perplexity embedding dimensions.

Fire a test request to Aegis 🛡️ to confirm response stream handling.

Changes pushed to GitHub branch:“feature/dev-weown-tools-deployment”.

I am moving directly onto the remaining MCP server validation tasks in GitHub Issue #2 next. Let's drive it forward!

CC: @ncimino