merge: integrate LAMB monorepo, Knowledge Stores, Library Manager, KV-cache and cost management#459
Open
Fr4n9 wants to merge 409 commits into
Open
merge: integrate LAMB monorepo, Knowledge Stores, Library Manager, KV-cache and cost management#459Fr4n9 wants to merge 409 commits into
Fr4n9 wants to merge 409 commits into
Conversation
…back loop
Step components dispatched 'update' events on every $effect run; the wizard's
handleStateUpdate replaced wizardState with a new object, which re-triggered
the step's $effect, dispatching again -- an infinite loop that suspended the
parent's reactivity (so wizardOpen=false from Esc/X/done never re-rendered
{#if wizardOpen}).
Fixes:
- handleStateUpdate skips no-op updates (deep-compares object/array values
via JSON serialization).
- Step0/Step1 wrap their dispatch logic in untrack() so reading wizardState
inside the effect doesn't subscribe.
- Wizard no longer owns isOpen as a $bindable; the parent's {#if wizardOpen}
controls visibility, and the wizard fires onclose() to notify.
- Parent /libraries route attaches a top-level <svelte:window> Esc handler
while the wizard is open; X-button and backdrop-click both go through the
same close path.
KnowledgeStoresList was overwriting the backend's correct is_owner field with a client-side comparison s.owner_user_id === \$user.id, but the user store does not expose an id property (only token/name/email). Every store ended up with is_owner=false, sending all owned stores into the "Shared" tab. Drop the broken remap and use the backend's is_owner directly -- matches LibrariesList's pattern.
When creating a Knowledge Store the caller may omit embedding_endpoint
(common for OpenAI; required for self-hosted Ollama or local). LAMB now
falls back to setups.default.providers.{vendor}.endpoint (or base_url /
api_endpoint) instead of forwarding an empty string to the KB Server.
- OrganizationConfigResolver.get_provider_endpoint(vendor) helper.
- create_knowledge_store resolves the endpoint via the resolver and
persists the resolved value, so add-content/query also pick it up.
ESLint cleanups across the new Knowledge Store components: - Use SvelteSet from svelte/reactivity for selectedIds (Set is not reactive in Svelte 5). - Drop unused 'tick' import in CreateKnowledgeWizard. - Drop unused listContent import in KnowledgeStoreDetail. - Bracket the /docs permalink anchors with eslint-disable for svelte/no-navigation-without-resolve (these are backend API URLs, not SvelteKit routes). - Drop unused errorMessage helper in knowledgeStoreService. - Drop svelte-ignore comments that no longer match a real warning.
…dator ChromaDB 0.5+ eagerly validates the OpenAI embedding function at collection-create time, raising ValueError if no api_key is set. Knowledge Store create only sends the real embedding key per-request on add-content/query. Provide harmless placeholders (OPENAI_API_KEY, EMBEDDINGS_APIKEY) in the kb-server v2 service so collection creation succeeds; per-request keys still override.
§19 documents how the implementation is verified: four Opus subagents in parallel review the backend, frontend, CLI, and test suites; Claude Code orchestrates them and runs the deterministic Playwright + pytest suites. Records the test commands and the success criteria.
The Knowledge Store + Library specs share LAMB DB state and snapshot resource counts to assert no leakage. With the previous default of unbounded workers in dev mode, Playwright spawned multiple workers and the specs interleaved -- one spec's afterAll deletion ran while another spec's beforeAll snapshotted counts, producing flaky "Expected 2, Received 1" assertions. Force workers=1 unconditionally (matches CI). fullyParallel was already false but only governs intra-file parallelism; without workers=1 the specs themselves still ran in parallel.
Run `npm run format` once across the tree to flush format debt that had been hidden by prettier-plugin-svelte crashing on every .svelte file at the prettier@^3.4.2 -> 3.8.x resolution. Entirely mechanical (whitespace, quote style, trailing commas); zero runtime behaviour change. The prettier version pin (3.5.3) that lets this run cleanly lands in the feature commit alongside the actual code changes.
Backend
- Org-aware /creator/knowledge-stores/options: the api_endpoint default for
each embedding vendor now overrides the static plugin default with the org's
configured endpoint. Fixes the case where the kb-server v2 container can't
reach Ollama on http://localhost:11434.
- library-manager rejects 0-byte file uploads with HTTP 400 before queuing the
import job (was silently producing empty 'ready' items).
- simple_augment falls back to a default {context}+{user_input} prompt template
when the assistant's prompt_template is empty AND a non-no_rag processor is
selected (was silently dropping retrieved RAG context).
- kb-server v2 logs an INFO 'Plugin load summary' line at startup listing the
embedding/vector_db/chunking plugins that loaded and any modules that failed.
- Add ollama>=0.3.0 to lamb-kb-server's pyproject (chromadb's
OllamaEmbeddingFunction imports it lazily; without it ks create returned 500).
- New backend pytest.ini scopes pytest collection to backend/tests/ so the
standalone test_gemini_image.py script isn't auto-discovered.
CLI (lamb-cli)
- Add --knowledge-store/--ks flag (repeatable) to lamb assistant create/update,
mapping to RAG_collections (the binding field for knowledge_store_rag).
- Default a {context}+{user_input} prompt_template when --rag-processor is
non-no_rag and no template is given, matching the backend fallback.
- Add --wait/--max-wait polling to lamb library upload/import-url/import-youtube
(mirrors knowledge_store add-content).
- Render lamb ks options as labeled sections instead of an empty detail view.
- Coerce None to '' in save_credentials (the LAMB login response sometimes
returns null for optional profile fields, which TOML cannot serialize).
Frontend
- AssistantForm: when rag_processor === 'knowledge_store_rag', branch off the
legacy KB picker and render a Knowledge Store picker fed by
knowledgeStoreService. Persist selection into RAG_collections. Edit-mode
pre-selection works. data-testid hooks added for Playwright.
- LibraryDetail: handleDeleteItemConfirm now closes the modal on error and
surfaces the backend's structured 409 conflict body, including the names of
Knowledge Stores referencing the item.
- Add knowledge.wizard.step9.heading i18n key to en/es/ca/eu (was falling back
silently to the inline 'You're all set' default).
- Pin prettier to 3.5.3 (compatible with prettier-plugin-svelte 3.5.1).
Tests
- New lamb-cli tests: test_knowledge_store.py (20+), test_library.py (75).
- New backend integration tests: test_creator_knowledge_stores_integration.py,
test_creator_libraries_integration.py, test_fr10_interlock.py,
test_knowledge_store_options.py, test_simple_augment.py + conftest.py.
- New Playwright specs: assistant_with_knowledge_store.spec.js, fr10_ui.spec.js,
library_wait_polling.spec.js.
- Extend lamb-cli test_assistant.py and test_chat.py for the new flag and
pytest-httpx 0.36 stricter mock matching. Fix vitest selector in
src/routes/page.svelte.test.js (h1 was auth-gated; target a stable element).
- library-manager test_edge_cases.py extended with a 0-byte upload regression.
Test counts: lamb-cli 238 -> 322, backend 39 -> 70, library-manager 52 -> 53,
lamb-kb-server v2 520 -> 580. Three new Playwright specs all
green; cross-browser smoke also green.
Docs
- Documentation/pre_existing_debt_handoff.md catalogues frontend ESLint /
svelte-check / vitest debt, backend dep CVEs, the OWI<->LAMB role-sync drift,
the Firecrawl key gap, and the AssistantForm Advanced-Mode UX gotcha — all
documented as separate-pass follow-ups outside this lifecycle verification.
- lamb-kb-server/Documentation/issue_334_known_bugs.md records the kb-server v2
bugs surfaced and fixed in this run.
…RGE_DEV_TO_PHASE4.md for details
- Fix +layout.js: change /i18n imports to @lamb/ui - Fix Nav.svelte: use relative paths for @lamb/ui internal imports, add hardcoded getConfig() workaround (window.LAMB_CONFIG) since config.js only exists in creator-app - Fix assistantStore.js: change ./userStore to @lamb/ui - Export renderMarkdownSafe and sanitizeHtml from @lamb/ui index.js - Add marked and dompurify as dependencies to @lamb/ui package.json - Update MERGE_DEV_TO_PHASE4.md with new TODOs discovered during build: subpath imports fix, dead imports cleanup, module-file-eval XSS sanitization, createEventDispatcher migration, hooks.server.js i18n review
…-microservice feat(#334): new KB Server microservice (lamb-kb-server/)
Three tidy-ups left over from the merge of projects/refactor/kbserver-lamb-integration: - Re-apply prettier to adminService.js, apiClient.js, sanitize.js (missed by the canonical formatting pass during conflict resolution). - Fix four eslint errors introduced in #337-new code: drop dead handleKeydown in CreateKnowledgeWizard (Esc handling lives in the parent), add eslint-disable-next-line for goto() calls that follow the codebase's existing pattern, and remove unused catch bindings in apiClient.js + adminService.js via optional catch binding. - Update issue_334_known_bugs.md to reflect that all four bugs are fixed in the merged code (commits 65a9cce, 3b7eefb, 4792f24, 434a6bc) with a fix-commit lookup table. Net eslint count goes from 380 to 374, matching origin/main's baseline. All 70 backend tests still pass.
The CLI's happy paths and basic flag validation were already well-covered (75 tests on library + ks). What was missing: assertions that the CLI correctly surfaces backend guardrails — FR-10 deletion conflicts, locked config rejection, async-polling cadence, and a wide range of HTTP status propagation. A regression that broke the CLI's *handling* of those signals would have slipped through. Adds 40 new tests across two files: lamb-cli/tests/test_commands/test_knowledge_store.py (+25) - TestKsLockedConfig — Click rejects --chunking, --embedding-vendor, --embedding-model, --vector-db, --embedding-endpoint on `ks update`; also confirms the rename payload never includes locked fields. - TestKsBackendErrors — 409, 403, 503, 500, 422 propagate as non-zero exits; 500-on-create has no leaked provisional row in `ks list`. - TestKsAddContentEdgeCases — 404 (NotFoundError typed), 409 retry, failure error_message visible in --wait output, --max-wait timeout is non-fatal. - TestKsPollingBackoff — locks the documented 1, 2, 4, 8, 16 (capped) schedule used by `ks add-content --wait`. - TestKsResilience — partial options response, chunks without permalinks, unknown server_status, UTF-8 round-trip, zero results, 200-item single POST, missing-token AuthenticationError, all-empty --items rejected pre-HTTP. lamb-cli/tests/test_commands/test_library.py (+15) - TestLibraryFR10 — `library delete-item` and `library delete` parse the 409 body's `blocking_knowledge_stores` list and render IDs + human names + the `lamb ks remove-content` recovery hint. Tolerates older 409 shapes that omit the structured field. - TestLibraryBackendErrors — 413, 415, 502, 404 (no captions), missing-optional-fields render. - TestLibraryPolling — same backoff schedule as KS, plus an explicit cap test that 8 polls produce a tail of [16, 16, 16]. - TestLibraryResilience — pre-flight file-existence check runs before any HTTP, expired-token AuthenticationError, ConnectError maps to typed NetworkError (no leaked traceback), UTF-8 in name + description, --limit/--offset reach the URL. To make the FR-10 tests pass, three small CLI-source changes: - errors.py: ApiError gains an optional `body` attribute carrying the full parsed response JSON, so command callbacks can render structured error fields without re-parsing. - client.py: _raise_for_status populates ApiError.body when the response body is a dict. - commands/library.py: `delete` and `delete-item` catch ApiError on 409, print the blocking-KS list via _render_fr10_conflict, then re-raise so the typed exit code is preserved. Verified locally: lamb-cli pytest goes from 322 to 362 passing tests.
_wait_for_items and _wait_for_library_items were printing errors for failed jobs but returning exit code 0, making --wait useless for scripts and CI. Track failures in a set and raise typer.Exit(1) when any item fails. Update two test assertions to match the corrected behaviour.
Feature/issue#277/phase4 lamba port
- Consolidated utils/apiClient.js into services/apiClient.js as the single source of truth. - Centralized 403 account disabled detection directly in API interceptors. - Simplified sessionGuard.js to only handle background polling. - Updated consumer imports and deleted redundant utils/apiClient.js.
fix(frontend): resolve duplicate API client race conditions
…_port' into feature/issue#277/phase4_lamba_port
) - Eliminate ~50+ manual localStorage.getItem('userToken') reads across 6+ services - Unify token storage key to 'userToken' (resolves inconsistency: authToken vs userToken) - Convert all services to rely on global apiClient interceptor for Bearer token injection - Remove manual Authorization header construction from service code Services refactored: - knowledgeBaseService.js: 17 functions, all manual token reads removed - libraryService.js: 10 authHeaders() calls removed - assistantService.js: fetch() calls converted to apiFetch/apiJson - templateService.js: getAuthHeaders() removed (9 functions) - rubricService.js: 16+ functions converted to centralized pattern Architecture: - Single source of truth: apiClient.js getStoredToken() → localStorage['userToken'] - Global axios interceptor auto-attaches Bearer token to all requests - Global 401 handler clears session + redirects to login - Dependency Inversion Principle satisfied: services no longer depend on auth details
- Upgraded @lamb/ui sessionManager with a registerOnClearSession() hook system, allowing consumer apps to register store-cleanup callbacks. - clearCurrentSession() now runs all registered callbacks after user.logout(), fixing a bug where the Nav logout button left creator-app stores (assistants, rubrics, templates) populated for the next user. - Simplified creator-app sessionManager: removed duplicate clearCurrentSession and ensureProfileLoaded, now re-exported from @lamb/ui. - Registered resetAllUserScopedStores as a callback in +layout.svelte onMount.
…phase4_lamba_port Feature/issue#277/phase4 lamba port
…-Project/lamb into feature/issue#277/phase4_lamba_port
Collapse the Create Knowledge wizard from 10 steps to 5 by merging the path/details/config screens for each entity. Persist wizard and quick-modal drafts in sessionStorage so accidental dismiss never wipes work, with a Resume/Discard banner on reopen. Extend the library-content step to accept URL and YouTube sources alongside file uploads, queued through the same pendingFiles/pendingUrlSources lifecycle. On the list pages, replace the My/Shared tabs with combinable filter chips (sharing, has-items, created date for libraries; sharing, embedding vendor, chunking strategy, content count, created date for knowledge stores) so users can intersect criteria. Add resizable table columns with a fixed total row width, persisted per-table to localStorage. Wire the page-size selector through to localStorage. Extract EntityListShell + FilterChip + ResizableTable as reusable primitives. Add a contextual + New Knowledge Store button on the KS sub-tab and a Create with Knowledge Store secondary action on the Libraries toolbar. Both pre-route the wizard via a new initialState prop so the entry point matches user intent. i18n keys added in en/es/ca/eu in lockstep.
Update knowledge_store_ui.spec to assert against the new chip-filter toolbar (replacing My/Shared tab assertions) and the 5-step wizard labelling. Add a vitest unit covering the FilterBar sort-order toggle to guard the dispatch wiring. Make the user-disable cleanup step in org_no_admin_and_role_promotion idempotent: when the prior org-delete step cascades the test user away via LambDatabaseManager.delete_organization, the cleanup short-circuits instead of failing on a row that no longer exists.
Document the deferred work needed to move /creator/libraries and /creator/knowledge-stores from client-side to server-side pagination: router signature changes, database-layer limit/offset additions, response shape (libraries|knowledge_stores + total), the get_assistants_by_owner_paginated precedent to mirror, frontend service updates, breaking-change coordination across lamb-cli and the integration tests. Recorded with verified file/line references so a future contributor can execute without re-exploring.
Address seven concrete issues from a hands-on UX review: - Page heading reads "Sources of Knowledge" (matching the navbar entry), removing the standalone "Knowledge" + truncate combo that felt clipped. - Filter chips have breathing room above and the toolbar above them no longer reflows when chips are added or removed (FilterBar locks its row layout, dropdowns get flex-shrink-0). - Create Knowledge wizard anchors its top edge with a fixed pt-8 sm:pt-12 backdrop offset and a min-h on the content area, so toggling Create new vs Use existing no longer makes the panel jump up and down. - Step 4 (KS content picker) auto-skips when both library and KS paths are 'new' and the user already queued content in Step 2 — Step8_ReviewCreate already auto-ingests every newly-uploaded item, so the explicit picker would be a tautology. The skip-rule still shows Step 4 for the existing-library / existing-KS combinations where it carries information. - KS Step 3 advanced section now gates the embedding-vendor dropdown by what the organization has actually configured: the backend /creator/knowledge-stores/options response tags each vendor with api_key_configured (true when OrganizationConfigResolver returns a non-empty key, or for the 'local' vendor which needs no key), the frontend disables un-configured options and falls back to the first enabled vendor instead of a hardcoded options[0]. An amber notice appears when no vendor is configured for the org. - Step 5 (Review & Create) is restructured around section cards (Library / Knowledge Store / Ingestion) with aligned key-value dl grids, an aria-live progress panel where each step renders a spinner / checkmark / X icon, an inline-SVG error banner with a Retry action, and a tighter primary CTA. DB-write logic is byte-for-byte unchanged. i18n keys updated in en/es/ca/eu in lockstep.
Changed from old monorepo path to @lamb/ui import to fix build error
…rvice
The 8 cost management functions were incorrectly passing token as second
argument to jsonRequest(), which has signature jsonRequest(path, init = {}).
This caused the token string to be interpreted as the init object.
Fixed functions:
- fetchCostOverview()
- fetchCostSummaryByOrg(organizationId)
- searchOrganizations(name)
- fetchAssistantUsageByModel(assistantId)
- fetchModelPricing()
- createModelPricing(data)
- updateModelPricing(id, data)
- deleteModelPricing(id)
Updated all call sites in components to remove token parameter:
- OrganizationFilterModal.svelte
- CostManagementPanel.svelte
- ModelPricingModal.svelte (4 calls)
- AssistantUsageBreakdown.svelte
Updated ModelPricingModal.svelte.test.js to not expect token in mock calls.
1. Fix test mocks in cost management components: - Replace $lib/stores/userStore and $lib/i18n mocks with @lamb/ui - Files: ModelPricingModal, AssistantUsageBreakdown, OrganizationFilterModal, CostManagementPanel 2. Fix SSR hydration mismatch in Checkbox.svelte: - Replace Math.random() ID generation with deterministic counter - Use <script context="module"> for shared counter across instances 3. Cleanup empty directory tree: - Remove frontend/svelte-app (empty after monorepo migration)
Move filteredCostData and costTotals declarations before activeSummary to fix reference-before-declaration error that prevented /admin from loading. The activeSummary $derived expression depends on costTotals, so it must be declared after costTotals is defined.
The NotificationModal component was referencing an undefined 'notification' variable, causing a ReferenceError that prevented the admin page from rendering. Added the missing $state declaration with default values.
Reject incompatible prompt_processor/rag_processor/document_rag metadata on create and update, validate Reference Document library refs in the form, return explicit errors from library_file_rag with HTTP 502 on load failure, and align assistant import validation with backend rules.
Use resolveCreatePromptProcessor() on create reset and non-advanced submit so org defaults and localStorage cannot pin simple_augment. Show both PPS in Advanced create; remove LAMB_LEGACY_PPS_DEFAULT and served-defaults override.
…raries content imports (#277)
…r_merge_kvcache_v2
…rge_kvcache_v2_with_cost_management Conflicts resolved: - backend/lamb/completions/main.py: keep cost-mgmt timing extraction and _require_document_context validation - backend/lamb/completions/rag/library_file_rag.py: drop unused empty_result local, keep _error_result for informative error returns
…st management branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request:
lamba_refactor_merge_kvcache_v2_with_cost_managementBranch:
lamba_refactor_merge_kvcache_v2_with_cost_management→devCommits: ~408 commits (vs
dev)Stats: 630 files changed, +86,501 / -38,774 lines
Summary
Integration branch consolidating LAMB's main development lines: the completions pipeline with a two-plugin architecture (KV-cache friendly), cache-aware cost management (per-request frozen costs, three-bucket token accounting, explicit-cache support, model pricing CRUD — Issue #411), Knowledge Stores (new KB Server), Library Manager, LTI modules (Phase 4), and the frontend migration to a pnpm monorepo. Includes post-merge fixes, design system restoration, and pre-merge validation hardening.
Branch lineage and merge history
This branch is an integration line, not a single feature. It merges several upstream development tracks into one testable whole before landing on
dev.feature/issue#277/phase4_lamba_portprojects/refactor/kbserver-lamb-integrationlibrary_manager_and_kv_cache_final_refactorlibrary_manager_and_kv_cache_final_refactor_with_cost_managementorigin/dev1. Completions Pipeline — Two-Plugin Architecture
Goal: Separate the legacy pipeline (
simple_augment+ KB-based RAGs) from the new KV-cache friendly pipeline (kvcache_augment+ Knowledge Stores / Library Manager), without modifying legacy code.1.1 New PPS:
kvcache_augment.pydocument_contextat the beginning of the system prompt (KV-cache friendly: stable prefix = cache hits ~97-98%).COMPATIBLE_RAGforlibrary_file_rag,knowledge_store_rag,query_rewriting_ks_rag,rubric_rag,no_rag.DEFAULT_RAG_PROMPT_TEMPLATEwhen RAG context exists but no custom template is set.backend/lamb/completions/main.pyand at save time viametadata_validators.py+assistant_router.py.kvcache_augmentis now the default for new assistants. Create mode usesresolveCreatePromptProcessor()so the form always selectskvcache_augmentwhen available. Legacysimple_augmentis locked read-only in edit mode.1.2 New RAG processors
library_file_rag.py: fetches documents from Library Manager via HTTP (/libraries/{id}/items/{id}/content). Replaces the embedded logic insingle_file_rag.py(which remains untouched as legacy).knowledge_store_rag.py: queries the new KB Server v2 (port 9092) with per-request embedding credentials.query_rewriting_ks_rag.py: rewrites the user query using a small-fast-model over the full conversation history, then queries Knowledge Stores._ks_query_helpers.py: shared query and source-extraction logic for KS RAG processors._query_rewriting_helper.py: small-fast-model query rewriting helper.1.3 Frontend — PPS/RAG compatibility and AssistantForm refactor
ragProcessorHelpers.js: mirrors backendPPS_COMPATIBLE_RAGwithgetCompatibleRagForPps(),resolveCreatePromptProcessor(),isLegacyPps(), etc.ConfigurationPanel.svelte: filters RAG dropdown by PPS compatibility; shows both PPS in Advanced create mode.simple_augmentshow an amber notice banner, disabled KB/KS selectors, disabled Top-K input, and disabled document toggle.AssistantForm.svelterefactored from ~1,747 LOC to ~535 LOC orchestrator + logic modules + UI subcomponents.LibraryItemSelector.svelte: library + item selector for Document RAG.Key backend files
backend/lamb/completions/pps/kvcache_augment.pybackend/lamb/completions/rag/library_file_rag.pybackend/lamb/completions/rag/knowledge_store_rag.pybackend/lamb/completions/rag/query_rewriting_ks_rag.pybackend/lamb/completions/rag/_ks_query_helpers.pybackend/lamb/completions/rag/_query_rewriting_helper.pybackend/lamb/completions/org_config_resolver.pybackend/lamb/completions/plugin_config.pybackend/creator_interface/metadata_validators.pyKey frontend files
frontend/.../assistants/AssistantForm.sveltefrontend/.../assistants/components/ConfigurationPanel.sveltefrontend/.../assistants/components/RagOptionsPanel.sveltefrontend/.../utils/ragProcessorHelpers.jsfrontend/.../components/LibraryItemSelector.svelte2. Knowledge Stores — New KB Server (port 9092)
Goal: Vectorization microservice with plugin architecture, replacing the legacy KB Server (port 9090) for new assistants.
2.1 Backend KB Server (
lamb-kb-server/)data/storage/{org_id}/{collection_id}/.2.2 LAMB Integration
/creator/knowledge-stores/*with ACL + proxy to KB Server.knowledge_stores+kb_content_linksin LAMB DB.backend/creator_interface/knowledge_store_client.py.backend/creator_interface/knowledge_store_router.py.knowledge_store_rag.py+query_rewriting_ks_rag.pyin the completions pipeline.lamb ks .../lamb knowledge-store ...inlamb-cli/.2.3 Frontend
KnowledgeStoresList.svelte,KnowledgeStoreDetail.svelte,AddContentToKSModal.svelte,IngestionProgressModal.svelte.CreateKnowledgeWizard.sveltewith stepper for Library + Knowledge Store.ksCache.js(stale-while-revalidate)./knowledge-storesredirects to/libraries?section=knowledge-stores.Key files
lamb-kb-server/backend/main.pyand plugin/router/service modulesbackend/creator_interface/knowledge_store_client.pybackend/creator_interface/knowledge_store_router.pyfrontend/.../knowledgeStores/*.sveltefrontend/.../knowledge/CreateKnowledgeWizard.sveltelamb-cli/src/lamb_cli/commands/knowledge_store.py3. Library Manager — Document Repository
Goal: Independent microservice (port 9091) for importing and structuring documents in markdown format with permalinks.
3.1 Improvements in this branch
FileTreeModal.svelte,FileTreeNode.svelte,MoveToFolderPicker.svelte,TreePreviewPane.svelte.ImagesRenderer,PagesRenderer,TextRenderer.3.2 LAMB Integration
/creator/libraries/*endpoints with ACL + proxy.library_manager_client.py.library_router.py.LibrariesList.svelte,LibraryDetail.svelte,ItemContentModal.svelte,ItemContentTabs.svelte,PluginPickerModal.svelte.librariesCache.js(stale-while-revalidate).Key files
library-manager/backend/routers/folders.pylibrary-manager/backend/routers/capabilities.pylibrary-manager/backend/services/folder_service.pylibrary-manager/backend/plugins/content_handlers/backend/creator_interface/library_manager_client.pybackend/creator_interface/library_router.pyfrontend/.../libraries/*.sveltefrontend/.../stores/librariesCache.js4. Cost Management & Cache Economics (#411)
Goal: Track the real USD cost of every completion with cache-aware accounting, make those costs immutable once logged, and surface them to system admins. This is the economic counterpart to the KV-cache pipeline: it measures the savings that prompt caching produces.
4.1 Three-bucket token accounting
Every request's
prompt_tokensis split into three buckets that satisfyprompt_tokens = non_cached + cache_read + cache_write:non_cachedmax(0, prompt − cache_read − cache_write)input_per_1mcache_readprompt_tokens_details.cached_tokenscache_read_per_1m(fallback: input)cache_writecache_creation_input_tokenscache_write_per_1m(fallback: input), 0 unlessrequires_explicit_cacheextract_token_buckets()(token_repartition.py) normalizes flat vs nested provider payloads and clamps so the identity always holds.4.2 Cost formula and immutability
compute_cost_usd(pricing, buckets)(cost_formula.py): three-bucket formula.cache_write(provider caches for free).log_token_usage()computescost_usdonce with the then-currentmodel_pricingand stores it onusage_logs.cost_usd. Reads never recompute.get_assistant_cost_usd()returns the storedassistant_usage_totals.cost_usd_total.cost_usdrows.4.3 Explicit KV cache (Anthropic / Alibaba)
explicit_cache.py→apply_cache_markers(messages): addscache_control: {type: "ephemeral"}on the second-to-last message for APIs that require explicit markers.model_pricing.requires_explicit_cache = 1.LLM_ALIBABA_CACHE_EXPERIMENTenv experiment.4.4 Model pricing (DB + CRUD)
model_pricingtable columns:provider,model_name,input_per_1m,output_per_1m,cache_read_per_1m,cache_write_per_1m,requires_explicit_cache,notes,updated_at.(provider="openai", model_name="qwen3.6-plus")because it uses the OpenAI-compatible connector./creator/admin/model-pricing(system-admin only).4.5 Admin API endpoints
/admin/cost-overview/admin/cost-overview/summary?organization_id=/admin/assistant/{id}/usage-by-model/admin/organizations/search?name=/admin/model-pricing[/{id}]/admin/assistant/{id}/quota4.6 Frontend (admin)
CostManagementPanel.svelte: summary cards with cache read/write sub-line, search, org filter, assistant table, expandable per-model breakdown, quota modal, pricing modal trigger.ModelPricingModal.svelte: pricing CRUD with inline row edit, cache rates, explicit cache flag.AssistantUsageBreakdown.svelte: per-model table with Non-cached / Cache read / Cache write columns.OrganizationFilterModal.svelte: org search + scoped summary.admin.costManagement.*in all 4 locales.Key files
backend/lamb/completions/token_repartition.pybackend/lamb/completions/cost_formula.pybackend/lamb/completions/explicit_cache.pybackend/lamb/database_manager.py(migrations 18/19, cost logging, pricing CRUD)backend/lamb/completions/main.pybackend/lamb/completions/connectors/openai.pybackend/creator_interface/organization_router.pybackend/tests/test_cost_management.pyfrontend/.../components/admin/CostManagementPanel.sveltefrontend/.../components/admin/ModelPricingModal.sveltefrontend/.../utils/costManagementHelpers.js5. Phase 4 — LTI Activity Module System + Frontend Monorepo
Source branch:
feature/issue#277/phase4_lamba_port(Issue #277).5.1 LTI Activity Module System
modules/chat/): LTI launch, JWT session management, instructor dashboard, student flow.modules/file_evaluation/): AI-powered file evaluation, grading, group work, Moodle grade passback.evaluation_service.py: AI evaluation via assistant.grade_service.py: grade management.lti_passback.py: Moodle grade sync via LTI outcomes.document_extractor.py: document content extraction.storage_service.py: submission storage.modules/base.py):LTIContextclass,on_student_launch()/on_instructor_launch()hooks.5.2 Frontend Monorepo Migration
@lamb/ui(shared),creator-app(main SPA),module-chat(LTI chat),module-file-eval(LTI file evaluation).@lamb/ui: Nav, Footer, LanguageSelector, ConfirmationModal, userStore, configStore, authService, configService, i18n (4 locales), sanitize utils.creator-app: full creator interface UI with design system.module-chat: independent SPA for/m/chat/(setup + dashboard).module-file-eval: independent SPA for/m/file-eval/(upload + grading).svelte-app/removal: complete migration to monorepo.5.3 Design System
frontend/packages/creator-app/src/lib/components/ui/(Button, IconButton, Modal, Badge, Card, Toast, Tabs, FormField, Dropdown, OverflowMenu, Dropzone, Stepper, Banner, Collapsible, Checkbox, EmptyState, Skeleton*).app.css: semantic colors (brand, success, warning, danger, info), shadows, radii, motion, typography.icons.js(re-export oflucide-svelte).flowbite-svelteremoved.statusBadge.jswith locked mapping (ready=success, processing=info, failed=danger, etc.).Key files
backend/lamb/modules/*backend/lamb/lti_router.pybackend/lamb/lti_activity_manager.pybackend/lamb/auth_context.pyfrontend/packages/ui/frontend/packages/creator-app/src/lib/components/ui/frontend/packages/module-chat/frontend/packages/module-file-eval/6. Merge from
origin/dev(#425)This PR integrates the latest
devcommits, most importantly the database migration redesign.6.1 Database migration redesign
backend/lamb/migrations.pywithMigrationRunnerandschema_versiontable.MigrationRunner.apply_all()runs migrations idempotently; each migration runs exactly once.run_migrations()frombackend/lamb/database_manager.py.backend/main.pyandLambDatabaseManager.__init__callMigrationRunner._migration_27,_migration_28,_migration_29:_migration_26: LTI columns (activity_type,setup_config,lis_outcome_service_url)._migration_27: Knowledge Stores tables (knowledge_stores,kb_content_links)._migration_28: cache-aware token costs and OpenAI seed pricing._migration_29: cache write/explicit cache,cost_usd, Qwen seed, backfill + rebuild totals.6.2 Other
devchangescitation_urlpriority in RAG processors (context_aware_rag.py,hierarchical_rag.py,simple_rag.py).7. Tests
7.1 Backend tests
test_cost_management.py: full Predictable OWI mirror-user passwords (derived from assistant/activity id) #411 backend suite.test_creator_knowledge_stores_integration.py: Knowledge Stores CRUD via Creator Interface.test_creator_libraries_integration.py,test_creator_libraries_content.py,test_creator_library_folders.py: Library CRUD, content, folders.test_fr10_interlock.py: FR-10 delete protection.test_knowledge_store_options.py: KS setup options + locked fields.test_ks_query_helpers.py,test_query_rewriting_helper.py,test_query_rewriting_ks_rag.py: KS RAG helpers.test_document_cache.py: in-memory document RAG cache.test_simple_augment.py: legacy PPS without document_context.7.2 Frontend tests
ConfigurationPanel.svelte(both PPS in create dropdown, legacy edit lock).RagOptionsPanel.svelte(legacy banner, disabled selectors).CostManagementPanel,AssistantUsageBreakdown,ModelPricingModal,OrganizationFilterModal.8. Bugs Fixed (Post-Merge and Pre-Merge)
8.1 Post-merge integration fixes
app.csswas a 10-line stub after merge).0.1→0.6.AssistantForm+ imports after merge conflicts.creator_flow,account_disable_security).document_contextrouting: only passed tokvcache_augment._optimizations_applied/_migrations_appliedflags to avoid DB migrations/PRAGMAs on every request.config.jssamples formodule-chatandmodule-file-eval.8.2 Pre-merge validation hardening
library_idanditem_id.library_file_ragreturns explicit errors instead of silent empty context; document load failure returns HTTP 502.9. Deployment / Operational Notes
backend/lamb/migrations.pyto run on startup. Verified:schema_versionreaches 29 on existing databases.lamb-kb-servermicroservice (port 9092).library-managermicroservice (port 9091).LLM_ALIBABA_CACHE_EXPERIMENTenv var; explicit cache is now DB-driven per model.pnpm --filter creator-app build.10. How to verify
Backend startup
Expected: no migration errors, no schema errors, modules discovered successfully.
Database verification
sqlite3 lamb_v4.db "SELECT * FROM LAMB_schema_version ORDER BY version DESC LIMIT 5;"Expected:
version = 29is present.Expected: KS tables, LTI columns, and cost columns exist.
Backend tests
cd backend python -m pytest tests/test_database_manager_init.py -v python -m pytest tests/test_creator_knowledge_stores_integration.py -v python -m pytest tests/test_cost_management.py -vFrontend checks
cd frontend pnpm --filter creator-app check pnpm --filter creator-app lintEnd-to-end smoke test (manual)
kvcache_augment+knowledge_store_rag.11. Review checklist
backend/lamb/database_manager.pystill contains all Knowledge Stores / Cost Management methods (no accidental--theirs).backend/lamb/migrations.pyhasLATEST_VERSION = 29and migrations 26-29 are idempotent.backend/main.pycallsMigrationRunnerbeforediscover_modules().kvcache_augmentonly receivesdocument_context;simple_augmentdoes not.creator_flow,kb_detail_modals).