Refactor extraction to frame-first knowledge base architecture with per-batch evidence-backed frames

[Copilot]

This change shifts extraction from directly writing graph structures to building a canonical knowledge base frame per research package (batch), using preprocessed LLM-readable inputs as the primary source. Each frame now stores detailed scientific content and extraction metadata (checks/timestamps/source links), enabling downstream database and KG generation from a stable intermediate representation.

• Schema: introduce per-batch knowledge frames

  • Added knowledge_base_frames table/model (KnowledgeBaseFrame) with:
    • one-to-one mapping to ingestion_batches (batch_id unique)
    • structured payload (frame_data) for concepts, experimental_data, statements, related_data
    • operational metadata (frame_metadata, check_count, first_extracted_at, last_extracted_at, status)
  • Added Alembic migration 004_add_knowledge_base_frames.py.

• Agent tools: frame construction + evidence normalization

  • Added frame-first tools:
    • initialize_knowledge_frame(batch_id, title)
    • add_knowledge_frame_items(batch_id, section, items, evidence_level, source_asset_id)
    • mark_knowledge_frame_checked(batch_id, summary)
  • Added canonical evidence level normalization:
    • Level 1: Causal experimental evidence
    • Level 2: Direct experimental observation
    • Level 3: Correlative evidence
    • Level 4: Predicted / inferred
  • Kept compatibility by making mark_batch_extracted(...) delegate to frame completion.

• Extraction runtime flow: frame-first orchestration

  • Updated extraction prompt to explicitly build a per-batch frame first, then optionally emit graph artifacts later.
  • Runner now ensures/updates frame lifecycle and provenance metadata (batch source links/assets) during extraction.
  • Returned extraction summary now includes frame-level section counts/status when available.

• Operational consistency in CLI flows

  • clear-knowledge and purge now include knowledge_base_frames cleanup/reset to keep state aligned with node/edge deletion semantics.
  • README updated to document frame-first extraction and new table role.

• Example: writing evidence-backed frame items

  add_knowledge_frame_items(
      batch_id="b2f5...8c1a",
      section="experimental_data",
      evidence_level="Level 2",
      source_asset_id="7b13...2ef0",
      items=[
          {
              "property": "carrier mobility",
              "value": 12.4,
              "unit": "cm^2/V·s",
              "conditions": {"temperature": "300 K"}
          }
      ],
  )

Original prompt

I want to major change the database logic. i want to use the preprocessed llm readable data as input, and let llm to create a knowledge base frame (a note for each research), i.e. one research package (main paper + any additional things if available) corresponds to one frame of data inside the knowledge base, just like the current knowledge graph extraction agent do. this frame shall contains some metadata, such as how many times checked, time extracted, link to the original data, etc. And then, based on the knowledge base, later extract the formatted database and knowledge graph. In this case, the knowledge base frame should contains detailed info for concepts, experimental data (and other related data), statements, etc. As well as an evidence level like: "Level 1: Causal experimental evidence; Level 2: Direct experimental observation; Level 3: Correlative evidence; Level 4: Predicted / inferred" In this way, please help me refactor the code and have a more reasonable data flow logic

Created from VS Code.