Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
215 lines (145 loc) · 5.52 KB

ARCHITECTURE.md

File metadata and controls

215 lines (145 loc) · 5.52 KB

NecroScope v1 Architecture

This document defines the architecture of NecroScope v1. It specifies modules, responsibilities, interfaces, and how data flows through the system. See README.md "V1 limitations" for known gaps in the current implementation.

Architecture must stay aligned with:

  • DESIGN.md for high level goals and scope
  • INGESTION.md for how logs are parsed and normalized
  • OUTPUT.md for how results are represented
  • TESTING.md for how behavior is validated

This file reflects the current v1 implementation. Sections marked as future work are not implemented yet.

1. Overview

NecroScope is conceptually split into three core layers:

  1. Ingestion - turn raw kernel logs into structured internal representations.
  2. Analysis - inspect the structured data, detect relevant events, and build higher level findings.
  3. Output - present results in deterministic formats suitable for humans and tools.

Each layer is implemented as one or more Python modules under a single top level package.

2. Module Layout

The v1 module layout follows this structure:

src/
  necro_scope/
    __init__.py

    ingestion/
      __init__.py
      parser.py
      detector.py
      loader.py
      calibration.py

    analysis/
      __init__.py
      engine.py
      rules.py      # detection logic, classifiers, heuristics
      severity.py
      stacktrace.py

    output/
      __init__.py
      formatter_json.py
      formatter_text.py
      schema.py

    models/
      __init__.py
      analysis_result.py
      error_block.py
      events.py
      log_metadata.py

    util/
      __init__.py
      anonymize.py
      text.py
      timeparse.py

    providers/
      __init__.py
      base.py
      factory.py
      heuristic.py

    llm/
      __init__.py
      client.py
      prompts.py

    web/
      __init__.py
      app.py
      routes.py

tests/
  analysis/
  ingestion/
  output/
  data/

Sections below define responsibilities and contracts for each area.

2.1 Ingestion

  • Read raw logs from a source (file, stdin, etc).
  • Tokenize and parse into a structured representation.
  • Extract metadata such as kernel version and command lines.
  • Boot boundaries are not detected in v1 and all events are assigned to boot 0.
  • Hand back a well defined in memory structure for the analysis layer.

Details are specified in INGESTION.md.

2.2 Analysis

  • Consume the ingestion output.
  • Detect errors, warnings, panics, oopses, and other notable events.
  • Correlate events with kernel versions, boots, and relevant context.
  • Produce a neutral internal representation of findings for the output layer.

Details are specified in DESIGN.md.

2.3 Output

  • Take analysis results and metadata.
  • Produce deterministic, stable output formats (for example JSON and text summaries).
  • Ensure ordering, naming, and structure match OUTPUT.md.

3. Data Flow

At a high level, the data flow is:

  1. Input

    • Raw kernel log text (for example captured dmesg output).

  2. Ingestion

    • Parsing and normalization according to INGESTION.md.
    • Construction of one or more internal objects representing boots, messages, and metadata.

  3. Analysis

    • Detection of interesting events.
    • Grouping, classification, and enrichment of events.

  4. Output

    • Conversion of analysis results into the output structures defined in OUTPUT.md.
    • Rendering to the final format that the user or caller sees.

Future diagrams can be added here once the structures are finalized.

4. Interfaces and Contracts

Key interfaces between layers for v1:

  • Ingestion: ingestion.ingest(lines) returns (LogMetadata, List[LogEvent], List[ErrorBlock]).
  • Analysis: analysis.engine.analyze(metadata, events, error_blocks) returns an AnalysisResult.
  • Output: output.formatter_json.format_json(result) and output.formatter_text.format_text(result) accept AnalysisResult.

The dataclasses in src/necro_scope/models/ define the concrete schemas.

5. Error Handling Strategy

This section will define how each layer deals with bad or partial input, and how errors are surfaced.

Planned topics:

  • What happens when ingestion encounters malformed lines.
  • How multi boot logs with partial data are handled.
  • How internal exceptions are translated into user visible signals.
  • When to fail hard vs emit partial results with warnings.

To be completed.

6. Cross Cutting Concerns

Cross cutting concerns apply across ingestion, analysis, and output.

Planned topics:

  • Logging and debug output for development.
  • Configuration handling (if any) for v1.
  • Privacy considerations when handling sensitive logs, consistent with DESIGN.md.
  • Performance considerations for large logs.

To be completed.

7. Extensibility and Future Enhancements

NecroScope v1 should be architected to allow:

  • Additional ingestion backends for other log formats.
  • Additional analysis rules without breaking existing behavior.
  • Additional output formats while keeping core structures stable.
  • Optional Rust components in specific hotspots, if justified later.

This section will be expanded once v1 is more mature and concrete extension points are identified.

8. Related Specifications

  • DESIGN.md - overall goals, scope, and requirements.
  • INGESTION.md - ingestion rules and schemas.
  • OUTPUT.md - output schemas and determinism rules.
  • TESTING.md - fixtures, test types, and expectations.
  • AGENTS.md - multi agent workflow and constraints for automated implementers and reviewers.

End of ARCHITECTURE.md