TODO.md

JSL TODO List

Critical Issues

• CRITICAL: Fix deeply nested environment serialization/deserialization ✅ FIXED in v0.4.0
  • Fixed cycle hash collision bug in serialization.py
  • Parent chain structure now preserved through serialization/deserialization

Immediate Fixes

• All tests passing (643 tests) ✅
• Add regex matching support (str-matches function) ✅ Already implemented
• Add fuzzy string matching support

JSON Manipulation Features (High Priority)

Based on dotsuite patterns, JSL needs better JSON manipulation:

Path Navigation

• Implement get-path for deep path access ✅ Already implemented
• Implement set-path for deep path modification ✅ Already implemented
• Implement has-path to check if deep path exists ✅ Already implemented
• Add safe navigation operators (get-safe, get-default) ✅ Already implemented
• Support array indexing in paths (e.g., items[0].price) ✅ Already implemented
• Support wildcard paths (e.g., users.*.email) ✅ Fixed in v0.4.0

Query-Based Operations

• Implement where operator for declarative filtering ✅ Special form
• Add query expression evaluation (=, !=, >, <, contains, matches) ✅ Already implemented
• Support compound queries with and/or/not ✅ Already implemented
• Add find-first to get first matching element ✅ Added in v0.4.0
• Add count-where to count matching elements ✅ Added in v0.4.0

Object Transformation

• Implement transform operator for pipeline transformations ✅ Special form
• Add assign operation (add/update fields) ✅ Already implemented
• Add pick operation (select specific fields) ✅ Already implemented
• Add omit operation (remove specific fields) ✅ Already implemented
• Add rename operation (rename fields) ✅ Already implemented
• Add default operation (set defaults for missing fields) ✅ Already implemented

Collection Enhancements

• Implement group-by for grouping collections ✅ Already implemented
• Implement index-by for converting to keyed objects ✅ Already implemented
• Implement pluck for extracting single field from collection ✅ Already implemented
• Implement unique for getting unique values ✅ Already implemented
• Implement flatten for flattening nested arrays ✅ Added in v0.4.0
• Implement zip for combining arrays ✅ Already implemented

Architecture Improvements

Evaluator Unification

• Consider whether stack evaluator should use Closure and Env classes
• Unify closure representation between evaluators
• Create shared test utilities for both evaluators

Performance

• Optimize object evaluation in stack evaluator
• Add benchmarking suite
• Profile and optimize hot paths
• Consider JIT compilation for frequently used functions

Resource Management

• Implement memory limits for stack evaluator
• Add timeout support for both evaluators
• Implement better resource exhaustion messages
• Add resource usage statistics

Language Features

Control Flow

• Add switch/case expression
• Add loop construct with break/continue
• Add for loop over collections
• Add pattern matching

Error Handling

• Improve error messages with line/column information
• Add stack traces for debugging
• Implement error recovery mechanisms
• Add validation functions

Type System (Optional)

• Add optional type annotations
• Implement type checking
• Add type inference
• Create type declaration syntax

Standard Library Expansion

Math Functions

• Add trigonometric functions (sin, cos, tan, etc.)
• Add statistical functions (mean, median, stddev)
• Add random number generation
• Add matrix operations

Date/Time

• Add date/time parsing
• Add date arithmetic
• Add formatting functions
• Add timezone support

Crypto

• Add hashing functions (SHA, MD5, etc.)
• Add encoding/decoding (base64, hex)
• Add UUID generation
• Consider encryption support

Tooling

Development Tools

• Create Language Server Protocol (LSP) implementation
• Add syntax highlighting for editors
• Create debugger interface
• Add profiler tools

Documentation

• Complete API documentation
• Add more tutorials
• Create cookbook with common patterns
• Add performance guide

Testing

• Current test coverage: 71% (643 tests passing) ✅
• Increase test coverage to 80%+
• Add property-based testing
• Create integration test suite
• Add performance regression tests

Network & Distribution

Serialization

• Optimize serialization format
• Add compression support
• Implement incremental serialization
• Add versioning support

Distribution

• Implement distributed evaluation
• Add remote procedure calls
• Create clustering support
• Implement work stealing

Security

Sandboxing

• Implement capability-based security
• Add permission system for host functions
• Create security audit tools
• Add code signing support

Validation

• Add input validation framework
• Implement schema validation
• Add sanitization functions
• Create security linting

Ecosystem

Package Management

• Design module system
• Create package repository
• Implement dependency resolution
• Add versioning support

Interoperability

• Add JavaScript interop
• Add Python interop beyond host functions
• Create REST API framework
• Add GraphQL support

Long-term Vision

Compilation

• Create WASM target
• Implement native compilation
• Add JVM bytecode generation
• Create LLVM backend

Optimization

• Implement tail call optimization
• Add constant folding
• Implement dead code elimination
• Add inlining optimization

Current Priorities

1. Fix remaining test ✅ All 643 tests passing
2. JSON manipulation features ✅ All core features implemented
3. Pedagogical documentation ✅ Added docs/pedagogy/ with reading guide and traces
4. Blog posts - Extract narratives from paper/implementation
5. Performance - Ensure scalability
6. Tooling - Better developer experience (LSP, syntax highlighting)

Blog Post Ideas

Potential blog posts extractable from JSL's design and implementation:

Technical Deep-Dives

• "Building a Lisp in 500 Lines of Python" - The recursive evaluator as a teaching tool
• "Content-Addressable Serialization: Handling Cycles Without Tears" - The serialization approach
• "From Recursion to Iteration: Defunctionalizing an Interpreter" - Recursive → stack evaluator transformation
• "Making Computation Resumable" - Stack-based evaluation and network mobility

Design Philosophy

• "JSON as Code: The Case for Homoiconic JSON" - Why JSON for code representation
• "Capability-Based Security in a Scripting Language" - JHIP and effect reification
• "Two Interpreters, One Language" - Why dual evaluators are pedagogically valuable

Practical Guides

• "Implementing Lexical Scoping: Environments as Linked Lists" - The Env class explained
• "How Closures Really Work" - From definition to serialization
• "Query Languages for JSON: The where and transform Story" - Declarative data manipulation

Academic

• "Network-Native Language Design" - Expand the SPLASH paper themes
• "Proving Interpreter Equivalence Through Testing" - The unified test suite approach

Notes

• The recursive evaluator is the reference implementation
• Stack evaluator needs to maintain behavioral parity
• All features must remain JSON-serializable
• Network transmission is a first-class concern
• Security through capability restriction is fundamental