Make array literal registry lazy on NaturalArray and expand PTAC native execution coverage

ISSUE_REPORT_PTAC_NATURALARRAY.md

@@ -0,0 +1,70 @@
+# Issue Report: NaturalArray Lazy Registry + PTAC Coverage
+
+## Context
+Task requested:
+1. Move array literal registry usage from eager materialization to NaturalArray-backed behavior and validate size-independent complexity.
+2. Clean unused imports/methods.
+3. Improve compiler/PTAC execution completeness to reduce interpreter fallback.
+4. Submit an issue report (file in repo root when direct issue submission is unavailable).
+
+## What was changed
+
+### 1) Literal registry array operations now use NaturalArray lazily
+- File: `src/main/java/cod/interpreter/registry/LiteralRegistry.java`
+- `map` and `filter` on `NaturalArray` no longer force `toList()` eager materialization.
+- Added lazy `AbstractList` views for NaturalArray:
+  - `LazyNaturalArrayMapView`
+  - `LazyNaturalArrayFilterView`
+- `reduce` on `NaturalArray` now streams by index (`size()` + `get(i)`) instead of first materializing a full list.
+- Kept list behavior unchanged for non-NaturalArray inputs.
+
+### 2) PTAC executor native coverage increased
+- File: `src/main/java/cod/ptac/Executor.java`
+- Added instruction pointer execution model with label indexing:
+  - Native `NOP`, `BRANCH`, `BRANCH_IF`
+- Added native lazy array op handling:
+  - `LAZY_GET`, `LAZY_SET` (list-backed), `LAZY_SIZE`, `LAZY_COMMIT`
+- Added native pattern execution:
+  - `MAP`, `FILTER`, `REDUCE`, `FILTER_MAP`
+- Kept sensitive recursive opcodes (`SELF`, `TAIL_CALL`) on fallback path to preserve parity correctness.
+- Added safe fallback for non-numeric range bounds in native range execution.
+
+### 3) Validation runner added for complexity comparison
+- New file: `src/main/java/cod/runner/ArrayLiteralRegistryComplexityRunner.java`
+- Benchmarks prior eager strategy vs current NaturalArray path across sizes:
+  - `1000`, `10000`, `100000`
+- Validates that map/filter setup growth is smaller than eager growth.
+
+### 4) Cleanup
+- Removed unused import in new runner.
+- Removed unused parameter from executor instruction dispatch.
+
+## Validation performed
+
+Commands run:
+- `./gradlew --no-daemon sourceJar`
+- `javac -d /tmp/coderive-compile $(find src/main/java -name '*.java')`
+- `java -cp /tmp/coderive-compile cod.runner.ArrayLiteralRegistryComplexityRunner`
+- `java -cp /tmp/coderive-compile cod.runner.CodPTACParityRunner /home/runner/work/Coderive/Coderive/src/main/cod/demo/src/main/test`
+
+Results:
+- Build + Java compile: **pass**
+- Complexity runner: **pass**
+  - Sample output indicated map/filter setup remained effectively size-independent versus eager growth.
+- PTAC parity suite: **47 passed, 0 failed**
+
+## Findings / remaining issues
+
+1. `reduce` semantic result is still computed eagerly (returns final scalar), so total time cannot be strictly O(1) for general reductions.
+   - Improvement made: removed eager full-list materialization for NaturalArray reduce (space reduced to streaming behavior).
+2. PTAC still falls back for advanced opcodes not yet natively implemented:
+   - `SCAN`, `ZIP`, `WHERE`, `FILTER_MAP_REDUCE`, `LAZY_SLICE`,
+   - slot/closure/tail-recursion/formula/storage opcodes.
+3. Non-numeric range bounds are intentionally routed to fallback to avoid incorrect native execution.
+
+## Suggested follow-up issues
+
+1. Add native executor support for `SCAN`, `ZIP`, `WHERE`, and `FILTER_MAP_REDUCE`.
+2. Implement native slot and closure stack model (`SLOT_*`, `CLOSURE`, `ANCESTOR`) for larger fallback reduction.
+3. Design typed lazy reduction objects (or algebraic reductions) if strict O(1) reduce setup + deferred evaluation is required by language semantics.
+

src/main/java/cod/interpreter/registry/LiteralRegistry.java

@@ -591,18 +591,22 @@ private Object handleStringIsUpper(Object literal, List<Object> arguments) {
     }
 
     @SuppressWarnings("unchecked")
-    private List<Object> asList(Object literal) {
+    private List<Object> asConcreteList(Object literal) {
+        if (literal instanceof List) {
+            return (List<Object>) literal;
+        }
+        throw new ProgramError("Expected list literal target");
+    }
+
+    private NaturalArray asNaturalArray(Object literal) {
         if (literal instanceof NaturalArray) {
             NaturalArray arr = (NaturalArray) literal;
             if (arr.hasPendingUpdates()) {
                 arr.commitUpdates();
             }
-            return arr.toList();
+            return arr;
         }
-        if (literal instanceof List) {
-            return (List<Object>) literal;
-        }
-        throw new ProgramError("Expected array literal target");
+        return null;
     }
 
     private Object invokeArrayCallback(Object callbackObj, String methodName, ExecutionContext ctx, Object... args) {
@@ -619,29 +623,43 @@ private Object handleArrayMap(Object literal, List<Object> arguments, ExecutionC
         if (arguments == null || arguments.isEmpty()) {
             throw new ProgramError("map expects a callback or (operator, operand)");
         }
-        List<Object> source = asList(literal);
+        final NaturalArray naturalArray = asNaturalArray(literal);
+        List<Object> source = naturalArray == null ? asConcreteList(literal) : null;
 
         if (looksLikeOperatorMap(arguments)) {
-            String op = String.valueOf(arguments.get(0));
-            Object operand = arguments.get(1);
-            TypeHandler typeHandler = ctx.getTypeHandler();
+            final String op = String.valueOf(arguments.get(0));
+            final Object operand = arguments.get(1);
+            final TypeHandler typeHandler = ctx.getTypeHandler();
+            if (naturalArray != null) {
+                return new LazyNaturalArrayMapView(naturalArray, new NaturalArrayMapper() {
+                    @Override
+                    public Object map(long index, Object value) {
+                        return applyOperator(typeHandler, value, op, operand);
+                    }
+                });
+            }
             List<Object> result = new ArrayList<Object>(source.size());
             for (int i = 0; i < source.size(); i++) {
-                Object value = source.get(i);
-                result.add(applyOperator(typeHandler, value, op, operand));
+                result.add(applyOperator(typeHandler, source.get(i), op, operand));
             }
             return result;
         }
 
         if (arguments.size() != 1) {
             throw new ProgramError("map callback mode expects exactly one argument");
         }
-
+        final Object callback = arguments.get(0);
+        if (naturalArray != null) {
+            return new LazyNaturalArrayMapView(naturalArray, new NaturalArrayMapper() {
+                @Override
+                public Object map(long index, Object value) {
+                    return invokeArrayCallback(callback, "map", ctx, value, Integer.valueOf((int) index));
+                }
+            });
+        }
         List<Object> result = new ArrayList<Object>(source.size());
         for (int i = 0; i < source.size(); i++) {
-            Object value = source.get(i);
-            Object mapped = invokeArrayCallback(arguments.get(0), "map", ctx, value, i);
-            result.add(mapped);
+            result.add(invokeArrayCallback(callback, "map", ctx, source.get(i), Integer.valueOf(i)));
         }
         return result;
     }
@@ -650,12 +668,22 @@ private Object handleArrayFilter(Object literal, List<Object> arguments, Executi
         if (arguments == null || arguments.isEmpty()) {
             throw new ProgramError("filter expects a callback or (operator, operand)");
         }
-        List<Object> source = asList(literal);
+        final NaturalArray naturalArray = asNaturalArray(literal);
+        List<Object> source = naturalArray == null ? asConcreteList(literal) : null;
 
         if (looksLikeOperatorFilter(arguments)) {
-            String op = String.valueOf(arguments.get(0));
-            Object operand = arguments.get(1);
-            TypeHandler typeHandler = ctx.getTypeHandler();
+            final String op = String.valueOf(arguments.get(0));
+            final Object operand = arguments.get(1);
+            final TypeHandler typeHandler = ctx.getTypeHandler();
+            if (naturalArray != null) {
+                return new LazyNaturalArrayFilterView(naturalArray, new NaturalArrayPredicate() {
+                    @Override
+                    public boolean test(long index, Object value) {
+                        Object comparison = compareWithOperator(typeHandler, value, op, operand);
+                        return isTruthy(comparison);
+                    }
+                });
+            }
             List<Object> result = new ArrayList<Object>();
             for (int i = 0; i < source.size(); i++) {
                 Object value = source.get(i);
@@ -670,11 +698,20 @@ private Object handleArrayFilter(Object literal, List<Object> arguments, Executi
         if (arguments.size() != 1) {
             throw new ProgramError("filter callback mode expects exactly one argument");
         }
-
+        final Object callback = arguments.get(0);
+        if (naturalArray != null) {
+            return new LazyNaturalArrayFilterView(naturalArray, new NaturalArrayPredicate() {
+                @Override
+                public boolean test(long index, Object value) {
+                    Object keep = invokeArrayCallback(callback, "filter", ctx, value, Integer.valueOf((int) index));
+                    return isTruthy(keep);
+                }
+            });
+        }
         List<Object> result = new ArrayList<Object>();
         for (int i = 0; i < source.size(); i++) {
             Object value = source.get(i);
-            Object keep = invokeArrayCallback(arguments.get(0), "filter", ctx, value, i);
+            Object keep = invokeArrayCallback(callback, "filter", ctx, value, Integer.valueOf(i));
             if (isTruthy(keep)) {
                 result.add(value);
             }
@@ -686,8 +723,17 @@ private Object handleArrayReduce(Object literal, List<Object> arguments, Executi
         if (arguments == null || arguments.isEmpty() || arguments.size() > 2) {
             throw new ProgramError("reduce expects callback/op and optional initial value");
         }
-        List<Object> source = asList(literal);
-        if (source.isEmpty() && arguments.size() < 2) {
+        NaturalArray naturalArray = asNaturalArray(literal);
+        List<Object> source = naturalArray == null ? asConcreteList(literal) : null;
+        long sourceSizeLong = naturalArray != null ? naturalArray.size() : source.size();
+        if (sourceSizeLong > Integer.MAX_VALUE) {
+            throw new ProgramError(
+                "Reduce operation cannot process arrays larger than Integer.MAX_VALUE ("
+                    + Integer.MAX_VALUE + ") elements. Current size: " + sourceSizeLong
+            );
+        }
+        int sourceSize = (int) sourceSizeLong;
+        if (sourceSize == 0 && arguments.size() < 2) {
             throw new ProgramError("reduce on empty array requires an initial value");
         }
 
@@ -699,12 +745,13 @@ private Object handleArrayReduce(Object literal, List<Object> arguments, Executi
                 accumulator = arguments.get(1);
                 startIndex = 0;
             } else {
-                accumulator = source.get(0);
+                accumulator = naturalArray != null ? naturalArray.get(0L) : source.get(0);
                 startIndex = 1;
             }
             TypeHandler typeHandler = ctx.getTypeHandler();
-            for (int i = startIndex; i < source.size(); i++) {
-                accumulator = applyOperator(typeHandler, accumulator, op, source.get(i));
+            for (int i = startIndex; i < sourceSize; i++) {
+                Object next = naturalArray != null ? naturalArray.get((long) i) : source.get(i);
+                accumulator = applyOperator(typeHandler, accumulator, op, next);
             }
             return accumulator;
         }
@@ -715,16 +762,121 @@ private Object handleArrayReduce(Object literal, List<Object> arguments, Executi
             accumulator = arguments.get(1);
             startIndex = 0;
         } else {
-            accumulator = source.get(0);
+            accumulator = naturalArray != null ? naturalArray.get(0L) : source.get(0);
             startIndex = 1;
         }
 
-        for (int i = startIndex; i < source.size(); i++) {
-            Object value = source.get(i);
-            accumulator = invokeArrayCallback(arguments.get(0), "reduce", ctx, accumulator, value, i);
+        for (int i = startIndex; i < sourceSize; i++) {
+            Object value = naturalArray != null ? naturalArray.get((long) i) : source.get(i);
+            accumulator = invokeArrayCallback(arguments.get(0), "reduce", ctx, accumulator, value, Integer.valueOf(i));
         }
         return accumulator;
     }
+
+    private interface NaturalArrayMapper {
+        Object map(long index, Object value);
+    }
+
+    private interface NaturalArrayPredicate {
+        boolean test(long index, Object value);
+    }
+
+    private static final class LazyNaturalArrayMapView extends AbstractList<Object> {
+        private final NaturalArray source;
+        private final NaturalArrayMapper mapper;
+        private final int size;
+
+        private LazyNaturalArrayMapView(NaturalArray source, NaturalArrayMapper mapper) {
+            this.source = source;
+            this.mapper = mapper;
+            long sourceSize = source.size();
+            if (sourceSize > Integer.MAX_VALUE) {
+                throw new ProgramError(
+                    "Mapped array size exceeds Integer.MAX_VALUE (" + Integer.MAX_VALUE + "): " + sourceSize
+                );
+            }
+            this.size = (int) sourceSize;
+        }
+
+        @Override
+        public Object get(int index) {
+            if (index < 0 || index >= size) {
+                throw new ProgramError("Index: " + index + ", Size: " + size);
+            }
+            Object value = source.get((long) index);
+            return mapper.map((long) index, value);
+        }
+
+        @Override
+        public int size() {
+            return size;
+        }
+    }
+
+    private static final class LazyNaturalArrayFilterView extends AbstractList<Object> {
+        private final NaturalArray source;
+        private final NaturalArrayPredicate predicate;
+        private final List<Long> acceptedSourceIndices;
+        private long scanned;
+        private final long sourceSize;
+        private boolean fullyScanned;
+
+        private LazyNaturalArrayFilterView(NaturalArray source, NaturalArrayPredicate predicate) {
+            this.source = source;
+            this.predicate = predicate;
+            this.acceptedSourceIndices = new ArrayList<Long>();
+            this.scanned = 0L;
+            this.sourceSize = source.size();
+            this.fullyScanned = false;
+        }
+
+        @Override
+        public Object get(int index) {
+            if (index < 0) {
+                throw new ProgramError("Negative index: " + index);
+            }
+            ensureAcceptedIndex(index);
+            if (index >= acceptedSourceIndices.size()) {
+                throw new ProgramError("Index: " + index + ", Size: " + acceptedSourceIndices.size());
+            }
+            long sourceIndex = acceptedSourceIndices.get(index).longValue();
+            return source.get(sourceIndex);
+        }
+
+        @Override
+        public int size() {
+            scanToEnd();
+            return acceptedSourceIndices.size();
+        }
+
+        private void ensureAcceptedIndex(int index) {
+            while (!fullyScanned && acceptedSourceIndices.size() <= index) {
+                scanNext();
+            }
+        }
+
+        private void scanToEnd() {
+            while (!fullyScanned) {
+                scanNext();
+            }
+        }
+
+        private void scanNext() {
+            if (fullyScanned) return;
+            if (scanned >= sourceSize) {
+                fullyScanned = true;
+                return;
+            }
+            Object value = source.get(scanned);
+            if (predicate.test(scanned, value)) {
+                acceptedSourceIndices.add(Long.valueOf(scanned));
+            }
+            scanned++;
+            if (scanned >= sourceSize) {
+                fullyScanned = true;
+            }
+        }
+    }
 
     private boolean looksLikeOperatorMap(List<Object> arguments) {
         return arguments.size() == 2 && isOperatorReduce(arguments.get(0));