Optimize memory usage and JSON parsing in monitoring/evaluation scripts#57
Conversation
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hi @Copilot! 👋
Your private repo does not have access to Sourcery.
Please upgrade to continue using Sourcery ✨
- Replace readlines() with deque-based tail in monitoring scripts - Add reusable performance_utils module with optimized patterns - Optimize JSON parsing in batch evaluator (search from end) - Add comprehensive performance optimization guide - Update scripts to use shared utilities Co-authored-by: Bryan-Roe <74067792+Bryan-Roe@users.noreply.github.com>
- Create benchmark_performance.py showing 4.6x average speedup - Add @timeit decorator for function timing - Add @memoize_with_ttl for time-based caching - All utilities tested and documented with examples Co-authored-by: Bryan-Roe <74067792+Bryan-Roe@users.noreply.github.com>
- Document all optimizations and their impact - Include benchmark results and usage examples - Provide recommendations for future work - Summarize 4.6x average speedup achievement Co-authored-by: Bryan-Roe <74067792+Bryan-Roe@users.noreply.github.com>
- Create PERFORMANCE_README.md for easy navigation - Include quick start examples and code snippets - Document all key optimizations with before/after - Provide validation commands and best practices Co-authored-by: Bryan-Roe <74067792+Bryan-Roe@users.noreply.github.com>
![bryan-roe-bot[bot]](https://avatars.githubusercontent.com/in/1073263?s=60&v=4){kind=small}
| f.write(json.dumps({'id': i, 'valid': i % 2 == 0}) + '\n') | ||
|
|
||
| print("\n2. stream_jsonl() - Valid records only:") | ||
| for obj in stream_jsonl(temp_jsonl, filter_fn=lambda x: x.get('valid')): |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
return only makes sense inside a function
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by return-not-in-function.
You can view more details about this finding in the Semgrep AppSec Platform.
|
|
||
| # Test stream_jsonl | ||
| with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.jsonl') as f: | ||
| temp_jsonl = Path(f.name) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Using 'f.name' without '.flush()' or '.close()' may cause an error because the file may not exist when 'f.name' is used. Use '.flush()' or close the file before using 'f.name'.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by tempfile-without-flush.
You can view more details about this finding in the Semgrep AppSec Platform.
|
|
||
| # Test tail_file | ||
| with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.log') as f: | ||
| temp_path = Path(f.name) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Using 'f.name' without '.flush()' or '.close()' may cause an error because the file may not exist when 'f.name' is used. Use '.flush()' or close the file before using 'f.name'.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by tempfile-without-flush.
You can view more details about this finding in the Semgrep AppSec Platform.
| t_new = time.time() - t0 | ||
|
|
||
| # Verify results match | ||
| assert count1 == count2 == 5000 |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
The application was found using assert in non-test code. Usually reserved for debug and test
code, the assert
function is commonly used to test conditions before continuing execution. However, enclosed
code will be removed
when compiling Python code to optimized byte code. Depending on the assertion and subsequent
logic, this could
lead to undefined behavior of the application or application crashes.
To remediate this issue, remove the assert calls. If necessary, replace them with either if
conditions or
try/except blocks.
Example using try/except instead of assert:
# Below try/except is equal to the assert statement of:
# assert user.is_authenticated(), "user must be authenticated"
try:
if not user.is_authenticated():
raise AuthError("user must be authenticated")
except AuthError as e:
# Handle error
# ...
# Return, do not continue processing
return
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by B101.
You can view more details about this finding in the Semgrep AppSec Platform.
| t_new = time.time() - t0 | ||
|
|
||
| # Verify results match | ||
| assert result1 == result2 |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
The application was found using assert in non-test code. Usually reserved for debug and test
code, the assert
function is commonly used to test conditions before continuing execution. However, enclosed
code will be removed
when compiling Python code to optimized byte code. Depending on the assertion and subsequent
logic, this could
lead to undefined behavior of the application or application crashes.
To remediate this issue, remove the assert calls. If necessary, replace them with either if
conditions or
try/except blocks.
Example using try/except instead of assert:
# Below try/except is equal to the assert statement of:
# assert user.is_authenticated(), "user must be authenticated"
try:
if not user.is_authenticated():
raise AuthError("user must be authenticated")
except AuthError as e:
# Handle error
# ...
# Return, do not continue processing
return
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by B101.
You can view more details about this finding in the Semgrep AppSec Platform.
| t_new = time.time() - t0 | ||
|
|
||
| # Verify results match | ||
| assert [line.strip() for line in result1] == [line.strip() for line in result2] |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
The application was found using assert in non-test code. Usually reserved for debug and test
code, the assert
function is commonly used to test conditions before continuing execution. However, enclosed
code will be removed
when compiling Python code to optimized byte code. Depending on the assertion and subsequent
logic, this could
lead to undefined behavior of the application or application crashes.
To remediate this issue, remove the assert calls. If necessary, replace them with either if
conditions or
try/except blocks.
Example using try/except instead of assert:
# Below try/except is equal to the assert statement of:
# assert user.is_authenticated(), "user must be authenticated"
try:
if not user.is_authenticated():
raise AuthError("user must be authenticated")
except AuthError as e:
# Handle error
# ...
# Return, do not continue processing
return
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by B101.
You can view more details about this finding in the Semgrep AppSec Platform.
| def wrapper(*args, **kwargs): | ||
| # Create cache key from args/kwargs | ||
| key_data = (args, tuple(sorted(kwargs.items()))) | ||
| cache_key = hashlib.md5(str(key_data).encode()).hexdigest() |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Detected MD5 hash algorithm which is considered insecure. MD5 is not collision resistant and is therefore not suitable as a cryptographic signature. Use SHA256 or SHA3 instead.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by insecure-hash-algorithm-md5.
You can view more details about this finding in the Semgrep AppSec Platform.
|
|
||
| # Old method: Load all into list | ||
| t0 = time.time() | ||
| with open(temp_path, 'r') as f: |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Missing 'encoding' parameter. 'open()' uses device locale encodings by default, corrupting files with special characters. Specify the encoding to ensure cross-platform support when opening files in text mode (e.g. encoding="utf-8").
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by unspecified-open-encoding.
You can view more details about this finding in the Semgrep AppSec Platform.
|
|
||
| # Old method: readlines() | ||
| t0 = time.time() | ||
| with open(temp_path, 'r') as f: |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Missing 'encoding' parameter. 'open()' uses device locale encodings by default, corrupting files with special characters. Specify the encoding to ensure cross-platform support when opening files in text mode (e.g. encoding="utf-8").
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by unspecified-open-encoding.
You can view more details about this finding in the Semgrep AppSec Platform.
| def wrapper(*args, **kwargs): | ||
| # Create cache key from args/kwargs | ||
| key_data = (args, tuple(sorted(kwargs.items()))) | ||
| cache_key = hashlib.md5(str(key_data).encode()).hexdigest() |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
The application was found using an insecure or risky digest or signature algorithm. MD2, MD4,
MD5 and SHA1 hash algorithms have been found to be vulnerable to producing collisions.
This means
that two different values, when hashed, can lead to the same hash value. If the application is
trying
to use these hash methods for storing passwords, then it is recommended to switch to a
password hashing
algorithm such as Argon2id or PBKDF2.
Note that the Crypto and Cryptodome Python packages are no longer recommended for
new applications, instead consider using the cryptography package.
Example of creating a SHA-384 hash using the cryptography package:
from cryptography.hazmat.primitives import hashes
# Create a SHA384 digest
digest = hashes.Hash(hashes.SHA384())
# Update the digest with some initial data
digest.update(b"some data to hash")
# Add more data to the digest
digest.update(b"some more data")
# Finalize the digest as bytes
result = digest.finalize()
For more information on secure password storage see OWASP:
For more information on the cryptography module see:
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by B303-1.
You can view more details about this finding in the Semgrep AppSec Platform.
|
|
||
| # Create test JSONL file | ||
| with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.jsonl') as f: | ||
| temp_path = Path(f.name) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Using 'f.name' without '.flush()' or '.close()' may cause an error because the file may not exist when 'f.name' is used. Use '.flush()' or close the file before using 'f.name'.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by tempfile-without-flush.
You can view more details about this finding in the Semgrep AppSec Platform.
|
|
||
| # Create large test file | ||
| with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.log') as f: | ||
| temp_path = Path(f.name) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Using 'f.name' without '.flush()' or '.close()' may cause an error because the file may not exist when 'f.name' is used. Use '.flush()' or close the file before using 'f.name'.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by tempfile-without-flush.
You can view more details about this finding in the Semgrep AppSec Platform.
| print(f" Speedup: {t1/t2:.1f}x") | ||
|
|
||
| # Wait for TTL expiration | ||
| time.sleep(1.1) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
time.sleep() call; did you mean to leave this in?
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by arbitrary-sleep.
You can view more details about this finding in the Semgrep AppSec Platform.
| def expensive_computation(x): | ||
| global call_count | ||
| call_count += 1 | ||
| time.sleep(0.05) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
time.sleep() call; did you mean to leave this in?
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by arbitrary-sleep.
You can view more details about this finding in the Semgrep AppSec Platform.
|
|
||
| @timeit | ||
| def slow_function(): | ||
| time.sleep(0.1) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
time.sleep() call; did you mean to leave this in?
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by arbitrary-sleep.
You can view more details about this finding in the Semgrep AppSec Platform.
There was a problem hiding this comment.
Hi @Bryan-Roe! 👋
Your private repo does not have access to Sourcery.
Please upgrade to continue using Sourcery ✨
There was a problem hiding this comment.
Pull request overview
Adds shared performance utilities and integrates them into monitoring/evaluation flows to reduce memory usage (log tailing) and speed up JSON extraction from command output, with accompanying benchmarks + docs.
Changes:
- Introduces
shared/performance_utils.py(tailing, JSONL streaming, reverse JSON search, caching, timing/memoization helpers). - Switches log tailing and metrics extraction in scripts/dashboard to use the shared optimized utilities.
- Adds benchmark script and performance documentation/implementation notes.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| shared/performance_utils.py | New reusable utilities for tailing, JSON parsing, streaming JSONL, caching, and profiling. |
| scripts/monitor_autonomous_training.py | Uses shared tail_file() for memory-efficient log tailing. |
| scripts/batch_evaluator.py | Uses shared find_json_in_output() for faster metrics extraction. |
| dashboard/serve.py | Uses shared tail_file() to avoid loading entire logs into memory. |
| scripts/benchmark_performance.py | Adds benchmark suite validating speed/memory improvements. |
| docs/PERFORMANCE_README.md | Quick-reference documentation for the new utilities and patterns. |
| docs/PERFORMANCE_OPTIMIZATION_GUIDE.md | Expanded guide describing patterns and anti-patterns. |
| docs/PERFORMANCE_IMPLEMENTATION_SUMMARY.md | Implementation summary + results and usage notes. |
| # Large file: read backwards in blocks | ||
| block_size = 8192 | ||
| with file_path.open("rb") as f: | ||
| pos = max(0, size - block_size) | ||
| f.seek(pos) | ||
| buf = f.read(block_size) | ||
|
|
||
| while True: | ||
| decoded = buf.decode("utf-8", errors="ignore") | ||
| lines = decoded.splitlines() | ||
|
|
||
| if len(lines) >= max_lines or pos == 0: | ||
| return lines[-max_lines:] | ||
|
|
||
| # Move further back | ||
| new_pos = max(0, pos - block_size) | ||
| read_size = pos - new_pos | ||
| f.seek(new_pos) | ||
| more = f.read(read_size) | ||
| buf = more + buf | ||
| pos = new_pos |
There was a problem hiding this comment.
tail_file_smart() can return corrupted/truncated lines for large files because splitlines() is applied to progressively prepended decoded blocks without handling partial-line boundaries across block splits. Consider implementing the backward scan in bytes (split on b'\\n') while carrying a remainder between blocks, or fall back to the streaming deque approach if correctness is more important than the block-based optimization.
| # Large file: read backwards in blocks | |
| block_size = 8192 | |
| with file_path.open("rb") as f: | |
| pos = max(0, size - block_size) | |
| f.seek(pos) | |
| buf = f.read(block_size) | |
| while True: | |
| decoded = buf.decode("utf-8", errors="ignore") | |
| lines = decoded.splitlines() | |
| if len(lines) >= max_lines or pos == 0: | |
| return lines[-max_lines:] | |
| # Move further back | |
| new_pos = max(0, pos - block_size) | |
| read_size = pos - new_pos | |
| f.seek(new_pos) | |
| more = f.read(read_size) | |
| buf = more + buf | |
| pos = new_pos | |
| # Large file: read backwards in blocks, operating on bytes to | |
| # correctly handle line boundaries across block splits. | |
| block_size = 8192 | |
| with file_path.open("rb") as f: | |
| pos = size | |
| lines_bytes = [] | |
| remainder = b"" | |
| while pos > 0 and len(lines_bytes) < max_lines: | |
| new_pos = max(0, pos - block_size) | |
| read_size = pos - new_pos | |
| f.seek(new_pos) | |
| chunk = f.read(read_size) | |
| # Prepend the previously saved remainder (partial line from earlier) | |
| buf = chunk + remainder | |
| parts = buf.split(b"\n") | |
| # The first element may be an incomplete line from earlier in the file. | |
| remainder = parts[0] | |
| # All subsequent elements are complete lines within this region. | |
| complete_lines = parts[1:] | |
| # We are scanning from the end backward, so add lines in reverse order. | |
| for line in reversed(complete_lines): | |
| lines_bytes.append(line) | |
| if len(lines_bytes) >= max_lines: | |
| break | |
| pos = new_pos | |
| # If we've reached the beginning of the file and still have a remainder, | |
| # it represents the first line in the file. | |
| if pos == 0 and remainder: | |
| lines_bytes.append(remainder) | |
| # If the file ended with a newline, the first collected "line" may be empty; | |
| # drop that to match the behavior of readlines()/deque on text files. | |
| if lines_bytes and lines_bytes[0] == b"": | |
| lines_bytes = lines_bytes[1:] | |
| # We collected lines from the end towards the beginning; reverse to restore | |
| # the natural order and limit to the requested number of lines. | |
| selected = list(reversed(lines_bytes[:max_lines])) | |
| # Decode each line as UTF-8, ignoring errors, and re-add a trailing newline | |
| # so the format is similar to readlines()/deque(f). | |
| result_lines: List[str] = [] | |
| for bline in selected: | |
| text = bline.decode("utf-8", errors="ignore") | |
| if text and not text.endswith("\n"): | |
| text = text + "\n" | |
| result_lines.append(text) | |
| return result_lines |
| >>> | ||
| >>> large_dataset = load_data() | ||
| >>> batch_process(large_dataset, batch_size=100, process_fn=save_batch) | ||
| """ |
There was a problem hiding this comment.
batch_process() will raise at runtime when batch_size is 0 (invalid step for range) and behaves unexpectedly for negative sizes. Add an explicit validation at the start (e.g., raise ValueError with a clear message) to fail fast and make misuse easier to diagnose.
| """ | |
| """ | |
| if batch_size <= 0: | |
| raise ValueError("batch_size must be a positive integer") |
| # Split and optionally reverse for end-first search | ||
| lines = output.rsplit('\n', max_lines) if search_from_end else output.split('\n', max_lines) | ||
| line_iter = reversed(lines) if search_from_end else lines |
There was a problem hiding this comment.
find_json_in_output(..., max_lines=...) does not actually cap the search to max_lines lines as documented: split(..., max_lines) / rsplit(..., max_lines) produce up to max_lines + 1 segments and can include more (or fewer, depending on presence of newlines) than intended. Consider switching to splitlines() and slicing (e.g., last max_lines for end-search, first max_lines for forward-search) so max_lines matches the docstring semantics.
| # Split and optionally reverse for end-first search | |
| lines = output.rsplit('\n', max_lines) if search_from_end else output.split('\n', max_lines) | |
| line_iter = reversed(lines) if search_from_end else lines | |
| # Split into logical lines and optionally reverse for end-first search | |
| all_lines = output.splitlines() | |
| if search_from_end: | |
| lines = all_lines[-max_lines:] if max_lines is not None else all_lines | |
| line_iter = reversed(lines) | |
| else: | |
| lines = all_lines[:max_lines] | |
| line_iter = iter(lines) |
| # Add shared directory to path for performance utilities | ||
| REPO_ROOT = Path(__file__).resolve().parents[1] | ||
| sys.path.insert(0, str(REPO_ROOT / "shared")) | ||
|
|
||
| from performance_utils import tail_file |
There was a problem hiding this comment.
Modifying sys.path to import performance_utils is brittle (can shadow other modules named performance_utils, and differs from the documented from shared.performance_utils import ... pattern). Prefer adding REPO_ROOT to sys.path and importing shared.performance_utils, or (better) make shared/ a proper package and use consistent absolute imports across monitor_autonomous_training.py, batch_evaluator.py, benchmark_performance.py, and dashboard/serve.py.
| # Add shared directory to path for performance utilities | |
| REPO_ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(REPO_ROOT / "shared")) | |
| from performance_utils import tail_file | |
| # Add repository root to path so we can import shared utilities as a package | |
| REPO_ROOT = Path(__file__).resolve().parents[1] | |
| sys.path.insert(0, str(REPO_ROOT)) | |
| from shared.performance_utils import tail_file |
| from shared.performance_utils import ( | ||
| tail_file, # Memory-efficient log tailing | ||
| stream_jsonl, # Generator-based JSONL reading | ||
| find_json_in_output, # Fast JSON extraction | ||
| FileCache, # In-memory file caching | ||
| timeit, # Function timing decorator | ||
| memoize_with_ttl # Time-based memoization | ||
| ) |
There was a problem hiding this comment.
The docs recommend from shared.performance_utils import ..., but the updated scripts import via sys.path.insert(.../shared) and from performance_utils import .... Align the documentation and the code to one canonical import approach to avoid confusion for users copying examples.
| import time | ||
| import json | ||
| from pathlib import Path | ||
| from collections import deque |
There was a problem hiding this comment.
deque is imported but not used in this benchmark script. Removing it reduces lint noise and keeps the benchmark focused on the utilities being measured.
| from collections import deque |
| from collections import deque | ||
| from pathlib import Path | ||
| from typing import List, Iterator, Optional, Callable, Any | ||
| from functools import wraps, lru_cache |
There was a problem hiding this comment.
lru_cache is imported but not used in this module. Consider removing the unused import (or adding a concrete use) to keep the utility module clean.
| from functools import wraps, lru_cache | |
| from functools import wraps |
github-actions
Bot
added
documentation
Identified and fixed memory-inefficient file reading and slow JSON parsing patterns across monitoring and batch evaluation scripts. Results: 4.6x average speedup, 6+ MB memory savings per operation.
Changes
Memory-efficient log tailing
readlines()loads entire files (100K+ lines) into memory for tail operationscollections.deque(file, maxlen=N)- O(n) → O(k) complexityscripts/monitor_autonomous_training.py,dashboard/serve.pyReverse-search JSON parsing
rsplit()+ reverse iteration, limit to last 50 linesscripts/batch_evaluator.pyReusable utilities
Created
shared/performance_utils.pywith 7 optimized patterns:tail_file()- deque-based log tailingstream_jsonl()- generator-based JSONL processing (1.1x faster, 1.2 MB saved)find_json_in_output()- reverse-search JSON extractionFileCache- in-memory cache with size limits@timeit- function timing decorator@memoize_with_ttl- time-based memoizationValidation
Benchmark suite (
scripts/benchmark_performance.py) validates improvements:Documentation in
docs/PERFORMANCE_OPTIMIZATION_GUIDE.mdcovers patterns, anti-patterns, and best practices.✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.