From 3e25beee8c2bb14af6613d218de551c9a372aaeb Mon Sep 17 00:00:00 2001 From: vibecodingtoolmaker <312078684+vibecodingtoolmaker@users.noreply.github.com> Date: Mon, 3 Aug 2026 16:29:29 +0200 Subject: [PATCH 1/7] Document beta table-submit workaround --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 0d1ff0a..35b8fa0 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,11 @@ returned as recovery output and the recovery location is logged. spacing currently applies only when **Draw Legend in Matrix Grid** is enabled. - When model unloading is enabled, saved matrix grids currently use the Forge grid output root instead of Forge's date subfolder layout. +- After correcting an invalid value in an actively edited per-LoRA table cell, the + first **Generate** click can commit the corrected cell and return focus to the table + without starting generation. Click **Generate** again; the corrected run then starts + normally. This is a Gradio table-editing usability issue and does not lose generated + images or settings. ## Privacy and security From 1c86b97d90b2371adeabb25a6dedf522bb0d9cc9 Mon Sep 17 00:00:00 2001 From: vibecodingtoolmaker <312078684+vibecodingtoolmaker@users.noreply.github.com> Date: Mon, 3 Aug 2026 18:11:21 +0200 Subject: [PATCH 2/7] Add welcome and AI transparency notice --- README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/README.md b/README.md index 35b8fa0..ae361b2 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,26 @@ fixed seed. It returns the results as one or more labeled, RAM-aware matrix page > models and LoRA collections. Keep recoverable individual images enabled for > important runs. +## Welcome + +Hello and welcome! I hope LoRA Tester makes it easier to compare large LoRA +collections, explore useful weight ranges, and keep the results understandable in a +single matrix. Feedback, careful testing, and constructive contributions are very +welcome. + +## AI-assisted development transparency + +LoRA Tester is created and maintained by **vibecodingtoolmaker** with substantial +AI-assisted development support from **OpenAI Codex**. Code, documentation, reviews, +and test ideas have been developed and refined through this collaborative workflow — +also known, with some affection, as *vibecoding*. + +The project direction, feature decisions, hands-on Forge testing, final review, and +release responsibility remain with the human maintainer. Codex is a development tool, +not a runtime dependency: the installed extension does not contact OpenAI or send +prompts, images, model information, or other user data to Codex. This project is not +affiliated with or endorsed by OpenAI. + ## Features - Loads the LoRA list automatically on first activation From aac2287edd51d4292e29d4108a0f6b75f086bea5 Mon Sep 17 00:00:00 2001 From: vibecodingtoolmaker <312078684+vibecodingtoolmaker@users.noreply.github.com> Date: Mon, 3 Aug 2026 18:32:16 +0200 Subject: [PATCH 3/7] Prepare v0.1.0-beta.1 release --- CHANGELOG.md | 10 ++++++++-- README.md | 10 +++++----- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 418c351..7f43251 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ## [Unreleased] +## [0.1.0-beta.1] - 2026-08-03 + ### Changed - Temporarily disabled the adaptive RAM watchdog for the v0.1.0 release while its thresholds are evaluated further. Disk spooling and pre-matrix model unloading remain active. @@ -20,11 +22,15 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ### Added -- Initial public pre-release for Forge Neo. +- Initial public beta release for Forge Neo. - Fixed-seed LoRA and weight-range comparisons in labeled matrix pages. - Per-LoRA trigger-word and weight overrides. - Folder-scoped LoRA selection with optional subfolder inclusion. - Disk-spooled source images and recovery manifests. -- Adaptive matrix-page packing and RAM protection. +- Adaptive matrix-page packing and an experimental RAM-watchdog implementation that + remains disabled in this release. - Optional checkpoint, text-encoder, and VAE unload before matrix creation. - Recovery-safe matrix-only output mode. + +[Unreleased]: https://github.com/vibecodingtoolmaker/Forge-Neo-Lora-Tester/compare/v0.1.0-beta.1...HEAD +[0.1.0-beta.1]: https://github.com/vibecodingtoolmaker/Forge-Neo-Lora-Tester/releases/tag/v0.1.0-beta.1 diff --git a/README.md b/README.md index ae361b2..4395695 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ LoRA Tester compares multiple LoRAs and weight ranges with one prompt and one fixed seed. It returns the results as one or more labeled, RAM-aware matrix pages. -> **Project status:** v0.1.0 pre-release. The extension is being stress-tested with large -> models and LoRA collections. Keep recoverable individual images enabled for -> important runs. +> **Project status:** v0.1.0-beta.1 public beta. The extension has been tested with +> large models and LoRA collections, but known limitations remain. Keep recoverable +> individual images enabled for important runs and please report reproducible issues. ## Welcome @@ -56,7 +56,7 @@ The extension does not install or download packages by itself. The optional ## Installation 1. Download or clone this repository into Forge Neo's `extensions` directory. -2. Keep the repository folder name as `Forge_Neo_Lora_tester`. +2. Keep the repository folder name as `Forge-Neo-Lora-Tester`. 3. Restart Forge Neo. The **LoRA Tester** accordion is available in both txt2img and img2img. @@ -233,7 +233,7 @@ returned as recovery output and the recovery location is logged. fields. Incomplete rows block generation and identify the affected LoRA in the UI and console instead of silently using another value. -## Known limitations in v0.1.0 beta +## Known limitations in v0.1.0-beta.1 - Adaptive RAM protection is visible but intentionally disabled while universal thresholds are evaluated. Disk spooling and optional model unloading remain active. From 2f60ada32da482040dde850afeeab57261c9e686 Mon Sep 17 00:00:00 2001 From: vibecodingtoolmaker <312078684+vibecodingtoolmaker@users.noreply.github.com> Date: Mon, 3 Aug 2026 18:48:20 +0200 Subject: [PATCH 4/7] Correct incomplete-weight documentation --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4395695..1aceb5e 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,8 @@ Use the weight columns as follows: - `0.8 | blank | blank` uses one dedicated weight. - `-1 | 1 | 0.25` creates an inclusive range from -1 to 1. - Three blank weight cells use the global specification. -- An incomplete combination falls back to the global value and logs a warning. +- An incomplete combination blocks generation and identifies the affected LoRA in + the UI and console. Correct the row, then start generation again. One click opens an editable cell and places the caret at the beginning without selecting or clearing its contents. Tab, Shift+Tab, Enter, and arrow-key cell From af2432a54d8b799caab4474ed26f99bbfb87c983 Mon Sep 17 00:00:00 2001 From: vibecodingtoolmaker <312078684+vibecodingtoolmaker@users.noreply.github.com> Date: Wed, 5 Aug 2026 09:09:18 +0200 Subject: [PATCH 5/7] Add extreme runs and repeated matrix reference # Conflicts: # CHANGELOG.md --- CHANGELOG.md | 9 ++ README.md | 22 ++- scripts/lora_tester.py | 229 +++++++++++++++++++++++------ tests/test_lora_tester_recovery.py | 107 +++++++++++++- 4 files changed, 319 insertions(+), 48 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f43251..7391cdd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ## [Unreleased] +### Changed + +- Enabled the fixed-seed reference image by default and repeat it on every matrix page. +- Raised the per-page image-dimension ceiling from 60,000 to 65,000 pixels while retaining automatic page splitting. + +### Added + +- Added an explicit Extreme Run Mode that raises the normal 500-cell limit to 10,000 cells with a long-running stability and disk-usage warning. + ## [0.1.0-beta.1] - 2026-08-03 ### Changed diff --git a/README.md b/README.md index 1aceb5e..eeab9fb 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,9 @@ affiliated with or endorsed by OpenAI. - Supports per-LoRA trigger text and weight overrides - Supports positive and negative weights and inclusive ranges such as `-3:3:0.5` - Uses the first generated seed for every comparison cell -- Optionally includes a baseline without a LoRA +- Generates a fixed-seed reference image without a tested LoRA by default +- Repeats that reference image at the top of every matrix page +- Offers an explicit Extreme Run Mode for up to 10,000 matrix cells - Writes completed cells to disk instead of retaining decoded images in RAM - Builds labeled row strips on disk before composing the final matrix - Retains the adaptive RAM-protection implementation behind a disabled release switch @@ -71,11 +73,12 @@ The **LoRA Tester** accordion is available in both txt2img and img2img. are included. Click **Refresh LoRA List** after adding or removing model files. 4. Enter one global weight or range. 5. Edit trigger words or weight overrides in the per-LoRA table if needed. -6. Configure the baseline, trigger position, label display, matrix columns, and - margin. +6. Configure the reference image, trigger position, label display, matrix columns, + and margin. The reference is enabled by default and repeated on every page. 7. Choose an output-retention mode. Keeping individual images is the safe default. 8. Keep model unloading enabled so matrix creation can use the released RAM. -9. Generate normally. The gallery returns matrix pages rather than individual cells. +9. Enable **Extreme Run Mode** only for deliberately large jobs above 500 cells. +10. Generate normally. The gallery returns matrix pages rather than individual cells. Generation is blocked while LoRA Tester is enabled and no LoRA is selected. Disabling the accordion leaves normal Forge generation unchanged. @@ -115,7 +118,11 @@ typing, deletion, Enter, and double-click editing. - Inclusive descending range: `3:-3:0.5` The syntax is `start:end:step`; direction is inferred automatically. One -specification is limited to 100 values and one run to 500 matrix cells. +specification is limited to 100 values. A normal run is limited to 500 matrix +cells. **Extreme Run Mode** raises this fixed limit to 10,000 cells and must be +enabled explicitly. Extreme runs can take many hours or days, consume very large +amounts of temporary disk space, and may expose long-duration Forge, driver, +model, extension, or system instability. ## Trigger metadata @@ -179,7 +186,10 @@ delay. ## Output and recovery -Only matrix pages are returned to the Forge gallery. **Matrix + individual images** +Only matrix pages are returned to the Forge gallery. Pages are kept below 65,000 +pixels on either axis and split automatically when necessary. When the default +reference generation is enabled, its fixed-seed image appears at the top of every +page. **Matrix + individual images** keeps every final source cell as a lossless PNG with generation metadata. Files are grouped per run and use names such as: diff --git a/scripts/lora_tester.py b/scripts/lora_tester.py index 66e07c6..024adbd 100644 --- a/scripts/lora_tester.py +++ b/scripts/lora_tester.py @@ -222,7 +222,8 @@ class LoRaTesterScript(scripts.Script): PROCESSING_STATE_ATTRIBUTE = "_lora_tester_state" MAX_WEIGHTS_PER_LORA = 100 MAX_TOTAL_CASES = 500 - MAX_MATRIX_DIMENSION = 60_000 + MAX_EXTREME_TOTAL_CASES = 10_000 + MAX_MATRIX_DIMENSION = 65_000 PAGE_SAFETY_FACTOR = 1.35 LABEL_FONT_DIVISOR = 36 LABEL_MIN_FONT_SIZE = 14 @@ -359,6 +360,16 @@ def _parse_weight_spec(cls, spec, description): return values + @classmethod + def _maximum_total_cases(cls, extreme_run_mode): + """Return the explicit normal or opt-in extreme run limit.""" + + return ( + cls.MAX_EXTREME_TOTAL_CASES + if extreme_run_mode + else cls.MAX_TOTAL_CASES + ) + @staticmethod def _block_generation(p, message): """Stop a LoRA Tester run while keeping the reason visible to the user.""" @@ -835,12 +846,15 @@ def _write_manifest(state, status): 'last_generation_memory': state.get('last_generation_memory'), 'ram_stop_reason': state.get('ram_stop_reason'), 'output_retention': state.get('output_retention'), + 'extreme_run_mode': state.get('extreme_run_mode', False), + 'case_limit': state.get('case_limit'), 'individual_output_dir': state.get('individual_output_dir'), 'individual_images_finalized': state.get('individual_images_finalized', 0), 'individual_finalize_error': state.get('individual_finalize_error'), 'individual_finalize_failures': state.get('individual_finalize_failures', 0), 'expected_cell_count': len(state.get('cases', [])), 'completed_cell_count': len(cells), + 'reference_row': state.get('reference_row'), 'generation_interrupted': state.get('generation_interrupted', False), 'matrix_only_delete_allowed': state.get( 'matrix_only_delete_allowed', @@ -1006,8 +1020,12 @@ def ui(self, is_img2img): with gr.Accordion("Advanced Options", open=False): with FormRow(): save_original = gr.Checkbox( - value=False, - label="Also generate without any LoRA (baseline)", + value=True, + label="Generate reference image without tested LoRA (Recommended)", + info=( + "The fixed-seed reference is repeated at the top of every " + "matrix page for direct comparison." + ), elem_id=f"{'img2img' if is_img2img else 'txt2img'}_lora_tester_baseline" ) @@ -1017,6 +1035,17 @@ def ui(self, is_img2img): elem_id=f"{'img2img' if is_img2img else 'txt2img'}_lora_tester_draw_legend" ) + extreme_run_mode = gr.Checkbox( + value=False, + label="Extreme Run Mode (up to 10,000 matrix cells)", + info=( + "CAUTION: Extreme runs can take many hours or days and may use " + "very large amounts of disk space. Forge, driver, model, extension, " + "or system instability cannot be ruled out during very long runs." + ), + elem_id=f"{'img2img' if is_img2img else 'txt2img'}_lora_tester_extreme_run" + ) + with FormRow(): matrix_cols = gr.Slider( minimum=1, @@ -1233,6 +1262,7 @@ def update_lora_settings(selected_loras, current_settings): trigger_position, lora_settings, save_original, + extreme_run_mode, draw_legend, matrix_cols, matrix_margin, @@ -1247,7 +1277,8 @@ def update_lora_settings(selected_loras, current_settings): def before_process(self, p: StableDiffusionProcessing, lora_tester_enabled, lora_selection, global_weight_spec, use_trigger_words, trigger_position, lora_settings, - save_original, draw_legend, matrix_cols, matrix_margin, + save_original, extreme_run_mode, draw_legend, + matrix_cols, matrix_margin, output_retention, unload_models_before_matrix, adaptive_ram, ram_budget, custom_ram_gb, minimum_free_ram_gb): @@ -1295,6 +1326,7 @@ def before_process(self, p: StableDiffusionProcessing, if row[0] not in (None, "") } + case_limit = self._maximum_total_cases(extreme_run_mode) cases = [] if save_original: cases.append({ @@ -1365,12 +1397,18 @@ def before_process(self, p: StableDiffusionProcessing, return for weight in weights: - if len(cases) >= self.MAX_TOTAL_CASES: - print( - f"[LoRA Tester] Reached the safety limit of {self.MAX_TOTAL_CASES} " - "matrix cells; remaining weights were skipped" + if len(cases) >= case_limit: + mode_hint = ( + "Reduce the selection or weight ranges." + if extreme_run_mode + else "Reduce the run or enable Extreme Run Mode." + ) + self._block_generation( + p, + f"The requested run contains more than {case_limit:,} matrix " + f"cells. {mode_hint}", ) - break + return formatted_weight = self._format_weight(weight) cases.append({ @@ -1380,9 +1418,6 @@ def before_process(self, p: StableDiffusionProcessing, 'weight': formatted_weight, }) - if len(cases) >= self.MAX_TOTAL_CASES: - break - if not cases: return @@ -1422,6 +1457,8 @@ def before_process(self, p: StableDiffusionProcessing, state = { 'cases': cases, + 'extreme_run_mode': bool(extreme_run_mode), + 'case_limit': case_limit, 'use_trigger_words': use_trigger_words, 'trigger_position': trigger_position, 'draw_legend': draw_legend, @@ -1442,6 +1479,7 @@ def before_process(self, p: StableDiffusionProcessing, 'manifest_path': str(session_dir / "manifest.json"), 'cells': {}, 'rows': [], + 'reference_row': None, 'ram_monitor': ram_monitor, 'ram_finalizer': ram_finalizer, 'fixed_seed': None, @@ -1736,6 +1774,7 @@ def postprocess_image_after_composite(self, p: StableDiffusionProcessing, pp, *a 'label': case['label'], 'trigger_words': list(case.get('trigger_words', [])), 'weight': case.get('weight'), + 'is_baseline': case.get('lora_tag_name') is None, 'width': int(original.width), 'height': int(original.height), 'persistent': False, @@ -1791,7 +1830,8 @@ def postprocess_image_after_composite(self, p: StableDiffusionProcessing, pp, *a def postprocess(self, p: StableDiffusionProcessing, processed: Processed, lora_tester_enabled, lora_selection, global_weight_spec, use_trigger_words, trigger_position, lora_settings, - save_original, draw_legend, matrix_cols, matrix_margin, + save_original, extreme_run_mode, draw_legend, + matrix_cols, matrix_margin, output_retention, unload_models_before_matrix, adaptive_ram, ram_budget, custom_ram_gb, minimum_free_ram_gb): @@ -1842,12 +1882,43 @@ def postprocess(self, p: StableDiffusionProcessing, processed: Processed, self._unload_forge_model_for_matrix(p, state) + reference_cell = next( + (cell for cell in cells if cell.get('is_baseline')), + None, + ) + comparison_cells = [ + cell for cell in cells if cell is not reference_cell + ] + reference_row = None + if reference_cell is not None: + reference_rows = self._build_row_strips( + state, + [reference_cell], + columns=1, + filename_prefix="reference-row", + track_as_matrix_rows=False, + ) + reference_row = reference_rows[0] + state['reference_row'] = reference_row + print( - f"[LoRA Tester] Creating row strips: {len(cells)} cells, " - f"{state['matrix_cols']} configured columns" + f"[LoRA Tester] Creating row strips: {len(comparison_cells)} comparison " + f"cell(s), {state['matrix_cols']} configured columns; reference image: " + f"{'repeated on every page' if reference_row else 'disabled'}" + ) + if comparison_cells: + rows = self._build_row_strips(state, comparison_cells) + else: + rows = [] + state['rows'] = [] + page_paths = self._build_matrix_pages( + state, + rows, + p, + processed, + grid_info, + reference_row=reference_row, ) - rows = self._build_row_strips(state, cells) - page_paths = self._build_matrix_pages(state, rows, p, processed, grid_info) self._validate_matrix_pages(page_paths) processed.images = [str(path) for path in page_paths] @@ -1875,11 +1946,19 @@ def postprocess(self, p: StableDiffusionProcessing, processed: Processed, ) except Exception as error: self._write_manifest(state, "matrix-error") - fallback_paths = [ - row['path'] - for row in state.get('rows', []) - if Path(row['path']).exists() - ] + reference_path = (state.get('reference_row') or {}).get('path') + fallback_paths = ( + [reference_path] + if reference_path and Path(reference_path).exists() + else [] + ) + fallback_paths.extend( + [ + row['path'] + for row in state.get('rows', []) + if Path(row['path']).exists() + ] + ) if not fallback_paths: fallback_paths = [ cell['path'] @@ -1929,6 +2008,7 @@ def _spool_fallback_images(self, p, state, processed): 'label': case['label'], 'trigger_words': list(case.get('trigger_words', [])), 'weight': case.get('weight'), + 'is_baseline': case.get('lora_tag_name') is None, 'width': int(image.width), 'height': int(image.height), 'persistent': False, @@ -2073,10 +2153,20 @@ def _draw_compact_cell_labels( return result - def _build_row_strips(self, state, cells): + def _build_row_strips( + self, + state, + cells, + columns=None, + filename_prefix="row", + track_as_matrix_rows=True, + ): """Create labeled row files while decoding only one source cell at a time.""" - columns = max(1, min(int(state['matrix_cols']), len(cells))) + if not cells: + return [] + requested_columns = state['matrix_cols'] if columns is None else columns + columns = max(1, min(int(requested_columns), len(cells))) cell_width = max(int(cell['width']) for cell in cells) cell_height = max(int(cell['height']) for cell in cells) background = ImageColor.getcolor(shared.opts.grid_background_color, "RGB") @@ -2122,7 +2212,9 @@ def _build_row_strips(self, state, cells): else: annotated_row = row_grid - row_path = Path(state['session_dir']) / f"row-{row_index:04d}.png" + row_path = Path(state['session_dir']) / ( + f"{filename_prefix}-{row_index:04d}.png" + ) self._save_png_atomic(annotated_row, row_path) row_record = { 'index': row_index, @@ -2146,27 +2238,34 @@ def _build_row_strips(self, state, cells): if row_grid is not None: row_grid.close() - state['rows'] = rows - self._write_manifest(state, "row-strips") + if track_as_matrix_rows: + state['rows'] = rows + self._write_manifest(state, "row-strips") # Keep every source cell until all matrix pages have been written and # validated. Matrix-only deletion is deliberately deferred to the final # cleanup; a page-composition failure must leave recoverable originals. - self._write_manifest(state, "row-strips-ready") + if track_as_matrix_rows: + self._write_manifest(state, "row-strips-ready") gc.collect() return rows - def _estimate_page_peak(self, state, rows): - width = max(row['width'] for row in rows) - height = sum(row['height'] for row in rows) - height += state['matrix_margin'] * max(0, len(rows) - 1) + def _estimate_page_peak(self, state, rows, reference_row=None): + page_rows = ([reference_row] if reference_row is not None else []) + list(rows) + if not page_rows: + return 0, 0, 0 + width = max(row['width'] for row in page_rows) + height = sum(row['height'] for row in page_rows) + height += state['matrix_margin'] * max(0, len(page_rows) - 1) if width > self.MAX_MATRIX_DIMENSION or height > self.MAX_MATRIX_DIMENSION: return width, height, float('inf') # Four bytes per page pixel is deliberately conservative for an RGB PIL # canvas. Add the measured/theoretical working row and PNG encoding room. canvas_bytes = width * height * 4 - largest_row_bytes = max(row['width'] * row['height'] * 4 for row in rows) + largest_row_bytes = max( + row['width'] * row['height'] * 4 for row in page_rows + ) working_row = max( state.get('maximum_row_peak_bytes', 0), largest_row_bytes * 2, @@ -2175,10 +2274,33 @@ def _estimate_page_peak(self, state, rows): estimated_peak = int(canvas_bytes * self.PAGE_SAFETY_FACTOR + working_row + encoding_reserve) return width, height, estimated_peak - def _build_matrix_pages(self, state, rows, p, processed, grid_info): + def _build_matrix_pages( + self, state, rows, p, processed, grid_info, reference_row=None + ): page_paths = [] row_offset = 0 + if not rows and reference_row is not None: + width, height, estimated_peak = self._estimate_page_peak( + state, [], reference_row + ) + print( + f"[LoRA Tester] Matrix page 1: reference image only, " + f"estimated peak {estimated_peak / MIB:.0f} MB" + ) + page_path = self._compose_matrix_page( + state, + [], + width, + height, + 1, + p, + processed, + grid_info, + reference_row=reference_row, + ) + return [Path(page_path)] + while row_offset < len(rows): effective_available = self._effective_available(state) page_budget = max(0, effective_available - state['minimum_free_bytes']) @@ -2186,7 +2308,9 @@ def _build_matrix_pages(self, state, rows, p, processed, grid_info): for row in rows[row_offset:]: candidate = selected_rows + [row] - _, _, estimated_peak = self._estimate_page_peak(state, candidate) + _, _, estimated_peak = self._estimate_page_peak( + state, candidate, reference_row + ) dimensions_safe = estimated_peak != float('inf') fits_ram = dimensions_safe and ( not state['adaptive_ram'] @@ -2221,10 +2345,13 @@ def _build_matrix_pages(self, state, rows, p, processed, grid_info): ) continue - width, height, estimated_peak = self._estimate_page_peak(state, selected_rows) + width, height, estimated_peak = self._estimate_page_peak( + state, selected_rows, reference_row + ) print( f"[LoRA Tester] Matrix page {page_number}: {len(selected_rows)} row(s), " f"estimated peak {estimated_peak / MIB:.0f} MB" + f"{' including repeated reference' if reference_row else ''}" ) page_path = self._compose_matrix_page( state, @@ -2235,6 +2362,7 @@ def _build_matrix_pages(self, state, rows, p, processed, grid_info): p, processed, grid_info, + reference_row=reference_row, ) page_paths.append(Path(page_path)) row_offset += len(selected_rows) @@ -2242,19 +2370,34 @@ def _build_matrix_pages(self, state, rows, p, processed, grid_info): return page_paths - def _compose_matrix_page(self, state, rows, width, height, page_number, - p, processed, grid_info): + def _compose_matrix_page( + self, + state, + rows, + width, + height, + page_number, + p, + processed, + grid_info, + reference_row=None, + ): background = ImageColor.getcolor(shared.opts.grid_background_color, "RGB") page = Image.new("RGB", (width, height), background) try: + page_rows = ( + ([reference_row] if reference_row is not None else []) + list(rows) + ) y = 0 - for row in rows: + for row_index, row in enumerate(page_rows): with Image.open(row['path']) as row_image: row_image.load() converted = row_image if row_image.mode == "RGB" else row_image.convert("RGB") x = (width - converted.width) // 2 page.paste(converted, (x, y)) - y += converted.height + state['matrix_margin'] + y += converted.height + if row_index + 1 < len(page_rows): + y += state['matrix_margin'] if converted is not row_image: converted.close() @@ -2331,6 +2474,11 @@ def _cleanup_spool_files(self, state, page_paths): if row_path and str(Path(row_path).resolve()) not in protected: cleanup_complete = self._safe_unlink(row_path) and cleanup_complete + reference_row = state.get('reference_row') or {} + reference_path = reference_row.get('path') + if reference_path and str(Path(reference_path).resolve()) not in protected: + cleanup_complete = self._safe_unlink(reference_path) and cleanup_complete + if state.get('recovery_retained'): self._write_manifest(state, "recovery-retained") elif cleanup_complete: @@ -2345,4 +2493,3 @@ def _cleanup_spool_files(self, state, page_paths): except OSError: # Unsaved gallery pages intentionally remain in Forge's temp tree. pass - diff --git a/tests/test_lora_tester_recovery.py b/tests/test_lora_tester_recovery.py index e0e40dc..7218a96 100644 --- a/tests/test_lora_tester_recovery.py +++ b/tests/test_lora_tester_recovery.py @@ -2,6 +2,7 @@ # Copyright (C) 2026 vibecodingtoolmaker import ast +import gc import json import os import tempfile @@ -33,7 +34,14 @@ def load_methods(*method_names): raise AssertionError(f"Methods not found: {sorted(missing)}") module = ast.fix_missing_locations(ast.Module(body=functions, type_ignores=[])) - namespace = {"Path": Path, "json": json, "os": os, "print": print} + namespace = { + "Path": Path, + "json": json, + "os": os, + "print": print, + "MIB": 1024 ** 2, + "gc": gc, + } exec(compile(module, str(SOURCE_PATH), "exec"), namespace) return {name: namespace[name] for name in requested} @@ -43,6 +51,9 @@ class RecoveryHarness: METHODS = load_methods( + "_maximum_total_cases", + "_estimate_page_peak", + "_build_matrix_pages", "_matrix_only_cleanup_decision", "_safe_unlink", "_write_manifest", @@ -54,6 +65,14 @@ class RecoveryHarness: RecoveryHarness._safe_unlink = staticmethod(METHODS["_safe_unlink"]) RecoveryHarness._write_manifest = staticmethod(METHODS["_write_manifest"]) RecoveryHarness._cleanup_spool_files = METHODS["_cleanup_spool_files"] +RecoveryHarness._maximum_total_cases = classmethod(METHODS["_maximum_total_cases"]) +RecoveryHarness._estimate_page_peak = METHODS["_estimate_page_peak"] +RecoveryHarness._build_matrix_pages = METHODS["_build_matrix_pages"] +RecoveryHarness._effective_available = staticmethod(lambda state: 0) +RecoveryHarness.MAX_TOTAL_CASES = 500 +RecoveryHarness.MAX_EXTREME_TOTAL_CASES = 10_000 +RecoveryHarness.MAX_MATRIX_DIMENSION = 65_000 +RecoveryHarness.PAGE_SAFETY_FACTOR = 1.35 class MatrixOnlyRecoveryTests(unittest.TestCase): @@ -111,6 +130,8 @@ def test_complete_run_deletes_matrix_only_sources(self): session.mkdir() source = session / "cell-0000.png" source.write_bytes(b"source") + reference_row = session / "reference-row-0000.png" + reference_row.write_bytes(b"reference") manifest = session / "manifest.json" manifest.write_text("{}", encoding="utf-8") state = { @@ -119,6 +140,7 @@ def test_complete_run_deletes_matrix_only_sources(self): "cases": [{}], "cells": {0: {"path": str(source), "label": "first"}}, "rows": [], + "reference_row": {"path": str(reference_row)}, "keep_individual_images": False, "matrix_only_delete_allowed": True, } @@ -126,9 +148,92 @@ def test_complete_run_deletes_matrix_only_sources(self): self.harness._cleanup_spool_files(state, []) self.assertFalse(source.exists()) + self.assertFalse(reference_row.exists()) self.assertFalse(manifest.exists()) self.assertFalse(session.exists()) +class ExtremeRunAndMatrixLayoutTests(unittest.TestCase): + def setUp(self): + self.harness = RecoveryHarness() + + def test_extreme_run_is_an_explicit_ten_thousand_cell_opt_in(self): + self.assertEqual(self.harness._maximum_total_cases(False), 500) + self.assertEqual(self.harness._maximum_total_cases(True), 10_000) + + def test_reference_row_is_included_in_every_page_estimate(self): + state = {"matrix_margin": 5, "maximum_row_peak_bytes": 0} + reference_row = {"width": 832, "height": 1_300} + rows = [ + {"width": 3_328, "height": 1_400}, + {"width": 3_328, "height": 1_400}, + ] + + width, height, estimated_peak = self.harness._estimate_page_peak( + state, + rows, + reference_row, + ) + + self.assertEqual(width, 3_328) + self.assertEqual(height, 1_300 + 1_400 + 1_400 + 10) + self.assertGreater(estimated_peak, 0) + + def test_sixty_five_thousand_pixel_page_limit_is_enforced(self): + state = {"matrix_margin": 5, "maximum_row_peak_bytes": 0} + reference_row = {"width": 832, "height": 1_300} + rows = [{"width": 3_328, "height": 64_000}] + + _, height, estimated_peak = self.harness._estimate_page_peak( + state, + rows, + reference_row, + ) + + self.assertGreater(height, self.harness.MAX_MATRIX_DIMENSION) + self.assertEqual(estimated_peak, float("inf")) + + def test_reference_row_is_passed_to_every_composed_page(self): + state = { + "matrix_margin": 0, + "maximum_row_peak_bytes": 0, + "adaptive_ram": False, + "minimum_free_bytes": 0, + } + reference_row = {"path": "reference.png", "width": 832, "height": 1_000} + rows = [ + {"path": f"row-{index}.png", "width": 3_328, "height": 30_000} + for index in range(5) + ] + composed_references = [] + + def compose_page( + state, + rows, + width, + height, + page_number, + p, + processed, + grid_info, + reference_row=None, + ): + composed_references.append(reference_row) + return f"matrix-page-{page_number}.png" + + self.harness._compose_matrix_page = compose_page + page_paths = self.harness._build_matrix_pages( + state, + rows, + None, + None, + "test", + reference_row, + ) + + self.assertEqual(len(page_paths), 3) + self.assertEqual(composed_references, [reference_row] * 3) + + if __name__ == "__main__": unittest.main() From 2bb1bee553a0f4216518887bbefee4c121f42d7a Mon Sep 17 00:00:00 2001 From: vibecodingtoolmaker <312078684+vibecodingtoolmaker@users.noreply.github.com> Date: Wed, 5 Aug 2026 09:10:19 +0200 Subject: [PATCH 6/7] Prepare v0.2.0-beta.1 release --- CHANGELOG.md | 5 ++++- README.md | 17 +++++++++-------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7391cdd..8e6b25e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ## [Unreleased] +## [0.2.0-beta.1] - 2026-08-05 + ### Changed - Enabled the fixed-seed reference image by default and repeat it on every matrix page. @@ -41,5 +43,6 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) - Optional checkpoint, text-encoder, and VAE unload before matrix creation. - Recovery-safe matrix-only output mode. -[Unreleased]: https://github.com/vibecodingtoolmaker/Forge-Neo-Lora-Tester/compare/v0.1.0-beta.1...HEAD +[Unreleased]: https://github.com/vibecodingtoolmaker/Forge-Neo-Lora-Tester/compare/v0.2.0-beta.1...HEAD +[0.2.0-beta.1]: https://github.com/vibecodingtoolmaker/Forge-Neo-Lora-Tester/compare/v0.1.0-beta.1...v0.2.0-beta.1 [0.1.0-beta.1]: https://github.com/vibecodingtoolmaker/Forge-Neo-Lora-Tester/releases/tag/v0.1.0-beta.1 diff --git a/README.md b/README.md index eeab9fb..781996c 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ LoRA Tester compares multiple LoRAs and weight ranges with one prompt and one fixed seed. It returns the results as one or more labeled, RAM-aware matrix pages. -> **Project status:** v0.1.0-beta.1 public beta. The extension has been tested with -> large models and LoRA collections, but known limitations remain. Keep recoverable -> individual images enabled for important runs and please report reproducible issues. +> **Project status:** v0.2.0-beta.1 public beta. Extreme Run Mode completed a +> 2,743-cell stress test covering 211 LoRAs with 13 weights each. Known limitations +> remain; keep recoverable individual images enabled for important runs and please +> report reproducible issues. ## Welcome @@ -53,7 +54,7 @@ affiliated with or endorsed by OpenAI. - A Forge environment that provides Gradio and Pillow The extension does not install or download packages by itself. The optional -`psutil`-based RAM watchdog is disabled for the v0.1.0 release. +`psutil`-based RAM watchdog is disabled for the v0.2.0 release. ## Installation @@ -153,9 +154,9 @@ per-iteration random-number generator before every cell so that all LoRA and wei combinations use the seed of the first generated image. This preserves a meaningful visual comparison. -## Adaptive RAM protection (disabled in v0.1.0) +## Adaptive RAM protection (disabled in v0.2.0) -The adaptive RAM watchdog is intentionally disabled for the v0.1.0 release while +The adaptive RAM watchdog is intentionally disabled for the v0.2.0 release while its cross-system thresholds are evaluated further. Its checkbox and budget fields are visible but read-only. No RAM-monitor thread is started, and the extension does not stop between LoRA cells based on physical-memory or Windows-commit thresholds. @@ -168,7 +169,7 @@ The configured matrix column count remains the width of a logical row, and every labeled row is saved separately before the final page is composed. The maximum safe image-dimension fallback also remains active. -For v0.1.0, the primary memory-saving mechanism is the enabled-by-default model +For v0.2.0, the primary memory-saving mechanism is the enabled-by-default model unload described below. Users should still avoid matrix dimensions that exceed the practical RAM and image-size limits of their system. @@ -244,7 +245,7 @@ returned as recovery output and the recovery location is logged. fields. Incomplete rows block generation and identify the affected LoRA in the UI and console instead of silently using another value. -## Known limitations in v0.1.0-beta.1 +## Known limitations in v0.2.0-beta.1 - Adaptive RAM protection is visible but intentionally disabled while universal thresholds are evaluated. Disk spooling and optional model unloading remain active. From 3071b6593231fa93044b729f9552a2f8bd9ba087 Mon Sep 17 00:00:00 2001 From: vibecodingtoolmaker <312078684+vibecodingtoolmaker@users.noreply.github.com> Date: Wed, 5 Aug 2026 15:48:30 +0200 Subject: [PATCH 7/7] docs: add development handoff guidance --- AGENTS.md | 91 ++++++++++++++ PROJECT_STATUS.md | 315 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 406 insertions(+) create mode 100644 AGENTS.md create mode 100644 PROJECT_STATUS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..015aa07 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,91 @@ +# AGENTS.md + +These instructions apply to the entire Forge Neo LoRA Tester repository. + +## Start here + +- Read `PROJECT_STATUS.md`, `README.md`, and `CHANGELOG.md` before changing code. +- Check `git status --short --branch`, `git branch -vv`, and the recent log. +- Work only inside this extension repository unless the user explicitly expands + scope. The surrounding Forge checkout contains user models, outputs, settings, + and unrelated local changes that must be preserved. +- The normal development branch is `develop`. Leave the development installation + on `develop` after release work. + +## Branch and release policy + +- Implement and test changes on `develop` first. +- `develop` contains development-only tests and handoff documentation. +- `main` is the minimal public release package. Do not add `tests/`, + `PROJECT_STATUS.md`, or `AGENTS.md` to `main` unless the maintainer explicitly + changes this policy. +- `develop` and `main` intentionally use parallel/cherry-picked commits. Do not + assume they can be fast-forwarded or blindly merged. +- For a release, synchronize both branches with their remote tips, commit/test on + `develop`, then cherry-pick only public product/documentation changes to `main`. + Resolve the development-test path as absent on `main`. +- Verify that `main:scripts/lora_tester.py` and + `develop:scripts/lora_tester.py` have identical blob hashes before tagging. +- Tags and GitHub releases must point to `main`, not `develop`. +- Beta versions are GitHub pre-releases. Update the README status, dated changelog + section, comparison links, and release notes together. +- Do not push branches, create tags, or publish/edit GitHub releases without explicit + user authorization for that release action. + +## Safety and architecture invariants + +- Do not introduce hard-coded installation, model, output, user-profile, or drive + paths. Derive paths from Forge processing objects, `shared.cmd_opts`, + `shared.models_path`, or the extension/repository location. +- Do not add runtime network requests, telemetry, downloaded code execution, or + automatic dependency installation. Metadata files are untrusted data. +- Preserve fixed-seed comparison: every baseline/LoRA/weight cell must use the first + resolved seed, including when the user enters seed `-1`. +- Preserve disk-first processing. Completed full-size cells must be atomically spooled + and large Forge/PIL result references released promptly. +- Preserve Matrix-only recovery safety. Never delete source images unless all + requested cells completed without interruption and every matrix page was written + and validated. On ambiguity, retain sources and the recovery manifest. +- Preserve automatic multi-page composition and the 65,000-pixel per-axis ceiling + unless encoder/gallery testing justifies a deliberate change. +- Normal runs are limited to 500 cells. The explicit Extreme Run Mode raises the + fixed limit to 10,000; never silently truncate an over-limit request. +- Reference generation is enabled by default. When present, render the reference as + a separate row on every successful matrix page without reordering comparison cells. +- Keep checkpoint/text-encoder/VAE unload enabled by default. Use Forge's full + checkpoint unload path rather than a VRAM-only eviction. +- `RAM_WATCHDOG_ENABLED` must remain `False` until the maintainer explicitly approves + reactivation after cross-system validation. +- The existing startup LoRA warning is an open eager-import bug. Fix it by lazy + resolution at unload time; do not replace it with a new startup LoRA scan. + +## Code and licensing + +- Keep SPDX header `AGPL-3.0-only` and copyright + `Copyright (C) 2026 vibecodingtoolmaker` in source/test files. +- Maintain compatibility with Forge Neo's public script/processing APIs and the + installed Gradio/Pillow versions. Avoid adding dependencies when Forge or the + Python standard library already provides the needed behavior. +- Preserve user changes and unrelated dirty files. Do not use destructive Git + commands such as `git reset --hard` or discard edits without explicit approval. +- Use atomic writes for images/manifests and validate paths remain within + Forge-derived output boundaries. + +## Required verification + +- Run the development tests with Forge's Python environment: + +From the Forge root, run: + +```powershell +& '.\venv\Scripts\python.exe' -m unittest discover ` + -s '.\extensions\Forge-Neo-Lora-Tester\tests' -v +``` + +- Run a syntax compilation with the same interpreter for + `scripts/lora_tester.py`. +- Run `git diff --check` and inspect the complete relevant diff. +- For matrix/recovery changes, add or update a unit test on `develop`. +- Automated tests do not replace manual Forge tests. For behavior changes, report the + exact txt2img/img2img, baseline, output-retention, model-unload, interruption, and + multi-page cases that still require manual testing. diff --git a/PROJECT_STATUS.md b/PROJECT_STATUS.md new file mode 100644 index 0000000..27e8df9 --- /dev/null +++ b/PROJECT_STATUS.md @@ -0,0 +1,315 @@ +# Forge Neo LoRA Tester – Project Status + +Last updated: 2026-08-05 + +Repository: https://github.com/vibecodingtoolmaker/Forge-Neo-Lora-Tester + +Local development checkout: `H:\DEVELOPMENT-sd-webui-forge-neo\extensions\Forge-Neo-Lora-Tester` + +This file is the handoff document for a new Codex chat. Read `AGENTS.md` first, +then this document, `README.md`, and the current Git diff before changing code. + +## Current state + +- Current public release: **v0.2.0-beta.1** (GitHub pre-release, 2026-08-05). +- Release tag: `v0.2.0-beta.1` -> `main` commit `14c6dae`. +- Release feature commit on `main`: `c27c621`. +- Current `develop` release commit: `2bb1bee`. +- Current `develop` feature commit: `af2432a`. +- The development installation must normally remain on branch `develop`. +- At the start of this handoff update, `develop` was clean and synchronized with + `origin/develop`. Re-check `git status --short --branch`; the new handoff files + may still be local and uncommitted. +- `main` intentionally contains only the public extension package. The test suite + is intentionally retained on `develop` and omitted from `main`. +- License: `AGPL-3.0-only`; copyright holder: `vibecodingtoolmaker`. + +## Proven working functionality + +- txt2img and img2img integration through Forge's always-on script lifecycle. +- The extension fully switches off when its accordion/checkbox is disabled. +- Folder-scoped LoRA inventory with optional recursive subfolder inclusion. +- First activation and manual refresh populate the LoRA list and editable table. +- Trigger words from `.civitai.info` and `.json`, plus per-LoRA manual overrides. +- Positive, negative, ascending, and descending weight ranges. +- One fixed seed for the baseline and every LoRA/weight cell, including seed `-1` + after the first random seed has been resolved. +- Disk spooling of every completed source cell; decoded full-size images are not + intentionally retained in Forge's result list. +- Matrix-only cleanup is recovery-safe: source cells are deleted only after a + complete run and successful validation of every matrix page. +- Interrupted runs create partial matrices and retain recoverable source images. +- Optional checkpoint, text encoder, and VAE unload before matrix composition; + enabled by default and the main practical RAM-saving mechanism. +- Multiple matrix pages with compact labels and automatic page packing. +- Individual images are retained in dedicated txt2img/img2img run directories when + the safe output-retention mode is selected. + +## Changes implemented on 2026-08-05 + +### Extreme Run Mode + +- Added a small opt-in **Extreme Run Mode** checkbox under Advanced Options. +- Normal limit remains 500 total matrix cells. +- Extreme limit is fixed at 10,000 total matrix cells. +- The generated baseline/reference counts as one matrix cell. +- Runs above the selected limit are visibly blocked; they are no longer silently + truncated after 500 cells. +- The UI warns that extreme runs may take hours or days, use large amounts of disk + space, and expose long-duration Forge/driver/model/extension/system instability. + +### Matrix dimensions and repeated reference + +- Increased `MAX_MATRIX_DIMENSION` from 60,000 to **65,000 pixels** per axis. +- Automatic page splitting remains mandatory. The 65,000 limit is independent of + available RAM/VRAM and stays below common roughly 65.5k encoder boundaries. +- Reference generation is now enabled by default; the user must actively disable it. +- The reference is removed from the normal LoRA-cell sequence and rendered as a + dedicated, centered reference row at the top of every successful matrix page. + This avoids duplicating or reordering LoRA results. +- Page-size/RAM estimation includes the repeated reference row. +- Reference row files participate in cleanup, manifests, and recovery output. + +### Validation and release + +- Real-world stress test completed successfully: + - 211 LoRAs + - 13 weights per LoRA (`-3` through `3` in `0.5` steps) + - 2,743 total LoRA cells reported by the user + - 832 x 1216 pixels per generated image + - total runtime: 4 h 24 min +- Seven `unittest` tests pass with Forge's Python environment. +- Forge-Python syntax compilation passes. +- `main`, `develop`, and tag `v0.2.0-beta.1` were pushed atomically. +- GitHub pre-release published with stress-test and upgrade notes. + +## Important technical decisions and invariants + +### Fixed seed + +`_force_batch_rng_seed()` rebuilds Forge's per-iteration RNG state so each cell +uses the first resolved seed instead of Forge's normal `seed + iteration` behavior. +Do not remove this reset when refactoring generation hooks. + +### Disk-first processing + +`postprocess_image_after_composite()` atomically saves each completed cell and +replaces large Forge/PIL references with a 1x1 placeholder. Matrix creation later +opens one source cell/row at a time. Disk usage may be large, but generation RAM +must not scale linearly with the number of cells. + +### Recovery-safe Matrix-only mode + +`_matrix_only_cleanup_decision()`, `_validate_matrix_pages()`, and +`_cleanup_spool_files()` form a safety boundary. Individual source files may be +permanently deleted only if all requested cells completed, generation was not +interrupted, and every matrix page exists and validates. On uncertainty or failure, +retain sources plus the manifest. + +### Page splitting + +`_estimate_page_peak()` rejects pages exceeding 65,000 pixels on either axis. +`_build_matrix_pages()` adds rows until the next row would cross that dimension or, +if the experimental RAM system is later enabled, the calculated memory budget. +Splitting is expected behavior and not an OOM indication. + +### Repeated reference + +The baseline cell is converted to a one-column `reference-row` strip. It is passed +separately to every `_compose_matrix_page()` call, while normal LoRA rows keep the +configured column count. If reference generation is disabled, no reference row is +created or repeated. + +### Model unload + +`_unload_forge_model_for_matrix()` calls Forge's checkpoint unload path and clears +prompt/sampler references before allocating matrix canvases. A VRAM-only eviction +is insufficient because it may move weights into system RAM. Keep this option +enabled by default unless a regression is proven. + +### RAM watchdog + +The adaptive RAM implementation remains in the source, but +`RAM_WATCHDOG_ENABLED = False`. Its UI is visible but read-only. Do not re-enable it +without explicit cross-system testing and a deliberate release decision. Disk +spooling, model unload, dimension limits, and multi-page composition remain active. + +### Branch and release layout + +`develop` and `main` have intentionally parallel/cherry-picked histories rather than +a simple fast-forward relationship. Do not blindly merge one into the other. +Production code blobs must match, but `main` omits development-only tests and handoff +files. See `AGENTS.md` for the release procedure. + +## Known bugs and open problems + +### High priority + +1. **Misleading startup warning / eager Forge-LoRA import** + - Startup can log `[LoRA Tester] Warning: LoRA system not found`. + - Cause: module-level import of `extensions_builtin.sd_forge_lora.networks` + occurs before Forge has fully installed the built-in LoRA directory on the + Python import path. This does not perform the filesystem LoRA scan. + - Consequence: `LORA_AVAILABLE` remains false for the session, so the optional + `networks.loaded_networks.clear()` step is skipped during model unload. + - Intended fix: remove the eager import and resolve Forge's already-loaded + top-level `networks` module lazily inside `_unload_forge_model_for_matrix()`. + Validate expected attributes before clearing; do not add a new startup scan. + +2. **Adaptive RAM watchdog is disabled** + - Threshold behavior was too hardware-specific and caused premature stops after + initial model/LoRA loading spikes. + - The 2,743-cell run succeeded with the watchdog disabled and model unload enabled. + - A universal design must account for Windows commit/pagefile headroom, persistent + model allocation, temporary loading spikes, and page-composition memory. + +### Accepted beta limitations / backlog + +3. **Matrix margin inconsistency** + - Vertical row spacing works. Horizontal cell spacing currently applies only when + `Draw Legend in Matrix Grid` is enabled. + +4. **Grid date folders** + - When model unload is enabled, saved grids use Forge's grid output root instead + of Forge's configured date-subfolder layout. + +5. **First Generate click may only commit a corrected table cell** + - After correcting an invalid active Dataframe cell, the first Generate click can + return focus to the table without starting. The second click starts normally. + - This is a Gradio table-submit/focus issue; no generated data is lost. + +6. **Cold-start invalid-input progress behavior** + - On a cold model start, an invalid table value was once shown in the progress + area and followed by delayed cleanup roughly 1–2 minutes later. Warm starts only + showed the trace warning. Observe and reproduce before changing processing flow. + +7. **No third trigger position** + - Existing `Start` means before the user's prompt; `End` means after the injected + LoRA tag. Backlog request: add `Before LoRA` to place trigger text immediately + before `` while leaving the user's prompt first. + +8. **Settings are not persisted** + - Static UI settings reset between sessions. LoRA inventory, selected LoRAs, + trigger values, and table rows are dynamic and should not be persisted blindly. + - Backlog: persist safe static txt2img/img2img settings separately. + +9. **Extreme one-row fallback edge case** + - If one prebuilt row alone exceeds the 65,000-pixel limit, or if a future enabled + RAM budget rejects even one row, the existing disk fallback returns that row + directly. In that rare path the repeated reference is not composed into it. + - Typical tested sizes do not hit this path. A future fix should reflow the source + cells into fewer columns rather than returning an oversized row. + +10. **No resume-after-restart workflow** + - Manifests and sources preserve recovery data after interruption, but there is no + UI to resume a multi-hour run after Forge or Windows restarts. + +11. **Individual-image finalization can be optimized** + - Retained images are first spooled under `tmp` and moved to the final run folder + after generation. A future implementation could spool directly into the final + folder while preserving atomic writes and recovery semantics. + +## Not yet sufficiently tested + +- A full 10,000-cell run. Current maximum validated run: 2,743 cells. +- Exact 65,000-pixel output with every Forge grid format, especially JPEG, and its + display in Gradio/browser galleries. +- Manual visual confirmation that the reference row appears correctly on every page + for multiple page counts, legends on/off, baseline on/off, and partial runs. +- Extreme multi-page img2img runs. +- Extreme runs with Matrix-only cleanup after full successful completion. +- Disk-full, permission-loss, path-length, antivirus-lock, and atomic-rename failures. +- Very wide rows: 10 columns with unusually large source images. +- Mixed or unexpected source image dimensions within one run. +- Low-RAM machines and systems with small/disabled Windows pagefiles. +- Long-run manual cancellation at several phases: generation, row-strip creation, + page composition, image finalization, and cleanup. +- Upgrade testing from both `v0.1.0-beta.1` and a fresh clone of `v0.2.0-beta.1`. +- Compatibility with future Forge Neo/Gradio/Pillow changes. + +## Recommended next steps + +1. Fix the eager LoRA-system import and eliminate the misleading startup warning. +2. Run a short Forge smoke test after that fix: normal generation with the tester + disabled, one baseline + two LoRAs, model unload on/off, txt2img and img2img. +3. Visually verify repeated references on at least a two-page matrix with legends + both enabled and disabled. +4. Add tests for the visible over-limit block in `before_process()`, not only the + pure 500/10,000 limit helper. +5. Test 65,000-pixel page encoding in every supported `grid_format`; reduce the + ceiling if any encoder or gallery rejects it. +6. Fix the margin/date-folder beta limitations. +7. Design static settings persistence without persisting dynamic LoRA selections. +8. Add optional preflight information (planned cells, approximate rows/pages, and a + conservative disk/time warning) without adding another user-editable cell limit. +9. Revisit adaptive RAM protection only after gathering measurements from different + RAM sizes, model families, pagefiles, and loading strategies. +10. Consider manifest-based resume support for multi-hour Extreme Run jobs. + +## Relevant files and functions + +### `scripts/lora_tester.py` + +- `RamMonitor`: dormant peak sampler used only if the release switch is enabled. +- `LoRaMetadataReader.get_lora_dir()`: derives Forge's configured LoRA directory. +- `LoRaMetadataReader.find_all_loras()`: recursive model and metadata inventory. +- `LoRaTesterScript.ui()`: all txt2img/img2img controls and callbacks. +- `_parse_weight_spec()`: finite single/range parser, maximum 100 weights per LoRA. +- `_maximum_total_cases()`: chooses 500 normal or 10,000 Extreme limit. +- `_compose_lora_prompt()`: Start/End trigger placement and LoRA injection. +- `before_process()`: validates UI data, expands cases, creates run state/manifest. +- `_force_batch_rng_seed()`: enforces the shared resolved seed for every cell. +- `before_process_batch()`: applies per-cell prompts/LoRA tags and RAM-stop logic. +- `postprocess_image_after_composite()`: atomic per-cell disk spool and RAM release. +- `_unload_forge_model_for_matrix()`: checkpoint/text encoder/VAE unload. +- `_finalize_individual_images()`: moves retained source PNGs to final run folders. +- `_write_manifest()`: persistent recovery state. +- `_build_row_strips()`: creates labeled comparison/reference row PNGs on disk. +- `_estimate_page_peak()`: dimension and conservative memory estimate. +- `_build_matrix_pages()`: page packing and repeated-reference propagation. +- `_compose_matrix_page()`: creates/saves one final page. +- `_cleanup_spool_files()`: recovery-safe source/row/manifest cleanup. + +### `javascript/lora_tester_dataframe.js` + +- Makes editable Dataframe cells show a caret on one click. +- Places the caret at the start without selecting/deleting the cell value. +- Protects the read-only LoRA-name column. +- Handles Tab/Shift+Tab/Enter/arrow navigation and avoids focus reclaim loops. + +### `tests/test_lora_tester_recovery.py` (`develop` only) + +- Matrix-only cleanup/recovery decisions. +- Partial-run source retention and manifest preservation. +- Complete-run cleanup including the temporary reference row. +- Normal/Extreme cell limits. +- Reference-row inclusion in page estimates and every composed page. +- 65,000-pixel page boundary. + +### Documentation and metadata + +- `README.md`: public user documentation and known limitations. +- `CHANGELOG.md`: release history; current empty `[Unreleased]` section. +- `metadata.ini`: Forge extension description and AGPL identifier. +- `LICENSE`: complete AGPL-3.0-only license text. +- `AGENTS.md`: durable development/release/safety rules for future Codex chats. + +## Verification commands + +Run from `H:\DEVELOPMENT-sd-webui-forge-neo`: + +```powershell +& '.\venv\Scripts\python.exe' -m unittest discover ` + -s '.\extensions\Forge-Neo-Lora-Tester\tests' -v +``` + +Run from the extension repository: + +```powershell +git status --short --branch +git diff --check +git branch -vv +``` + +The automated tests intentionally avoid importing a live Forge processing stack. +They do not replace manual Forge UI/generation tests.