⚡ Bolt: Parallelize existing rules fetch with folder deletion by abhimehro · Pull Request #410 · abhimehro/ctrld-sync

abhimehro · 2026-02-20T14:45:07Z

💡 What: Modified sync_profile to initiate get_all_existing_rules (for kept folders) in parallel with the folder deletion phase.
🎯 Why: The folder deletion phase (and subsequent 60s wait) blocks the main thread. Fetching existing rules is an independent I/O-bound operation that can run concurrently, effectively hiding its latency.
📊 Impact: Reduces sync time by overlapping fetch latency with deletion/wait time. For scenarios with deletions, this can save ~seconds to minutes depending on profile size.
🔬 Measurement: Verified with tests/test_parallel_fetch.py showing reduction from ~2s to ~1s in simulated conditions.

PR created automatically by Jules for task 1449692924143549150 started by @abhimehro

Previously, `sync_profile` would delete folders, wait for propagation (up to 60s), and ONLY THEN fetch existing rules from remaining folders. This caused serial execution of two independent I/O-bound operations. This change moves the `get_all_existing_rules` call to run in the background (using `shared_executor`) while the main thread handles folder deletions and the propagation wait. This effectively overlaps the latency of fetching rules with the deletion/wait time. Key changes: - Calculate `folders_to_delete` and `folders_to_scan` (kept folders) upfront. - Offload `get_all_existing_rules` to `shared_executor` before starting deletions. - Retrieve the result of the fetch after the deletion phase completes. - Handles `no_delete` flag correctly (scans all folders). Impact: - Significantly reduces end-to-end sync time by hiding the rule fetch latency behind the deletion wait. - Verified with `tests/test_parallel_fetch.py` (simulated ~50% reduction in fetch+delete phase). Co-authored-by: abhimehro <84992105+abhimehro@users.noreply.github.com>

google-labs-jules · 2026-02-20T14:45:08Z

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.

For security, I will only act on instructions from the user who triggered this task.

trunk-io · 2026-02-20T14:45:12Z

😎 Merged manually by @abhimehro - details.

github-actions · 2026-02-20T14:45:19Z

👋 Development Partner is reviewing this PR. Will provide feedback shortly.

gemini-code-assist · 2026-02-20T14:45:25Z

Summary of Changes

Hello @abhimehro, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant performance optimization to the sync_profile function by parallelizing the retrieval of existing rules with the folder deletion process. By identifying folders to be kept and initiating their rule fetching in a background thread while deletions occur, the system effectively hides latency, leading to faster synchronization times, especially in scenarios involving folder deletions.

Highlights

Parallel Rule Fetching: The sync_profile function now initiates fetching of existing rules for kept folders in parallel with the folder deletion phase.
Performance Improvement: This change significantly reduces sync time by overlapping the I/O-bound rule fetching latency with the deletion and wait times, potentially saving seconds to minutes.
New Test Case: A dedicated test (test_parallel_fetch.py) has been added to verify the effectiveness and correct parallel execution of this optimization.

Changelog

main.py
- Refactored _fetch_if_valid to separate folders into those to be deleted and those to be scanned for rules.
- Introduced shared_executor.submit to run get_all_existing_rules concurrently for folders that are not being deleted.
- Replaced the synchronous call to get_all_existing_rules with a call to existing_rules_future.result() to retrieve the background task's outcome, including error handling.
test_main.py
- Updated mock arguments in existing tests to include mock_args.clear_cache = False.
tests/test_parallel_fetch.py
- Added a new test file containing TestParallelFetch to specifically validate the parallel execution logic.
- Implemented mocks to simulate slow get_all_existing_rules and delete_folder operations and assert that the total execution time is reduced due to parallelization.

Activity

This pull request was automatically created by Jules for task 1449692924143549150, initiated by @abhimehro.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request successfully implements parallel fetching of existing rules concurrently with folder deletion, which is a significant performance improvement. The new test test_parallel_fetch.py effectively demonstrates this optimization. The changes are well-explained and address a clear performance bottleneck. I've identified a few areas for minor improvement related to clarity and error handling, but overall, the changes are positive.

gemini-code-assist · 2026-02-20T14:46:47Z

main.py

+            # Identify folders to delete and folders to keep (scan)
+            folders_to_delete = []
+            folders_to_scan = existing_folders.copy()

-                # Identify folders to delete
-                folders_to_delete = []
+            if not no_delete:
                for folder_data in folder_data_list:
                    name = folder_data["group"]["group"].strip()
                    if name in existing_folders:
                        folders_to_delete.append((name, existing_folders[name]))
+                        if name in folders_to_scan:
+                            del folders_to_scan[name]


The logic for folders_to_delete and folders_to_scan is duplicated inside and outside the if not no_delete: block. This can lead to confusion and potential bugs if no_delete is true, as folders_to_delete would be empty, but folders_to_scan would still be a copy of existing_folders and not reflect what's actually being scanned. The folders_to_scan should only be modified if no_delete is false.

Suggested change

# Identify folders to delete and folders to keep (scan)

folders_to_delete = []

folders_to_scan = existing_folders.copy()

# Identify folders to delete

folders_to_delete = []

if not no_delete:

for folder_data in folder_data_list:

name = folder_data["group"]["group"].strip()

if name in existing_folders:

folders_to_delete.append((name, existing_folders[name]))

if name in folders_to_scan:

del folders_to_scan[name]

folders_to_delete = []

folders_to_scan = existing_folders.copy()

if not no_delete:

for folder_data in folder_data_list:

name = folder_data["group"]["group"].strip()

if name in existing_folders:

folders_to_delete.append((name, existing_folders[name]))

if name in folders_to_scan:

del folders_to_scan[name]

gemini-code-assist · 2026-02-20T14:46:47Z

main.py

+            except Exception as e:
+                log.error(f"Failed to fetch existing rules in background: {sanitize_for_log(e)}")


The log.error message here is generic. It would be more helpful to include the profile_id in the error message to provide better context for debugging.

Suggested change

except Exception as e:

log.error(f"Failed to fetch existing rules in background: {sanitize_for_log(e)}")

except Exception as e:

log.error(f"Failed to fetch existing rules for profile {profile_id} in background: {sanitize_for_log(e)}")

gemini-code-assist · 2026-02-20T14:46:47Z

tests/test_parallel_fetch.py

+import os
+
+# Add root to path
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))


The sys.path.append is generally discouraged in tests as it can lead to unexpected module resolution issues. It's better to use relative imports or ensure the test runner correctly sets up the Python path.

tests/test_parallel_fetch.py

main.py

tests/test_parallel_fetch.py

+import unittest
+from unittest.mock import MagicMock, patch
+import time
+import concurrent.futures


tests/test_parallel_fetch.py

+
+# Add root to path
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import main


tests/test_parallel_fetch.py

+    @patch("main.fetch_folder_data")
+    @patch("main.validate_folder_url")
+    @patch("main.countdown_timer")
+    def test_parallel_execution(self, mock_timer, mock_validate, mock_fetch, mock_verify, mock_delete, mock_get_rules):


tests/test_parallel_fetch.py

+    @patch("main.fetch_folder_data")
+    @patch("main.validate_folder_url")
+    @patch("main.countdown_timer")
+    def test_parallel_execution(self, mock_timer, mock_validate, mock_fetch, mock_verify, mock_delete, mock_get_rules):


tests/test_parallel_fetch.py

+        }
+
+        # Latency simulation
+        def slow_get_rules(*args, **kwargs):


tests/test_parallel_fetch.py

+        }
+
+        # Latency simulation
+        def slow_get_rules(*args, **kwargs):


tests/test_parallel_fetch.py

+            time.sleep(1.0)
+            return {"rule_from_keep"}
+
+        def slow_delete(*args, **kwargs):


tests/test_parallel_fetch.py

+            time.sleep(1.0)
+            return {"rule_from_keep"}
+
+        def slow_delete(*args, **kwargs):


tests/test_parallel_fetch.py

+            known_folders = call_args[0][2] if len(call_args[0]) > 2 else call_args[1]['known_folders']
+
+            self.assertIn("keep_folder", known_folders)
+            self.assertNotIn("test_folder", known_folders, "Should not fetch rules from deleted folder")


tests/test_parallel_fetch.py

+            # Verify get_all_existing_rules was called with ONLY keep_folder
+            call_args = mock_get_rules.call_args
+            # args: client, profile_id, known_folders
+            known_folders = call_args[0][2] if len(call_args[0]) > 2 else call_args[1]['known_folders']


tests/test_parallel_fetch.py

+            time.sleep(1.0)
+            return {"rule_from_keep"}
+
+        def slow_delete(*args, **kwargs):


tests/test_parallel_fetch.py

+        }
+
+        # Latency simulation
+        def slow_get_rules(*args, **kwargs):


tests/test_parallel_fetch.py

+            time.sleep(1.0)
+            return {"rule_from_keep"}
+
+        def slow_delete(*args, **kwargs):


tests/test_parallel_fetch.py

+            time.sleep(1.0)
+            return {"rule_from_keep"}
+
+        def slow_delete(*args, **kwargs):


tests/test_parallel_fetch.py

+import unittest
+from unittest.mock import MagicMock, patch
+import time
+import concurrent.futures


main.py

tests/test_parallel_fetch.py

+            known_folders = call_args[0][2] if len(call_args[0]) > 2 else call_args[1]['known_folders']
+
+            self.assertIn("keep_folder", known_folders)
+            self.assertNotIn("test_folder", known_folders, "Should not fetch rules from deleted folder")


tests/test_parallel_fetch.py

+            # Verify get_all_existing_rules was called with ONLY keep_folder
+            call_args = mock_get_rules.call_args
+            # args: client, profile_id, known_folders
+            known_folders = call_args[0][2] if len(call_args[0]) > 2 else call_args[1]['known_folders']


tests/test_parallel_fetch.py

+    @patch("main.fetch_folder_data")
+    @patch("main.validate_folder_url")
+    @patch("main.countdown_timer")
+    def test_parallel_execution(self, mock_timer, mock_validate, mock_fetch, mock_verify, mock_delete, mock_get_rules):


tests/test_parallel_fetch.py

+        }
+
+        # Latency simulation
+        def slow_get_rules(*args, **kwargs):


tests/test_parallel_fetch.py

+        }
+
+        # Latency simulation
+        def slow_get_rules(*args, **kwargs):


tests/test_parallel_fetch.py

+            time.sleep(1.0)
+            return {"rule_from_keep"}
+
+        def slow_delete(*args, **kwargs):


tests/test_parallel_fetch.py

+            time.sleep(1.0)
+            return {"rule_from_keep"}
+
+        def slow_delete(*args, **kwargs):


main.py

Copilot

Pull request overview

This pull request aims to optimize the sync workflow by parallelizing the fetch of existing rules with the folder deletion phase. The change modifies sync_profile to start fetching rules from folders that will be kept while deletions are in progress, potentially hiding the fetch latency behind the 60-second deletion propagation wait.

Changes:

Modified main.py to submit get_all_existing_rules to a background executor while deletions proceed
Added test coverage in tests/test_parallel_fetch.py to verify the parallel execution
Updated test_main.py to include the new clear_cache argument in mock objects

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
main.py	Refactored sync_profile to identify folders to keep/delete upfront and start background fetch of existing rules while deletions run
tests/test_parallel_fetch.py	New test file with timing-based test to verify parallel execution of delete and fetch operations
test_main.py	Added clear_cache=False to mock_args objects to match updated parse_args signature

tests/test_parallel_fetch.py

Copilot · 2026-02-20T14:51:31Z

tests/test_parallel_fetch.py

+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import main
+
+class TestParallelFetch(unittest.TestCase):


Inconsistent Test Framework: This test file uses unittest.TestCase while the similar test file tests/test_parallel_deletion.py uses pytest. For consistency and to leverage pytest's better fixtures and parametrization, consider converting this test to use pytest instead of unittest. This would also make it easier to use monkeypatch and other pytest features.

main.py

Copilot · 2026-02-20T14:51:31Z

main.py

+            existing_rules_future = shared_executor.submit(
+                get_all_existing_rules, client, profile_id, folders_to_scan
+            )


Critical Thread Safety Issue: The httpx.Client object is not thread-safe when used from multiple threads simultaneously. Here, the client is submitted to shared_executor to run get_all_existing_rules, which internally creates another ThreadPoolExecutor (5 workers) that also uses the same client object. This means the same httpx.Client instance is being used from multiple threads concurrently, which can lead to race conditions, corrupted connection state, or crashes.

According to httpx documentation, Client objects should not be shared across threads without synchronization. The recommended approach is to either:

Create a new client within the background task, or

Use httpx.AsyncClient with asyncio for concurrent operations, or

Add proper locking around all client operations (not recommended due to performance impact)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-02-22T10:03:48Z

👋 Development Partner is reviewing this PR. Will provide feedback shortly.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-02-22T10:03:57Z

👋 Development Partner is reviewing this PR. Will provide feedback shortly.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-02-22T10:04:08Z

👋 Development Partner is reviewing this PR. Will provide feedback shortly.

github-actions · 2026-02-22T10:04:14Z

👋 Development Partner is reviewing this PR. Will provide feedback shortly.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

github-actions · 2026-02-22T10:04:45Z

👋 Development Partner is reviewing this PR. Will provide feedback shortly.

tests/test_parallel_fetch.py

+
+# Add root to path
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import main


tests/test_parallel_fetch.py

+
+# Add root to path
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import main


tests/test_parallel_fetch.py

…ization - Corrected a potential corruption/syntax error in `main.py` that caused CI failures. - Verified parallel fetch optimization (fetching existing rules while deleting folders) with `tests/test_parallel_fetch.py`. - Updated `test_main.py` to match latest `main.py` changes (arguments/execution flow). - Verified no regressions with `tests/test_performance_regression.py`. This commit ensures the codebase is in a consistent state and the optimization is correctly applied without syntax errors. Co-authored-by: abhimehro <84992105+abhimehro@users.noreply.github.com>

github-actions · 2026-02-22T10:10:18Z

👋 Development Partner is reviewing this PR. Will provide feedback shortly.

Copilot AI review requested due to automatic review settings February 20, 2026 14:45

github-actions bot added the python label Feb 20, 2026

Copilot started reviewing on behalf of abhimehro February 20, 2026 14:45 View session

gemini-code-assist bot reviewed Feb 20, 2026

View reviewed changes

github-advanced-security bot found potential problems Feb 20, 2026

View reviewed changes

Copilot AI reviewed Feb 20, 2026

View reviewed changes

Update test_parallel_fetch.py

d363bbb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

abhimehro and others added 2 commits February 22, 2026 04:03

Update test_parallel_fetch.py

966ee5f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update test_parallel_fetch.py

0af1b69

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update main.py

32ffa58

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update test_parallel_fetch.py

2fc4d54

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

github-advanced-security bot found potential problems Feb 22, 2026

View reviewed changes

tests/test_parallel_fetch.py Fixed Show fixed Hide fixed

tests/test_parallel_fetch.py Fixed Show fixed Hide fixed

abhimehro merged commit 6c28965 into main Feb 28, 2026
20 of 21 checks passed

abhimehro deleted the bolt/parallel-fetch-optimization-1449692924143549150 branch February 28, 2026 08:43

abhimehro mentioned this pull request Feb 28, 2026

⚡ Bolt: Optimize cache I/O and rule processing loop #413

Closed

		except Exception as e:
		log.error(f"Failed to fetch existing rules in background: {sanitize_for_log(e)}")

Conversation

abhimehro commented Feb 20, 2026

Uh oh!

google-labs-jules bot commented Feb 20, 2026

Uh oh!

trunk-io bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

gemini-code-assist bot commented Feb 20, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Uh oh!

Uh oh!

Check warning

Check warning

Check warning

Check notice

Check notice

Check notice

Check notice

Uh oh!

Uh oh!

Uh oh!

Check warning

Check warning

Check warning

Check notice

Check notice

Check notice

Check notice

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 22, 2026

Uh oh!

github-actions bot commented Feb 22, 2026

Uh oh!

github-actions bot commented Feb 22, 2026

Uh oh!

github-actions bot commented Feb 22, 2026

trunk-io bot commented Feb 20, 2026 •

edited

Loading