Skip to content

feat: Implement concurrent file summarization with ThreadPoolExecutor #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
agsaru merged 3 commits into from
Jul 7, 2026
+25 −7
Merged

feat: Implement concurrent file summarization with ThreadPoolExecutor #23

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
249731c
feat: Implement concurrent file summarization with ThreadPoolExecutor
mrunmayeekokitkar Jul 6, 2026
645d631
fix: Handle edge case when total_documents is 0
mrunmayeekokitkar Jul 6, 2026
a8517b5
reduced concurrent workers
agsaru Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions repo2readme/cli/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
import os
from repo2readme.utils.tree import generate_tree
from repo2readme.utils.detect_language import detect_lang
from concurrent.futures import ThreadPoolExecutor, as_completed
import threading


@click.group()
Expand Down Expand Up @@ -131,9 +133,13 @@ def format_size(size_bytes):

summaries = []
errors = []
with Progress() as progress:
task = progress.add_task("[cyan]Generating summaries...[/cyan]", total=total_documents)
for doc in documents:

# Skip summarization if there are no documents
if total_documents > 0:
summaries_lock = threading.Lock()
errors_lock = threading.Lock()

def process_document(doc):
meta = doc["metadata"]
try:
lang = detect_lang(meta.get("file_type", "text"))
Expand All @@ -142,11 +148,23 @@ def format_size(size_bytes):
language=lang,
content=doc["content"]
)

summaries.append(summary)
with summaries_lock:
summaries.append(summary)
except Exception as e:
errors.append(f"Error processing {meta.get('file_path')}: {e}")
progress.update(task, advance=1)
with errors_lock:
errors.append(f"Error processing {meta.get('file_path')}: {e}")

with Progress() as progress:
task = progress.add_task("[cyan]Generating summaries...[/cyan]", total=total_documents)

# Limit concurrent workers to avoid overwhelming API providers
max_workers = min(2, total_documents)

with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {executor.submit(process_document, doc): doc for doc in documents}

for future in as_completed(futures):
progress.update(task, advance=1)

rprint("[cyan]Generating README...[/cyan]")

Expand Down