feat: make indexing and compile concurrency configurable (#174)

* feat(indexer): expose pageindex_max_concurrency to cap indexing concurrency

PageIndex fans out one indexing LLM call per structure node; on a large
document that can open a socket per node and exhaust the process
file-descriptor limit ("[Errno 24] Too many open files"). This wires an
OpenKB config knob through to PageIndex's IndexConfig.max_concurrency cap.

- New `pageindex_max_concurrency` KB config key (default None = let PageIndex
  apply its own default).
- indexer forwards it via `_build_index_config` only when set AND the installed
  PageIndex's IndexConfig declares the field, so OpenKB keeps working against a
  pinned PageIndex that predates it (IndexConfig forbids unknown kwargs).

* feat(config): make compile concurrency configurable (closes #173)

Concept/entity generation ran at a hardcoded concurrency of 5
(DEFAULT_COMPILE_CONCURRENCY) with no way to lower it when the LLM provider
rate-limits. Add a `compile_concurrency` config key (default 5) and thread it
through every add / recompile / cloud-import compile call via a shared
`_compile_concurrency` resolver (null / non-positive → the compiler default).

Pairs with `pageindex_max_concurrency` (indexing side) from this PR — both
concurrency knobs are now KB-configurable.

* fix(config): address xhigh code-review findings on the concurrency knobs

- resolve_compile_concurrency moves to config.py (matching resolve_timeout's
  convention) and now rejects bool (bool is an int subclass, so
  `compile_concurrency: true` previously silently became concurrency=1) and
  logs a warning on any malformed value, same as the other resolvers.
- _build_index_config now warns when pageindex_max_concurrency is configured
  but the installed PageIndex doesn't support the field yet, instead of
  silently dropping it with no signal to the user.
- Replaced the tautological test_forwards_max_concurrency_when_supported
  (which branched on the same runtime condition as the code under test, so it
  never exercised the forwarding assertion under CI's pinned PageIndex) with
  fake IndexConfig doubles that make both branches deterministic regardless
  of the installed pageindex version.
- Added a cross-check test pinning DEFAULT_CONFIG["compile_concurrency"]
  against the compiler's own DEFAULT_COMPILE_CONCURRENCY so the two literals
  can't silently drift apart.
- Added an end-to-end test proving index_long_document's own loaded config
  (not just a hand-built dict) reaches PageIndexClient's IndexConfig.
- Hoisted the compile-concurrency resolution out of `recompile --all`'s
  per-document loop (was recomputed every iteration despite being
  loop-invariant).
- Documented both new config.yaml keys in config.yaml.example and
  examples/configuration/README.md (kept in sync).

* refactor(config): unify pageindex_max_concurrency + compile_concurrency into `concurrency`

PageIndex indexing and OpenKB's own concept/entity compilation never run
concurrently with each other for the same document (they're sequential
phases of one add), and both knobs exist for the exact same reason — the
user hit a provider rate limit or an fd ceiling. Splitting them by internal
subsystem leaked OpenKB's architecture into the config surface for no
practical benefit, since a user tuning one for a rate limit would set the
other to the same value anyway.

- Single `concurrency` config key (default null = each stage keeps its own
  built-in default).
- config.resolve_concurrency(config) -> int | None: validates (rejects bool
  and non-positive values with a warning), returns None on unset/invalid —
  no default substitution inside; callers decide what None means for them.
- cli.py's compile call sites: `resolve_concurrency(config) or
  DEFAULT_COMPILE_CONCURRENCY`.
- indexer.py's _build_index_config now routes through resolve_concurrency
  (openkb validates before forwarding to PageIndex, rather than relying
  solely on PageIndex's own future validator) instead of raw config.get.
- This also removes the prior dual-literal-default drift risk entirely:
  there's no second numeric default to keep in sync, since an unset
  `concurrency` simply forwards nothing to PageIndex and lets the compiler
  use its own DEFAULT_COMPILE_CONCURRENCY.
- Updated config.yaml.example and examples/configuration/README.md (kept in
  sync) to the single key.

* build(deps): bump pageindex to 0.3.0.dev2 (ships IndexConfig.max_concurrency)

0.3.0.dev2 is the first published release that declares
IndexConfig.max_concurrency, so `concurrency` in config.yaml now takes
effect for the indexing stage in a normal `pip install` / `uv sync`, not
just against a local editable checkout. Vetted: downloaded the wheel and
confirmed its sha256 matches PyPI (b0cb1f6e…) and that IndexConfig declares
`max_concurrency: int | None` with a positivity field_validator. uv.lock
regenerated via `uv lock --upgrade-package pageindex`.

The runtime `"max_concurrency" in IndexConfig.model_fields` guard in
indexer._build_index_config is now always-true under the pinned dependency,
but is kept as graceful degradation for mismatched local installs.

* refactor(config): keep concurrency out of DEFAULT_CONFIG (align with other optional knobs)

Every other optional tuning knob (timeout, extra_headers, litellm,
entity_types, parallel_tool_calls) is resolved via its resolver's .get()
and lives only in config.yaml.example, not DEFAULT_CONFIG — which holds just
the core keys openkb init writes (model, language, pageindex_threshold).
concurrency was the lone exception; resolve_concurrency already reads it via
.get(), so the DEFAULT_CONFIG entry was redundant. Remove it for consistency.

* build(deps): bump pageindex 0.3.0.dev2 -> 0.3.0.dev3

Vetted: identical Requires-Dist/Requires-Python to dev2; IndexConfig,
PageIndexClient, and IndexConfig.max_concurrency (used by #174) all still
present; wheel/sdist hashes verified against PyPI. uv.lock regenerated via
`uv lock` (change scoped to the pageindex entry). Also aligns the stale
dev1 reference in the config README.

* feat(cli): add --version flag

The CLI group had no way to report its version (`openkb version` / `openkb -v`
both fail). Add `@click.version_option(package_name="openkb")` so `openkb
--version` prints "openkb <version>", read from installed package metadata
(tracks the hatch-vcs-derived version, no hardcoded string).

Author: KylinMountain

config.yaml.example

@@ -2,6 +2,12 @@ model: gpt-5.4                   # LLM model (any LiteLLM-supported provider)
 language: en                     # Wiki output language
 pageindex_threshold: 20          # PDF pages threshold for PageIndex
 
+# Optional: cap concurrent LLM calls during ingest (PageIndex indexing and
+# concept/entity compilation — they never overlap, so one setting covers
+# both). Lower it if you hit provider rate limits or "too many open files" on
+# large PDFs. Omit to let each stage apply its own default.
+# concurrency: 5
+
 # Optional: whether the LLM agents (query, chat, lint, skill) may call tools
 # in parallel. Leave it UNSET (commented out) to keep OpenKB's per-agent
 # defaults. Setting it applies the SAME value to every agent:

examples/configuration/README.md

@@ -12,7 +12,7 @@ pip install openkb
 ```
 
 OpenKB pins a **pre-release** of its PageIndex dependency
-(`pageindex==0.3.0.dev1`), which some installers skip by default. If an install
+(`pageindex==0.3.0.dev3`), which some installers skip by default. If an install
 can't resolve `pageindex`, allow pre-releases:
 
 ```bash
@@ -70,6 +70,12 @@ model: gpt-5.4                   # LLM model (any LiteLLM-supported provider)
 language: en                     # Wiki output language
 pageindex_threshold: 20          # PDF pages threshold for PageIndex
 
+# Optional: cap concurrent LLM calls during ingest (PageIndex indexing and
+# concept/entity compilation — they never overlap, so one setting covers
+# both). Lower it if you hit provider rate limits or "too many open files" on
+# large PDFs. Omit to let each stage apply its own default.
+# concurrency: 5
+
 # Optional: whether the LLM agents (query, chat, lint, skill) may call tools
 # in parallel. Leave it UNSET (commented out) to keep OpenKB's per-agent
 # defaults. Setting it applies the SAME value to every agent:
@@ -105,6 +111,7 @@ pageindex_threshold: 20          # PDF pages threshold for PageIndex
 | `model` | `gpt-5.4` | LLM used for all compile/query/chat work. |
 | `language` | `en` | Language the wiki is written in. |
 | `pageindex_threshold` | `20` | PDFs with this many pages **or more** take the long-doc (PageIndex) path; shorter ones go through the short-doc path. See [`pageindex-cloud/`](../pageindex-cloud/). |
+| `concurrency` | `null` | Caps concurrent LLM calls OpenKB makes during ingest — both PageIndex's indexing of a long document and OpenKB's own concept/entity compilation. The two never run at once for the same document, so one setting covers both. Lower it if you hit provider rate limits or "too many open files" on large PDFs. `null` lets each stage apply its own default. |
 | `parallel_tool_calls` | unset | Whether the LLM agents (query, chat, lint, skill) may call tools in parallel. Unset keeps OpenKB's per-agent defaults; `true`/`false` force allow/sequential for every agent; `null` omits the setting (provider default). **Amazon Bedrock needs `null`** (see below). |
 | `entity_types` | 7 defaults | Custom vocabulary for entity pages. `other` is always kept. |
 | `litellm:` | – | A pass-through block for LiteLLM. See below. |

openkb/cli.py

@@ -47,13 +47,14 @@ def filter(self, record: logging.LogRecord) -> bool:
 litellm.suppress_debug_info = True
 from dotenv import load_dotenv
 
-from openkb.agent.compiler import compile_long_doc
+from openkb.agent.compiler import DEFAULT_COMPILE_CONCURRENCY, compile_long_doc
 from openkb.config import (
     DEFAULT_CONFIG,
     load_config,
     save_config,
     load_global_config,
     register_kb,
+    resolve_concurrency,
     resolve_extra_headers,
     set_extra_headers,
     resolve_parallel_tool_calls,
@@ -561,6 +562,7 @@ def commit_body(snapshot) -> None:
                     kb_dir,
                     model,
                     doc_description=index_result.description,
+                    max_concurrency=resolve_concurrency(config) or DEFAULT_COMPILE_CONCURRENCY,
                 ),
                 label=f"Compiling long doc (doc_id={index_result.doc_id})",
             )
@@ -569,7 +571,13 @@ def commit_body(snapshot) -> None:
                 raise RuntimeError(f"Converted document has no source artifact: {file_path.name}")
             source_path = result.source_path
             _run_compile_with_retry(
-                lambda: compile_short_doc(doc_name, source_path, kb_dir, model),
+                lambda: compile_short_doc(
+                    doc_name,
+                    source_path,
+                    kb_dir,
+                    model,
+                    max_concurrency=resolve_concurrency(config) or DEFAULT_COMPILE_CONCURRENCY,
+                ),
                 label="Compiling short doc",
             )
 
@@ -690,6 +698,7 @@ def commit_body(_snapshot) -> None:
                         kb_dir,
                         model,
                         doc_description=cloud.description,
+                        max_concurrency=resolve_concurrency(config) or DEFAULT_COMPILE_CONCURRENCY,
                     ),
                     label=f"Compiling imported doc (doc_id={doc_id})",
                 )
@@ -748,6 +757,7 @@ def append_cloud_log() -> None:
 
 
 @click.group()
+@click.version_option(package_name="openkb", prog_name="openkb", message="%(prog)s %(version)s")
 @click.option("-v", "--verbose", is_flag=True, default=False, help="Enable verbose logging.")
 @click.option(
     "--kb-dir",
@@ -1642,6 +1652,7 @@ def _classify(meta: dict) -> str:
     _setup_llm_key(kb_dir)
     config = load_config(openkb_dir / "config.yaml")
     model: str = config.get("model", DEFAULT_CONFIG["model"])
+    max_concurrency = resolve_concurrency(config) or DEFAULT_COMPILE_CONCURRENCY
 
     # Import lazily and reference via the module so tests can patch
     # ``openkb.agent.compiler.compile_*`` and see the call.
@@ -1677,7 +1688,16 @@ def _classify(meta: dict) -> str:
             click.echo(f"[{i}/{total}] Recompiling long doc {name}...")
             start = time.time()
             try:
-                asyncio.run(compiler.compile_long_doc(name, summary_path, doc_id, kb_dir, model))
+                asyncio.run(
+                    compiler.compile_long_doc(
+                        name,
+                        summary_path,
+                        doc_id,
+                        kb_dir,
+                        model,
+                        max_concurrency=max_concurrency,
+                    )
+                )
             except Exception as exc:
                 click.echo(f"  [ERROR] Compilation failed: {exc}")
                 logging.getLogger(__name__).debug("Recompile traceback:", exc_info=True)
@@ -1697,7 +1717,15 @@ def _classify(meta: dict) -> str:
             click.echo(f"[{i}/{total}] Recompiling short doc {name}...")
             start = time.time()
             try:
-                asyncio.run(compiler.compile_short_doc(name, source_path, kb_dir, model))
+                asyncio.run(
+                    compiler.compile_short_doc(
+                        name,
+                        source_path,
+                        kb_dir,
+                        model,
+                        max_concurrency=max_concurrency,
+                    )
+                )
             except Exception as exc:
                 click.echo(f"  [ERROR] Compilation failed: {exc}")
                 logging.getLogger(__name__).debug("Recompile traceback:", exc_info=True)

openkb/config.py

@@ -200,6 +200,30 @@ def resolve_timeout(config: dict) -> float | None:
     return value
 
 
+def resolve_concurrency(config: dict) -> int | None:
+    """Resolve the optional ``concurrency:`` key — the cap on concurrent LLM
+    calls OpenKB makes during ingest (PageIndex indexing and concept/entity
+    compilation alike; they never run at the same time for one document, so
+    one setting covers both).
+
+    Returns ``None`` when absent, explicitly ``null``, or invalid (rejecting
+    bools and non-positive values with a warning when present but unusable —
+    an explicit ``null``/absent key is the normal "unset" case and stays
+    silent, matching ``resolve_timeout``). Callers apply their own default (or
+    omit the setting entirely) when this returns ``None``.
+    """
+    value = config.get("concurrency")
+    if value is None:
+        return None
+    if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+        logger.warning(
+            "config: 'concurrency' must be a positive integer, got %r — ignoring it.",
+            value,
+        )
+        return None
+    return value
+
+
 def resolve_litellm_settings(config: dict) -> dict[str, Any]:
     """Resolve the optional ``litellm:`` mapping of LiteLLM module settings.
 

openkb/indexer.py

@@ -11,7 +11,7 @@
 
 from pageindex import IndexConfig, PageIndexClient
 
-from openkb.config import load_config
+from openkb.config import load_config, resolve_concurrency
 from openkb.tree_renderer import render_summary_md
 
 logger = logging.getLogger(__name__)
@@ -153,6 +153,33 @@ def _write_long_doc_artifacts(
     return summary_path
 
 
+def _build_index_config(config: dict[str, Any]) -> IndexConfig:
+    """Build the PageIndex ``IndexConfig`` for local indexing.
+
+    Forwards the KB's ``concurrency`` setting to PageIndex, which caps how many
+    indexing LLM calls run at once (guarding against "too many open files" fd
+    exhaustion on large documents). The value is only passed when set *and* the
+    installed PageIndex's ``IndexConfig`` declares the field, so OpenKB keeps
+    working against a pinned PageIndex that predates it (``IndexConfig``
+    forbids unknown kwargs).
+    """
+    kwargs: dict[str, Any] = {
+        "if_add_node_text": True,
+        "if_add_node_summary": True,
+        "if_add_doc_description": True,
+    }
+    concurrency = resolve_concurrency(config)
+    if concurrency is not None:
+        if "max_concurrency" in IndexConfig.model_fields:
+            kwargs["max_concurrency"] = concurrency
+        else:
+            logger.warning(
+                "config: 'concurrency' is set but the installed PageIndex "
+                "version does not support it yet — ignoring it."
+            )
+    return IndexConfig(**kwargs)
+
+
 def index_long_document(pdf_path: Path, kb_dir: Path, doc_name: str | None = None) -> IndexResult:
     """Index a long PDF document using PageIndex and write wiki pages.
 
@@ -166,11 +193,7 @@ def index_long_document(pdf_path: Path, kb_dir: Path, doc_name: str | None = Non
     model: str = config.get("model", "gpt-5.4")
     pageindex_api_key = os.environ.get("PAGEINDEX_API_KEY", "")
 
-    index_config = IndexConfig(
-        if_add_node_text=True,
-        if_add_node_summary=True,
-        if_add_doc_description=True,
-    )
+    index_config = _build_index_config(config)
 
     client = PageIndexClient(
         api_key=pageindex_api_key or None,

pyproject.toml

@@ -32,7 +32,7 @@ keywords = ["ai", "rag", "retrieval", "knowledge-base", "llm", "pageindex", "age
 # returning empty Responses output (BerriAI/litellm#25429) and
 # auto-injects GitHub Copilot IDE-auth headers.
 dependencies = [
-    "pageindex==0.3.0.dev1",
+    "pageindex==0.3.0.dev3",
     "markitdown[docx,pptx,xlsx,xls]==0.1.5",
     "trafilatura==2.0.0",
     "click==8.4.0",

tests/test_add_command.py

@@ -97,6 +97,29 @@ def test_add_single_file_compile_failure_rolls_back_converted_artifacts(self, tm
         assert not (kb_dir / "wiki" / "sources" / "notes.md").exists()
         assert HashRegistry(kb_dir / ".openkb" / "hashes.json").all_entries() == {}
 
+    def test_add_forwards_concurrency_from_config(self, tmp_path):
+        from unittest.mock import AsyncMock
+
+        from openkb.cli import add_single_file
+
+        kb_dir = self._setup_kb(tmp_path)
+        (kb_dir / ".openkb" / "config.yaml").write_text(
+            "model: gpt-4o-mini\nconcurrency: 3\n", encoding="utf-8"
+        )
+        doc = tmp_path / "notes.md"
+        doc.write_text("# Notes\n\nBody", encoding="utf-8")
+
+        with (
+            patch(
+                "openkb.agent.compiler.compile_short_doc", new_callable=AsyncMock
+            ) as mock_compile,
+            patch("openkb.cli._setup_llm_key"),
+        ):
+            outcome = add_single_file(doc, kb_dir)
+
+        assert outcome == "added"
+        assert mock_compile.call_args.kwargs["max_concurrency"] == 3
+
     def test_add_single_file_uses_add_mutation_coordinator(self, tmp_path):
         from openkb.cli import add_single_file
         from openkb.converter import ConvertResult

tests/test_cli.py

@@ -9,6 +9,17 @@
 from openkb.schema import AGENTS_MD
 
 
+def test_version_flag_reports_installed_version():
+    import importlib.metadata
+
+    runner = CliRunner()
+    result = runner.invoke(cli, ["--version"])
+    assert result.exit_code == 0
+    # Reports the package's installed version (via importlib.metadata), so it
+    # tracks the hatch-vcs-derived version without a hardcoded string.
+    assert importlib.metadata.version("openkb") in result.output
+
+
 def test_init_creates_structure(tmp_path):
     runner = CliRunner()
     with runner.isolated_filesystem(temp_dir=tmp_path), patch("openkb.cli.register_kb"):

tests/test_config.py

@@ -6,6 +6,7 @@
     get_parallel_tool_calls,
     get_timeout,
     load_config,
+    resolve_concurrency,
     resolve_extra_headers,
     resolve_litellm_settings,
     resolve_model_settings,
@@ -135,6 +136,57 @@ def test_default_config_values():
     assert DEFAULT_CONFIG["pageindex_threshold"] == 20
 
 
+def test_concurrency_not_in_default_config():
+    # Like the other optional tuning knobs (timeout, extra_headers,
+    # parallel_tool_calls), concurrency stays out of DEFAULT_CONFIG —
+    # resolve_concurrency reads it via .get(), so an absent key resolves to
+    # None without relying on load_config's merge.
+    assert "concurrency" not in DEFAULT_CONFIG
+
+
+def test_load_concurrency_override(tmp_path):
+    config_path = tmp_path / "config.yaml"
+    config_path.write_text("concurrency: 12\n", encoding="utf-8")
+    assert load_config(config_path)["concurrency"] == 12
+
+
+def test_resolve_concurrency_absent_is_none():
+    assert resolve_concurrency({}) is None
+
+
+def test_resolve_concurrency_valid_value():
+    assert resolve_concurrency({"concurrency": 3}) == 3
+
+
+def test_resolve_concurrency_rejects_bool(caplog):
+    with caplog.at_level(logging.WARNING, logger="openkb.config"):
+        result = resolve_concurrency({"concurrency": True})
+    assert result is None
+    assert "concurrency" in caplog.text
+
+
+def test_resolve_concurrency_rejects_non_positive(caplog):
+    with caplog.at_level(logging.WARNING, logger="openkb.config"):
+        assert resolve_concurrency({"concurrency": 0}) is None
+    assert "concurrency" in caplog.text
+    caplog.clear()
+    with caplog.at_level(logging.WARNING, logger="openkb.config"):
+        assert resolve_concurrency({"concurrency": -1}) is None
+    assert "concurrency" in caplog.text
+
+
+def test_resolve_concurrency_rejects_non_int():
+    assert resolve_concurrency({"concurrency": "3"}) is None
+
+
+def test_resolve_concurrency_none_is_silent(caplog):
+    # Explicit null / absent is the normal "unset — caller applies its own
+    # default, or omits the setting entirely" case — no warning.
+    with caplog.at_level(logging.WARNING, logger="openkb.config"):
+        assert resolve_concurrency({"concurrency": None}) is None
+    assert caplog.text == ""
+
+
 def test_load_missing_file_returns_defaults(tmp_path):
     missing = tmp_path / "nonexistent" / "config.yaml"
     config = load_config(missing)