Larger semantic chunks sent to Ollama embeddings cause indexing failure

[jani-laakso]

What happened?

CCE aborts indexing because it sends oversized semantic chunks directly to Ollama’s /api/embed endpoint.
NOTE: This was perfectly valid Go code, around 70 lines, just contained enough longer type names / strings to blow up chunks to oversized.

During cce init, indexing starts normally, but the first embedding batch fails with:

Embedding failed: Client error '400 Bad Request' for url 'http://localhost:11434/api/embed'

After locally patching CCE to print the Ollama response body, the actual backend error is:

{"error":"the input length exceeds the context length"}

The failed request batch contained large chunks, for example:

text count: 64
chunk sizes: 1147, 898, 0, 281, 195, 1029, 3814, 11115, 54309, ...

The oversized chunks came from normal repository files, not minified/generated one-line blobs. Examples included:

• A Markdown document or large Markdown section emitted as one chunk of around 54k characters.
• Another Markdown section emitted as one chunk of around 11k characters.
• Normal multi-line source-code functions emitted as single chunks around 5–7k characters.

This appears to happen because CCE treats semantic units, such as whole Markdown sections or whole source-code functions, as embedding inputs without applying a defensive maximum size before calling the embedding backend.

This makes indexing fail completely for repositories that contain large-but-valid documentation sections or source-code functions. Empty chunks also appear to be included in the embedding batch, as shown by a 0-length chunk in the same request.

What did you expect?

CCE should split or cap oversized chunks before sending them to the embedding backend.

Expected behavior:

• cce init should complete indexing instead of aborting on one oversized chunk.
• CCE should not send any input to /api/embed that can exceed the selected model’s context length.
• Empty chunks should be skipped.
• Large semantic chunks should be split into smaller embedding-safe child chunks.
• Split chunks should preserve parent metadata such as file path, symbol or heading name, and line range where available.
• If a chunk cannot be embedded, CCE should warn and continue indexing the rest of the repository instead of failing the whole indexing run.

Conceptually, semantic chunks and embedding inputs should be treated separately:

semantic chunk: whole function, class, Markdown section, etc.
embedding input: smaller model-safe slice of that semantic chunk

A reasonable fallback would be something like:

MAX_EMBED_CHARS = 3000
OVERLAP_CHARS = 200

if chunk is empty:
    skip
elif chunk length <= MAX_EMBED_CHARS:
    embed as-is
else:
    split into child chunks with overlap
    preserve parent metadata

Token-aware splitting would be better, but even a conservative character cap would prevent hard failures with Ollama embedding models.

Steps to reproduce

1. Install CCE and initialize it in a repository with normal but large Markdown documentation sections or source-code functions.

2. Use Ollama as the embedding backend with nomic-embed-text.

3. Verify Ollama is running and the model is available:

   ollama list
   curl -s http://localhost:11434/api/version

4. Run:

   ```bash
   cce init
   ```

5. Observe that indexing starts, then fails during the first embedding batch:

   ```text
   Embedding failed: Client error '400 Bad Request' for url 'http://localhost:11434/api/embed'
   ```

6. Patch CCE locally to print the Ollama response body before `resp.raise_for_status()` in the embedder.

7. Run again:

   ```bash
   cce init
   ```

8. Observe the real Ollama error:

   ```json
   {"error":"the input length exceeds the context length"}
   ```

9. Also observe that the failed embedding request contains oversized chunks, for example chunks around 11k and 54k characters, plus at least one empty chunk.

```


### Relevant logs or error output

```shell
jani@io$ cce init

  Code Context Engine  ·  dlm
  ────────────────────────────────────────────

  Detecting embedding backend... ready (ollama, 768-d, nomic-embed-text)
  Ollama detected — LLM summarization enabled.

  ✓ Git hooks installed  (3 hooks, auto-updates on commit)
  ✓ MCP server already configured in .mcp.json
  ✓ MCP server already configured for OpenAI Codex
    ~/.codex/config.toml  →  [mcp_servers.cce-dlm-9ef807]
  ✓ Memory hooks already configured
  · Memory capture not yet active — `cce serve` hasn't been started for this project.
    Run `cce serve` in a separate terminal so the loopback hook server starts;
    until it's running, hooks fire successfully but capture is silently dropped.
    Verify any time with `cce sessions status`.
  ✓ .gitignore updated with CCE entries

  Indexing project...
    ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  50/1559 files  3%
    Embedding 322 chunks (batch 1/32)…
    ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  0/322 chunks embedded  0%Embedding failed: Client error '400 Bad Request' for url 'http://localhost:11434/api/embed'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
Traceback (most recent call last):
  File "/home/jani/.local/share/uv/tools/code-context-engine/lib/python3.12/site-packages/context_engine/indexer/pipeline.py", line 465, in _embed_and_ingest
    emb.embed(batch_chunks, progress_fn=embed_progress_fn)
  File "/home/jani/.local/share/uv/tools/code-context-engine/lib/python3.12/site-packages/context_engine/indexer/embedder.py", line 503, in embed
    self._embed_all(miss_chunks, batch_size, progress_fn=progress_fn)
  File "/home/jani/.local/share/uv/tools/code-context-engine/lib/python3.12/site-packages/context_engine/indexer/embedder.py", line 543, in _embed_all
    for i, vec in enumerate(iterator(texts, batch_size=batch_size)):
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jani/.local/share/uv/tools/code-context-engine/lib/python3.12/site-packages/context_engine/indexer/embedder.py", line 354, in iter_embed
    for vec in self._embed_batch(texts[i : i + batch_size]):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jani/.local/share/uv/tools/code-context-engine/lib/python3.12/site-packages/context_engine/indexer/embedder.py", line 342, in _embed_batch
    resp.raise_for_status()
  File "/home/jani/.local/share/uv/tools/code-context-engine/lib/python3.12/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'http://localhost:11434/api/embed'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

  ✗ Error: Embedding failed: Client error '400 Bad Request' for url 'http://localhost:11434/api/embed'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
  ✓ Indexed 0 chunks from 0 files

  Done!  Restart your AI coding agent to activate CCE.
```

### Python version

3.13.12

### OS

Linux Debian

### CCE version

cce, version 0.4.21

[jani-laakso]

Local workaround

This works
I patched embedder.py locally with a defensive cap before sending texts to Ollama:

def _cap_embed_text(text: str, max_chars: int = 3000) -> str:
    if not text:
        return " "

    if len(text) <= max_chars:
        return text

    cut = text.rfind("\n", 0, max_chars)

    if cut < max_chars * 0.6:
        cut = max(
            text.rfind(" ", 0, max_chars),
            text.rfind("\t", 0, max_chars),
        )

    if cut < max_chars * 0.6:
        cut = max_chars

    return text[:cut].rstrip() + "\n\n[TRUNCATED_FOR_EMBEDDING]"

texts = [_cap_embed_text(t, 3000) for t in texts]

This prevents Ollama from rejecting oversized inputs and preserves vector count by converting empty chunks to " ".

This is only a workaround. It is intentionally simple, but not ideal because it truncates the tail of large functions/documents, so some searchable context is lost.

A better fix would probably be to split oversized semantic chunks into smaller child chunks before embedding, with overlap and preserved parent metadata. That would avoid hard failures without silently losing useful content.

After this indexing succeeds ..

Indexing project...
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 50/1559 files 3%
Embedding 322 chunks (batch 1/32)…
██████████████████████████████ 322/322 chunks embedded 100%
Writing 322 chunks to vector + FTS + graph index (batch 1/32)…
█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 100/1559 files 6%
██████████████████████████████ 9/9 chunks embedded 100%0%0%
Embedding 95 chunks (git history)…
██████████████████████████████ 95/95 chunks embedded 100%
Writing 95 chunks to vector + FTS + graph index (git history)…
✓ Indexed 2,241 chunks from 1,555 files, skipped 4 non-text, 4% cache hit

Done! Restart your AI coding agent to activate CCE.

[jani-laakso]

embedder-cce-ollama-context-cap.patch

[tdyas]

I encountered this issue today when trying to use code-context-engine to index https://github.com/pantsbuild/pants/

[rajkumarsakthivel]

Fixed in PR #88. The Ollama embedder now:

1. Truncates all texts to 3,000 chars before sending to /api/embed. This handles the large Markdown sections (54k chars) and long functions you hit. The limit is conservative enough for dense-tokenizing content (~1 char/token for YAML with ${{ }}).

2. Skips empty chunks (the 0-length chunk in your batch).

3. Falls back to one-at-a-time with halving retry if a batch still fails after truncation. So even if a text is still too long after truncation, it halves until it fits rather than crashing the entire indexing run.

Your analysis of the problem was spot-on. The semantic chunks (whole Markdown sections, whole functions) were being sent directly as embedding inputs without a size guard.

Embedding quality for truncated chunks is slightly reduced (the tail of a 54k-char section is lost), but the alternative was a hard crash. The 3,000-char window captures the most important part of any chunk (the opening lines with signatures, docstrings, and key logic).