Skip to content

Add LLMLingua Prompt Compressor in DSPy#17

Open
mango-panther wants to merge 6 commits into
swe-productivity:mainfrom
mango-panther:prompt_compressor
Open

Add LLMLingua Prompt Compressor in DSPy#17
mango-panther wants to merge 6 commits into
swe-productivity:mainfrom
mango-panther:prompt_compressor

Conversation

@mango-panther
Copy link
Copy Markdown

@mango-panther mango-panther commented Mar 18, 2026

Related to issue #13
This PR adds Prompt Compressor using the LLMLingua library in DSPy
It introduces new PromptCompressor class which uses the functionality from LLMLingua

Below is the code snipper that I have used to check whether working of code:

import dspy

class CheckCitationFaithfulness(dspy.Signature):
    """Verify that the text is based on the provided context."""

    context: str = dspy.InputField(desc="facts here are assumed to be true", compression=True)
    text: str = dspy.InputField(compression=False)
    faithfulness: bool = dspy.OutputField()
    evidence: list[str] = dspy.OutputField(desc="List of exact quotes from the text supporting evidence for claims")



def main():

    # create compressor 
    compressor = dspy.PromptCompressor(
        model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
        dynamic_context_compression_ratio=0.55,           # desired compressed size as fraction of original tokens
        preserve_fields=["username, history"],        # list of regex/keys to preserve fully (e.g., user names)
        target_token=None,      # hard limit to compressed result if desired
        device="cpu",                # or "cuda"
        cache=True                   # cache results for identical inputs
    )
    lm = dspy.LM("ollama_chat/llama3.2:1b", api_base="http://localhost:11434", api_key=api_key)
    dspy.settings.configure(lm=lm, compressor=compressor)
    evaluator = dspy.Predict(CheckCitationFaithfulness)
    long_document_context = (
        "Belgium, officially the Kingdom of Belgium, is a country in Northwestern Europe. "
        "The country is bordered by the Netherlands to the north, Germany to the east, "
        "Luxembourg to the southeast, France to the southwest, and the North Sea to the "
        "northwest. It covers an area of 30,689 square kilometers and has a population "
        "of more than 11.5 million. The capital and largest city is Brussels."
    )
    
    # 5. Execute the prompt!
    result = evaluator(
        context=long_document_context,
        text="Belgium's capital is Brussels and it neighbors the Netherlands."
    )

    print(result.faithfulness)
    print(result.evidence)
    print(result.compression_metadata)

if __name__ == "__main__":
    main()

Its output is:

True
["Belgium's capital is Brussels and it neighbors the Netherlands.", 'The Kingdom of Belgium has its capital in Brussels.']
{'total_original_tokens': 82, 'total_compressed_tokens': 37, 'ratio': 0.45121951219512196, 'model': 'microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank'}

I have also updated docs, and checked them

@mango-panther
Copy link
Copy Markdown
Author

I dont know what to do with cache=True as LLMLingua doesn't have that particular parameter specifically, I guess they are by default doing caching, though it was not prompt caching, its KV Caching

@mango-panther
Copy link
Copy Markdown
Author

@TomeHirata Could you also review this one when you get time

Comment thread docs/docs/api/primitives/PromptCompressor.md Outdated
Comment thread dspy/primitives/compressor.py Outdated
Comment thread dspy/primitives/compressor.py Outdated
Comment thread dspy/primitives/compressor.py Outdated
Comment thread dspy/primitives/compressor.py Outdated
Comment thread dspy/primitives/compressor.py Outdated
Comment thread dspy/primitives/compressor.py Outdated
Comment thread dspy/predict/predict.py Outdated
@TomeHirata
Copy link
Copy Markdown
Collaborator

Can you add tests?

@mango-panther
Copy link
Copy Markdown
Author

I dont know what to do with cache=True as LLMLingua doesn't have that particular parameter specifically, I guess they are by default doing caching, though it was not prompt caching, its KV Caching

@TomeHirata Could you answer this one

@mango-panther
Copy link
Copy Markdown
Author

@TomeHirata I have apply changed based on comments, updated docs based on other api references and also added tests.

@mango-panther
Copy link
Copy Markdown
Author

@TomeHirata Can you review this again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants