[LFX Term 1 2026] Restoring LLM Edge Benchmark Suite Single Task Bench

[NishantSinghhhhh]

What type of PR is this?
/kind bug
/kind feature

What this PR does / why we need it:
This PR restores and fixes the llm-edge-benchmark-suite (single_task_bench) so it successfully executes end-to-end within the current Ianvs SingleTaskLearning paradigm.

Previously, the LLM benchmarking pipeline was failing due to missing dependencies, pipeline contract mismatches (missing/strict method signatures), and inaccurate C++ stderr log parsing for latency metrics.

Key changes include:

• Added Dependencies: Created a requirements.txt to explicitly include llama-cpp-python and other necessary inference packages.
• Pipeline Contract Alignment: Added preprocess and postprocess methods to LlamaCppModel with optional arguments (data=None, **kwargs) to satisfy the strict signature checks of the Ianvs pipeline without throwing TypeError.

[gemini-code-assist]

Code Review

This pull request introduces an LLM edge benchmark suite, adding a requirements file and implementing a base model class. The predict method was refactored to use streaming for measuring prefill latency, and dummy preprocess, postprocess, and train methods were added to satisfy pipeline requirements. A review comment suggests that the train method should return the model path from an instance variable rather than kwargs, as the latter may be empty during execution, leading to an incorrect return value.

[NishantSinghhhhh]

@MooreZheng sir, I have added Readme for LLM Edge Benchmark Suite Single Task Bench, a Requirements.txt and minimal changes in the Basemodel.py to make it work,

Screencast.from.2026-04-23.18-52-16.webm

[NishantSinghhhhh]

1. Optional pre-Process and Post process
2. Not changing the indentation
3. no unnecessary comments
4. Reasoning for prediction as 0

[MooreZheng]

1. try to make the preprocess optionable to save future burden
2. check whether there is problem for the prediction fuction. If not, just leave it there.

[NishantSinghhhhh]

@MooreZheng Sir,

Have looked into both questions — here's where things stand:

1. preprocess : The function needs to stay. The framework calls it unconditionally — SingleTaskLearning invokes job.preprocess() with zero arguments at singletask_learning.py:114, so removing it causes an AttributeError before the pipeline even starts. I've kept it as a minimal pass-through (data=None, **kwargs) so the body is as lightweight as possible. Making it truly optional would require a change in Ianvs core itself.

2. predict : No structural issues here. Streaming/token accumulation, latency timing, memory capture, and hyperparameters are all working correctly. The one actual bug in the predict path was in _parse_timings: the regex was matching against llama_print_timings:, but modern llama-cpp-python emits llama_perf_context_print: instead — so prefill_latency was silently returning 0. I've widened the regex to cover both patterns; predict itself is otherwise untouched.

Two minor items I left as-is for now:
• data = data[:10] — the hardcoded 10-prompt cap appears intentional
• mem_usage is sampled at end-of-prompt (steady-state), not at peak

[NishantSinghhhhh]

Screencast.from.2026-05-20.16-35-27.webm

The example runs without the pre-process function !!

[MooreZheng]

Good! Very close to the final version. Just squash the commit into one and this PR is good to go.

[NishantSinghhhhh]

@MooreZheng Sir @hsj576 Sir squashed the commits !!

[MooreZheng]

/lgtm

[hsj576]

/lgtm

[MooreZheng]

/approve

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hsj576, MooreZheng, NishantSinghhhhh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [MooreZheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment