Skip to content

Example: streaming LLM responses with runPrompt #12

Description

@Cryptoteep

Context

LLMCall returns Promise<string>, which fits the happy path but doesn't surface streaming. Many users will want to stream tokens to the UI while still getting a final string for eval/assertions.

Proposal

This is primarily a docs + example task. Add examples/streaming.ts showing a pattern for streaming where:

  1. The call function writes chunks to a sink (e.g. stdout or a callback) as they arrive.
  2. It accumulates and returns the full string at the end.
  3. runPrompt works unchanged (it just awaits the final string).
import { runPrompt } from 'promptkit'
import { summarize } from './prompts'

const streamCall = async (prompt: string) => {
  const stream = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: prompt }],
    stream: true,
  })
  let full = ''
  for await (const chunk of stream) {
    const delta = chunk.choices[0]?.delta?.content ?? ''
    if (delta) {
      process.stdout.write(delta)
      full += delta
    }
  }
  return full
}

const result = await runPrompt(summarize, { text: '…' }, { call: streamCall })

Implementation notes

  • Add the example file.
  • Add a section to docs/getting-started.md titled "Streaming responses".
  • Cross-link from the README quick-start.
  • Optionally mention that eval assertions still run on the complete output.

Acceptance

  • examples/streaming.ts added and commented.
  • docs/getting-started.md updated.
  • README links to the example.

/label "good first issue" "area:docs" "enhancement"

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions