Priority Level
Medium
Task Summary
Reduce brittleness and per model required customization by always building examples as strings and call the tokenizer directly. Do not concatenate token ids ourselves.
Other goals and requirements:
- Reduce duplicate code in
assembler.py
- Examples after assembly should round trip tokenizer's decode and encode with zero changes (not true today)
- Test that structured generation patterns match assembled examples
Technical Details & Implementation Plan
No response
Dependencies
No response
Priority Level
Medium
Task Summary
Reduce brittleness and per model required customization by always building examples as strings and call the tokenizer directly. Do not concatenate token ids ourselves.
Other goals and requirements:
assembler.pyTechnical Details & Implementation Plan
No response
Dependencies
No response