whenever we change the prompt in tigent.yml, the system prompt, or any relevant code, we want to make sure we don't break current behavior.
Ideas we had so far
- Add tests to the code with webhook fixtures and desired outcome in form of labels
- Create a dedicated evaluations repository on the @tigent org with issues and labels, then use the issues as input (dynamically requested when running evaluations) and the applied labels as output
- Use the vercel/ai repository itself by adding
evaluations which is an array of numbers and use issues/PRs on the ai repository as evals
whenever we change the prompt in
tigent.yml, the system prompt, or any relevant code, we want to make sure we don't break current behavior.Ideas we had so far
evaluationswhich is an array of numbers and use issues/PRs on the ai repository as evals