can you do a second ragas notebook which is based on https://dkaarthick.medium.com/ragas-for-rag-in-llms-a-comprehensive-guide-to-evaluation-metrics-3aca142d6e38 and take the sample from there into a notebook format with different cells and more explanation in the notebook.
here the sample in case you cannot read the artikel:
code:
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
answer_correctness,
context_precision,
context_recall,
)
Example data
data = {
"query": ["What is the capital of France?"],
"generated_response": ["Paris is the capital of France."],
"retrieved_documents": [["Paris is the capital of France. It is a major European city known for its culture."]]
}
Convert the data to a Hugging Face Dataset
dataset = Dataset.from_dict(data)
Define the metrics you want to evaluate
metrics = [
faithfulness,
answer_relevancy,
answer_correctness,
context_precision,
context_recall,
]
Evaluate the dataset using the selected metrics
results = evaluate(dataset, metrics)
Display the results
for metric_name, score in results.items():
print(f"{metric_name}: {score:.2f}")
explanation:
Explanation of the Code
Data Structure: The dataset comprises three key components:
query: The input question to the RAG model.
generated_response: The response generated by the model.
retrieved_documents: The documents retrieved by the model to help generate the response.
Metrics: The code evaluates the generated responses using the faithfulness, answer_relevancy, answer_correctness, context_precision, and context_recall metrics.
Output: The results are printed, providing a score for each metric, helping to understand how well the model performs across these dimensions.
Conclusion
RAGAS offers a specialized suite of metrics that go beyond traditional text generation evaluation methods like BLEU and ROUGE. These traditional metrics, while effective for general text similarity tasks, fall short in capturing the nuances of factual accuracy, context relevance, and support in RAG applications. RAGAS metrics such as Faithfulness, Answer Relevancy, Context Precision, and Context Recall provide a more comprehensive evaluation framework for RAG models, ensuring that generated responses are not only relevant but also grounded in factual evidence.
can you do a second ragas notebook which is based on https://dkaarthick.medium.com/ragas-for-rag-in-llms-a-comprehensive-guide-to-evaluation-metrics-3aca142d6e38 and take the sample from there into a notebook format with different cells and more explanation in the notebook.
here the sample in case you cannot read the artikel:
code:
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
answer_correctness,
context_precision,
context_recall,
)
Example data
data = {
"query": ["What is the capital of France?"],
"generated_response": ["Paris is the capital of France."],
"retrieved_documents": [["Paris is the capital of France. It is a major European city known for its culture."]]
}
Convert the data to a Hugging Face Dataset
dataset = Dataset.from_dict(data)
Define the metrics you want to evaluate
metrics = [
faithfulness,
answer_relevancy,
answer_correctness,
context_precision,
context_recall,
]
Evaluate the dataset using the selected metrics
results = evaluate(dataset, metrics)
Display the results
for metric_name, score in results.items():
print(f"{metric_name}: {score:.2f}")
explanation:
Explanation of the Code
Data Structure: The dataset comprises three key components:
query: The input question to the RAG model.
generated_response: The response generated by the model.
retrieved_documents: The documents retrieved by the model to help generate the response.
Metrics: The code evaluates the generated responses using the faithfulness, answer_relevancy, answer_correctness, context_precision, and context_recall metrics.
Output: The results are printed, providing a score for each metric, helping to understand how well the model performs across these dimensions.
Conclusion
RAGAS offers a specialized suite of metrics that go beyond traditional text generation evaluation methods like BLEU and ROUGE. These traditional metrics, while effective for general text similarity tasks, fall short in capturing the nuances of factual accuracy, context relevance, and support in RAG applications. RAGAS metrics such as Faithfulness, Answer Relevancy, Context Precision, and Context Recall provide a more comprehensive evaluation framework for RAG models, ensuring that generated responses are not only relevant but also grounded in factual evidence.