This repo is forked from https://github.com/Liyan06/MiniCheck (MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents [EMNLP 2024]) which is based on vLLM. However, in this repo, vLLM got removed entirely and has been replaced with transformers. For installation and usage, see below.
pip install "minicheck @ git+https://github.com/F4biian/MiniCheck-Transformers.git@main"This code does not work with the latest transformers version, which is why it is forcefully downgraded to
transformers==4.39.0!
You might be asked by nltk to download punkt_tab. Just do so in a python script once:
import nltk
nltk.download('punkt_tab')The code for using Bespoke-MiniCheck-7B is almost the same as before except that the parameters tensor_parallel_size and enable_prefix_caching have been removed from the MiniCheck class.
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
# import nltk
# nltk.download('punkt_tab')
# exit()
from minicheck.minicheck import MiniCheck
doc = "A group of students gather in the school library to study for their upcoming final exams."
claim_1 = "The students are preparing for an examination."
claim_2 = "The students are on vacation."
# model_name can be one of:
# ['roberta-large', 'deberta-v3-large', 'flan-t5-large', 'Bespoke-MiniCheck-7B']
# bespokelabs/Bespoke-MiniCheck-7B will be auto-downloaded from Huggingface for the first time
# Bespoke-MiniCheck-7B is the most performant fact-checking model in the MiniCheck series AND
# it outperforms ALL exisiting specialized fact-checkers and off-the-shelf LLMs regardless of size.
scorer = MiniCheck(model_name='Bespoke-MiniCheck-7B')
pred_label, raw_prob, _, _ = scorer.score(docs=[doc, doc], claims=[claim_1, claim_2])
# Output of this repo's code:
print(pred_label) # [1, 0]
print(raw_prob) # [np.float64(0.9840496778488159), np.float64(0.010986425913870335)]
# Output of original repo's code:
print(pred_label) # [1, 0]
print(raw_prob) # [0.9840446675150499, 0.010986349594852094]