Conversation
| return pd.DataFrame({ | ||
| "Course Name": [ | ||
| "Introduction to Programming", | ||
| "Advanced Programming", | ||
| "Cooking Basics", | ||
| "Advanced Culinary Arts", | ||
| "Data Structures", | ||
| "Algorithms", | ||
| "French Cuisine", | ||
| "Italian Cooking" | ||
| ], | ||
| "Department": [ | ||
| "CS", "CS", "Culinary", "Culinary", | ||
| "CS", "CS", "Culinary", "Culinary" | ||
| ], | ||
| "Level": [ | ||
| 100, 200, 100, 200, | ||
| 300, 300, 200, 200 | ||
| ] | ||
| }) |
There was a problem hiding this comment.
This file contains a bunch of ruff formatting changes. It's that weird thing where the CI does not work the same way as local. In any case this formatting is better.
| class TestFilterWithProbs(BaseTest): | ||
| def test_filter_with_probs(self, sample_df): | ||
| """Test semantic filter with probabilities returned to the user""" | ||
| lm = LM(model="gpt-4o-mini") | ||
| lotus.settings.configure(lm=lm) | ||
| result = sample_df.sem_filter("{Course Name} will be fun", return_probs=True) | ||
| print(result) | ||
| assert "probs_filter" in result.columns | ||
|
|
||
| def test_filter_with_probs_and_return_all(self, sample_df): | ||
| """Test semantic filter with probabilities returned to the user""" | ||
| lm = LM(model="gpt-4o-mini") | ||
| lotus.settings.configure(lm=lm) | ||
| result = sample_df.sem_filter("{Course Name} will be fun", return_probs=True, return_all=True) | ||
| print(result) | ||
| assert "probs_filter" in result.columns | ||
|
|
||
| for idx, row in result.iterrows(): | ||
| if row["filter_label"]: | ||
| assert row["probs_filter"] > 0.5 | ||
| else: | ||
| assert row["probs_filter"] <= 0.5 |
There was a problem hiding this comment.
Core tests are here
| vs = FaissVS() | ||
|
|
||
| lotus.settings.configure(lm=gpt_4o, helper_lm=gpt_4o_mini, rm=rm) | ||
| lotus.settings.configure(lm=gpt_4o, helper_lm=gpt_4o_mini, rm=rm, vs=vs) |
There was a problem hiding this comment.
Not related to my logic changes. But vs needs to be set here for the example to run, given the recent merge.
| cleaned_token = logprob.token.lower().strip() | ||
| if cleaned_token not in ["true", "false"]: |
There was a problem hiding this comment.
Needed to add this .lower().strip() cleaning in a few places so we can process True, True true, etc.
liana313
left a comment
liana313
left a comment
There was a problem hiding this comment.
For the most part looks good -- I left a couple of comments about naming. Once those are updated, we can also update the filter docs section
| new_df["raw_output" + suffix] = filtered_raw_outputs | ||
|
|
||
| if return_scores: | ||
| new_df["score"] = filtered_scores |
There was a problem hiding this comment.
we should add an index to the end of the output col labels, since if a user filters twice, there will be a naming conflict. we should also add a test for filtering twice
2 participants
Introduces
return_probs. Output can look likeOr
Depending on if
return_allis set.This PR also fixes a bug in the way that logprobs were analyzed for the True/False prob calculations.