COMPAS fairness analysis end-to-end example

[ashryaagr]

I have added a fairness analysis of COMPAS dataset. On my local system, this example is being rendered perfectly when I use Frankiln.serve()

[ashryaagr]

I have used VegaLite for plotting Count Plots. I have added it to Project.toml. But due to some system issues, I am unable to update Manifest.toml to the latest one available here. So I haven't committed it. Due to this the build is failing.

[tlienart]

Cool thanks for this! I'll look into the VegaLite story, will also ask someone else's opinion on this as Fairness is not my area

[vollmersj]

Thanks a lot for the pull request. In the tutorial you focus on assessing fairness of an AdaBoostClassifier. However the first step is to analysis the prediction from COMPAS itself following compas e.g.

Looking at the figure 8 we can see that COMPAS is fair regarding the FDR for race but as ProPublica found, the FPR for African-Americans is almost twice as the FPR for Caucasians. For age we observe the same results for the same two metrics: FDR results are fair but FPR for ¡25 is 1.6X higher than 25-45. On the other hand, if we consider false positive errors distribution considering Sex we observe the contrary: the model is fair for FPR but the FDR of Female is 1.34 times higher than for Male.

This should not be too hard since all of this is available in this notebook.

In particular the output of the risk score from COMPASS to assess its fairness.

COMPAS produces a risk score that predicts a person's likelihood of commiting a crime in the next two years. The output is a score between 1 and 10 that maps to low, medium or high. For Aequitas, we collapse this to a binary prediction. A score of 0 indicates a prediction of "low" risk according to COMPAS, while a 1 indicates "high" or "medium" risk.

Only in a second step I would look at other algorithms.

More extensive analysis can be found at aequitas.

Let me know if you want to discuss.