The Right Tool for the Job: Open-Source Auditing Tools in Machine Learning
In recent years, discussions about fairness in machine learning, AI ethics
and algorithm audits have increased. Many entities have developed framework
guidance to establish a baseline rubric for fairness and accountability.
However, in spite of increased discussions and multiple frameworks, algorithm
and data auditing still remain difficult to execute in practice. Many
open-source auditing tools are available, but users aren't always aware of the
tools, what they are useful for, or how to access them. Model auditing and
evaluation are not frequently emphasized skills in machine learning. There are
also legal reasons for the proactive adoption of these tools that extend beyond
the desire for greater fairness in machine learning. There are positive social
issues of public perception and goodwill that matter in our highly connected
global society. Greater awareness of these tools and the reasons for actively
utilizing them may be helpful to the entire continuum of programmers, data
scientists, engineers, researchers, users and consumers of AI and machine
learning products. It is important for everyone to better understand the input
and output differentials, how they are occurring, and what can be done to
promote FATE (fairness, accountability, transparency, and ethics) in machine-
and deep learning. The ability to freely access open-source auditing tools
removes barriers to fairness assessment at the most basic levels of machine
learning. This paper aims to reinforce the urgent need to actually use these
tools and provides motivations for doing so. The exemplary tools highlighted
herein are open-source with software or code-base repositories available that
can be used immediately by anyone worldwide.