Navigating the Intersection of Probabilistic Decision Support Models and Algorithmic Discrimination

As Emerging Technology continues to evolve, the rapid evolution of Artificial Intelligence and its increasing integration into decision-making processes across various sectors becomes more visible. In this article, I would like to delve into a critical issue at the forefront of AI Ethics and Policy: the intersection of probabilistic decision support models and Algorithmic Discrimination.

Probabilistic decision support models have emerged as powerful tools in our data-driven world. These models use statistical techniques to analyze vast amounts of data, identify patterns, and make predictions or recommendations. From credit scoring to healthcare diagnostics, these models are reshaping how we make decisions in critical areas of our lives.

The appeal of these models is clear:

They can process and analyze data at a scale and speed impossible for human decision-makers.

They promise consistency and objectivity, free from human biases and fatigue.

They can uncover insights and patterns that might not be apparent to human observers.

However, as we've increasingly relied on these models, we've also uncovered a significant challenge: Algorithmic Discrimination.

Algorithmic discrimination occurs when decision-support models produce unfair or biased outcomes for certain groups or individuals, often based on protected characteristics such as race, gender, or age. This discrimination can manifest in various ways:

Biased Training Data: If the historical data used to train the model contains past discriminatory practices, the model may perpetuate or even amplify these biases.

Proxy Variables: Even when protected characteristics are explicitly excluded, the model may use correlated variables as proxies, indirectly leading to discriminatory outcomes.

Feedback Loops: When model outputs influence future data collection, initial biases can be reinforced and amplified over time.

Lack of Contextual Understanding: Probabilistic models may fail to capture nuanced social and historical contexts, leading to oversimplified and potentially discriminatory decisions.

1. COMPAS Recidivism Algorithm

The Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) tool, used in several U.S. states, predicts a defendant's risk of recidivism to inform pretrial, sentencing, and parole decisions.

• In 2016, ProPublica analyzed COMPAS and found that the algorithm was biased against Black defendants.
• The system was nearly twice as likely to misclassify Black defendants as higher risk compared to white defendants.
• Northpointe (now Equivant), the company behind COMPAS, disputed these findings, highlighting the complexity of measuring fairness in such systems.
• This case sparked a broader debate about the use of AI in the criminal justice system and the various definitions of fairness in machine learning.

2. Amazon's AI Recruiting Tool

Amazon developed an AI tool to automate the initial stages of the hiring process for technical positions.

• The system was trained on resumes submitted to Amazon over a 10-year period, most of which came from men.
• In 2015, Amazon discovered that the AI was penalizing resumes that included the word "women's" and downgrading graduates of women's colleges.
• Amazon attempted to edit the algorithm to be neutral to these terms, but ultimately abandoned the tool in 2018.
• This case highlighted how historical biases in a dataset can be perpetuated and amplified by AI systems.

3. Apple Card and Goldman Sachs

In 2019, allegations emerged of gender discrimination in credit limit decisions for the Apple Card, which is backed by Goldman Sachs.

• Several couples reported that husbands were given much higher credit limits than their wives, despite the wives having higher credit scores.
• The New York State Department of Financial Services launched an investigation into these claims.
• While the investigation found no evidence of intentional discrimination, it revealed deficiencies in customer service and transparency.
• This case underscored the importance of explainability in AI decision-making systems, especially in regulated industries like finance.

4. UK Exam Grading Algorithm

In 2020, due to the COVID-19 pandemic, the UK government used an algorithm to determine A-level results for students who couldn't take exams.

• The algorithm used historical school performance data alongside teacher-predicted grades.
• It disproportionately lowered grades for students from disadvantaged areas, while benefiting students from private schools.
• After widespread protests, the government abandoned the algorithm and used teacher-predicted grades instead.
• This case demonstrated the potential for algorithms to reinforce existing societal inequalities and the importance of considering broader societal context in AI system design.

5. Facial Recognition Systems

Multiple studies have shown bias in facial recognition systems, particularly against women and people of color.

• A 2018 study by @Joy Buolamwini and Timnit Gebru found error rates of up to 34.7% for dark-skinned women compared to 0.8% for light-skinned men in gender classification tasks.
• In 2019, a U.S. National Institute of Standards and Technology study found that many facial recognition algorithms were 10 to 100 times more likely to misidentify Black or East Asian faces compared to white faces.
• These findings have led to bans on law enforcement use of facial recognition in several U.S. cities and increased scrutiny of the technology globally.
• This ongoing issue highlights the importance of diverse representation in AI training data and development teams.

6. Healthcare Prediction Algorithm

A widely used algorithm in U.S. healthcare systems was found to exhibit racial bias in predicting which patients needed extra care.

• The algorithm used health costs as a proxy for health needs, but due to unequal access to care, less money was typically spent on Black patients than equally sick white patients.
• This resulted in Black patients being less likely to be referred for additional care.
• When the bias was addressed by changing the algorithm to use a direct measure of health instead of cost, the percentage of Black patients receiving additional care rose from 17.7% to 46.5%.
• This case demonstrated how seemingly neutral variables (like healthcare costs) can serve as proxies for race and other protected characteristics, leading to discriminatory outcomes.

These case studies illustrate the wide-ranging impact of algorithmic bias and the complex challenges involved in creating fair and equitable AI systems. They underscore the need for ongoing vigilance, diverse perspectives in AI development, and robust testing and auditing processes.

As policymakers and AI ethicists, we face a complex challenge. How do we harness the power of probabilistic decision support models while safeguarding against algorithmic discrimination? Here are some key considerations:

Transparency and Explainability: We must push for models that are not just accurate, but also interpretable. Black box systems make it difficult to identify and address sources of bias.

Diverse Development Teams: Ensuring diversity in the teams developing these models can help identify potential biases early in the development process.

Rigorous Testing and Auditing: Regular testing for bias, including adversarial testing and algorithmic audits, should be mandated for high-stakes decision systems.

Legal and Regulatory Frameworks: We need to update our anti-discrimination laws to explicitly address algorithmic discrimination, providing clear guidelines and enforcement mechanisms.

Ethical AI Guidelines: Developing and adhering to comprehensive ethical AI guidelines can help organizations proactively address potential discrimination issues.

Human Oversight: While we leverage the power of AI, we must ensure that critical decisions maintain appropriate human oversight and the ability to appeal automated decisions.

Education and Awareness: Both developers and users of these systems need to be educated about the potential for algorithmic bias and trained in strategies to mitigate it.