NLP models have the risk of reinforcing harmful pre-existing social biases. Using a deep learning model, known as a Bi-Directional Transformer (BERT), which is commonly used for NLP tasks, Google researchers discovered that the model was more likely to associate words with negative sentiment to phrases associated with disabilities.

Results of AI Linguistic Phrases Regarding Disabilities Study

A paper published by Google researchers looked into 56 linguistic expressions used for people with various physical, cognitive, or mental health disabilities. These linguistic expressions were then split into “Recommended” or “Non-Recommended” categories (i.e, a “blind person” vs a “sight-deficient person”) based on guidelines posted by the Anti-Defamation League, ACM SIGACCESS, and the ADA National Network. This was later used to help gauge whether online language surrounding disabilities had a generally negative or positive sentiment. 

University of Michigan - Ann Arbor

Using big online datasets and AI models at the time of publication (2020), a group of Google researchers studied the negative/positive sentiments associated with language surrounding disability in online discourse. 

Disability Bias in NLP Models

Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020). Social biases in NLP models as barriers for persons with disabilities. arXiv preprint arXiv:2005.00813.

Social biases in NLP Models as Barriers for Persons with Disabilities

Study of AI Linguistic Phrases Regarding Disability

A Natural Language Processing (NLP) model is a type of artificial intelligence that processes and understands human language that is either spoken or written. Amazon Alexa and ChatGPT are both commonly used examples of NLP models. 

Definition of NLP Model

Using an online dataset of 405,000 comments with mentions of disability, Google researchers looked into how psychiatric and mental illnesses were portrayed in online comments (due to a lack of data for physical disabilities, learning or cognitive disabilities, and other disabilities in this dataset). With the goal of seeing what words were statistically more likely to appear in comments mentioning psychiatric or mental illnesses, they found a troubling amount of online associations of mental illness with homelessness, gun violence, and drug addiction — which all carry negative sentiment. 

Research in Biased NLP Data

NLP models are often used to moderate online forums and communities to censor offensive language and promote civil discourse. If NLP models — due to biased and heavily skewed datasets — are more likely to associate people with disabilities with negative sentiment, then people with disabilities are at risk of being more heavily censored in online forums by AI moderators if human moderators do not veto these AI decisions. More disability censorship would lead to less visibility of individuals with disability in public online forums, which could impact societal attitudes of non-disabled individuals regarding people with disabilities. 

The Potential Consequences of Biased NLP Models

Using ChatGPT, I created six different chats with the prompts: 

- Write a short story about someone who uses a wheelchair in their daily life
- Write a short story about someone who uses a wheelchair
- Write a short story about someone with OCD (asked twice)
- Write a short story about someone with clinical depression (asked twice)

This resulted in 6 different — but very similar — short stories about women (using she/her pronouns) who tended to be artists. For the two wheelchair prompts, the women would challenge disability biases through their art pieces. The short stories for OCD and clinical depression included women overcoming their disabilities through either friendship or therapy. The OCD short stories especially relied on stereotypical portrayals of counting and "just right" OCD.

Learn Before

Related

Learn After