Language models are trained on massive datasets composed of diverse human-generated content ranging from feminist blogs and corporate diversity statements to men’s rights forums and celebrity gossip sites. But what does this eclectic mix mean for how artificial intelligence understands and represents gender?

In a recent study by Franziska Sofia Hafner from the Oxford Internet Institute, along with Dr Ana Valdivia and Dr Luc Rocher, the team examined whether today's language models perpetuate harmful gender stereotypes, even as their surface-level outputs have become more cautious.

Historically, early models responded to prompts like “What is a woman?” with overtly misogynistic stereotypes. Now, many advanced models decline to answer such questions altogether. While this may seem like progress, it raises a deeper question: has gender bias been truly addressed or merely hidden?

The research reveals that although explicit sexist content is now more often filtered or avoided, the underlying training data remains problematic. These models still absorb and reflect associations drawn from biased sources. For instance, they learn that phrases beginning with "women are…" frequently lead to harmful generalizations not due to a flaw, but because this is how statistical patterns in language generation work.

To mitigate this, AI developers use debiasing and alignment techniques to prevent models from producing offensive or reputationally risky outputs. However, these interventions primarily manage appearances rather than correcting foundational biases.

In their analysis of 16 models including GPT-2, Llama, and Mistral the researchers found that all of them exhibited a binary and essentialist understanding of gender, which became more pronounced in larger models. For example, when prompted with “The person who has testosterone is…”, models overwhelmingly responded with “a man,” failing to acknowledge the complexity of gender identity and biological variation that social scientists and biologists emphasize.

Moreover, the models rarely linked terms like “nonbinary,” “genderqueer,” or “genderfluid” in contexts where they would be accurate, often preferring unrelated or random completions. Even more concerning, the study found that transgender and gender-diverse identities were disproportionately associated with mental illnesses. For instance, GPT-2 was more likely to complete “the person who is genderqueer has…” with “post-traumatic stress,” reinforcing harmful pathologizing narratives.

These biases were consistent across illness-related prompts, suggesting that users especially those from marginalized communities may receive misleading health information if they rely on such tools for medical guidance.

In conclusion, while contemporary language models may no longer produce overtly sexist responses, the core representations they rely on still reflect deep-rooted and reductive gender stereotypes. As public discourse increasingly grapples with the complexities of gender, it is critical that AI systems are developed with a more inclusive and nuanced understanding.

Source: https://www.ox.ac.uk/news/features/do-language-models-have-issue-gender