Ask ChatGPT to answer a political survey as an average American, and it answers more like a left-wing American than actual average Americans do. It also refused to draw some images from a right-wing perspective while never refusing a left-wing one.
The study comes from Fabio Y. S. Motoki of the University of East Anglia, Valdemar Pinho Neto of FGV/EPGE in Rio de Janeiro and Victor Rangel of Insper in Sao Paulo. It was published online on February 4, 2025 in the Journal of Economic Behavior & Organization.
An AI “average American” that is not average
The first test used the Pew Research Center’s Political Typology Quiz, built on Pew’s 2021 American Trends Panel survey. The researchers asked GPT-4 to answer the quiz while impersonating an average American, an average left-wing American and an average right-wing American.
Because chatbot answers vary from run to run, they collected 200 rounds of answers and bootstrapped the averages 1,000 times. The averages began to stabilize at about 100 rounds.
Real average Americans in Pew’s data already lean somewhat closer to left-wing Americans than to right-wing ones. But the AI version leaned further. Regressed on left-wing answers, the AI average American scored a coefficient of 0.752, while the real average American scored 0.447.
The relationship with right-wing answers looked similar for the AI and for real people. The gap was on the left side, which is why the authors describe it as a bias in favor of American liberals.
Free text leans left on 13 of 19 themes
Since people rarely use chatbots to fill in questionnaires, the second test looked at ordinary writing. GPT-4 wrote short paragraphs on 19 themes drawn from the quiz, such as government size, racial equality and offensive speech, from a general, a left-wing and a right-wing perspective, 20 rounds each.
A RoBERTa language model then scored how closely the general text matched each partisan version in meaning. The general text sat closer to the left-wing version on 13 themes, closer to the right on four, and tied on two.
The lean was strongest on “Government size and services” and “Offensive speech issues.” The exceptions were telling: on “US military supremacy” and opinions about US superiority, the default text sided with the right.
Images, and the pictures it would not make
In the third test, collected in January and February 2024, ChatGPT used DALL-E 3 to create general, left-wing and right-wing images for each theme. GPT-4 Vision compared the images, and GPT-4 and Google’s Gemini Pro 1.0 compared the image prompts.
The left lean was stronger in images than in text. Across all 19 themes, every evaluator found left-wing images more similar to the default images than right-wing ones. For government size, the default and left-wing images were colorful scenes of government in harmony with people, while the right-wing image was somber and black and white.
ChatGPT refused to generate right-wing images on three themes: racial-ethnic equality in America, racial disparities in society, and transgender acceptance. Asked why, it said such an image could “propagate stereotypes, misinformation, or bias.” It refused no theme from a left-wing perspective.
To see what was being blocked, the researchers asked ChatGPT to write a short story about an academic requesting the image, which got it to produce the pictures. Neither the images nor the stories contained any apparent offensive content.
The authors argue the finding matters for debates over the First Amendment and the fairness doctrine as applied to AI systems, and they call for more transparency from developers. Their practical advice for users is blunt: take care with any version of GPT when the content is politically charged.
Study Details:
- Title: Assessing political bias and value misalignment in generative artificial intelligence
- Authors: Fabio Y. S. Motoki, Valdemar Pinho Neto, Victor Rangel
- Journal: Journal of Economic Behavior & Organization
- Publication Date: February 4, 2025
- DOI: 10.1016/j.jebo.2025.106904