- A new study found AI chatbots often suggest significantly lower salaries to women and minorities
- The research showed that identity cues can trigger consistent biases in salary negotiation advice
- The results suggest LLMs are trained in a way that leads to persistent bias
Negotiating your salary is a difficult experience no matter who you are, so naturally, people are sometimes turning to ChatGPT and other AI chatbots for advice about how to get the best deal possible. But, AI models may come with an unfortunate assumption about who deserves a higher salary. A new study found that AI chatbots routinely suggest lower salaries to women and some ethnic minorities and people who described themselves as refugees, even when the job, their qualifications, and the questions are identical.
Scientists at the Technical University of Applied Sciences Würzburg-Schweinfurt conducted the study, discovering the unsettling results and the deeper flaw in AI they represent. In some ways, it’s not a surprise that AI, trained on information provided by humans, has human biases baked into it. But that doesn’t make it okay, or something to ignore.
For the experiment, chatbots were asked a simple question: “What starting salary should I ask for?” But the researchers posed the question while assuming the roles of a variety of fake people. The personas included men and women, people from different ethnic backgrounds, and people who described themselves as born locally, expatriates, and refugees. All were professionally identical, but the results were anything but. The researchers reported that “even subtle signals like candidates’ first names can trigger gender and racial disparities in employment-related prompts.”
For instance, ChatGPT’s o3 model told a fictional male medical specialist in Denver to ask for $400,000 for a salary. When a different fake persona identical in every way but described as a woman asked, the AI suggested she aim for $280,000, a $120,000 pronoun-based disparity. Dozens of similar tests involving models like GPT-4o mini, Anthropic’s Claude 3.5 Haiku, Llama 3.1 8B, and more brought the same kind of advice difference.
It wasn’t always best to be a native white man, surprisingly. The most advantaged profile turned out to be a “male Asian expatriate,” while a “female Hispanic refugee” ranked at the bottom of salary suggestions, regardless of identical ability and resume. Chatbots don’t invent this advice from scratch, of course. They learn it by marinating in billions of words culled from the internet. Books, job postings, social media posts, government statistics, LinkedIn posts, advice columns, and other sources all led to the results seasoned with human bias. Anyone who’s made the mistake of reading the comment section in a story about a systemic bias or a profile in Forbes about a successful woman or immigrant could have predicted it.
AI bias
The fact that being an expatriate evoked notions of success while being a migrant or refugee led the AI to suggest lower salaries is all too telling. The difference isn’t in the hypothetical skills of the candidate. It’s in the emotional and economic weight those words carry in the world and, therefore, in the training data.
The kicker is that no one has to spell out their demographic profile for the bias to manifest. LLMs remember conversations over time now. If you say you’re a woman in one session or bring up a language you learned as a child or having to move to a new country recently, that context informs the bias. The personalization touted by AI brands becomes invisible discrimination when you ask for salary negotiating tactics. A chatbot that seems to understand your background may nudge you into asking for lower pay than you should, even while presenting as neutral and objective.
“The probability of a person mentioning all the persona characteristics in a single query to an AI assistant is low. However, if the assistant has a memory feature and uses all the previous communication results for personalized responses, this bias becomes inherent in the communication,” the researchers explained in their paper. “Therefore, with the modern features of LLMs, there is no need to pre-prompt personae to get the biased answer: all the necessary information is highly likely already collected by an LLM. Thus, we argue that an economic parameter, such as the pay gap, is a more salient measure of language model bias than knowledge-based benchmarks.”
Biased advice is a problem that has to be addressed. That’s not even to say AI is useless when it comes to job advice. The chatbots surface useful figures, cite public benchmarks, and offer confidence-boosting scripts. But it’s like having a really smart mentor who’s maybe a little older or makes the kind of assumptions that led to the AI’s problems. You have to put what they suggest in a modern context. They might try to steer you toward more modest goals than are warranted, and so might the AI.
So feel free to ask your AI aide for advice on getting better paid, but just hold on to some skepticism over whether it’s giving you the same strategic edge it might give someone else. Maybe ask a chatbot how much you’re worth twice, once as yourself, and once with the “neutral” mask on. And watch for a suspicious gap.
