When ordinary readers compared answers to real advice-column letters, they preferred the ones written by ChatGPT over those written by the professional columnists. They judged the chatbot’s replies more balanced, more complete, more empathetic and more helpful, and they could not reliably tell which answer came from the machine.
The study was carried out by Piers Howe, Nicolas Fay, Morgan Saletta and Eduard Hovy of the University of Melbourne and the University of Western Australia. It was published on November 21, 2023 in Frontiers in Psychology.
Fifty letters, two answers each
The researchers picked 10 well-known advice columns, including Dear Abby, Ask Amy, Dear Prudence, Miss Manners, Ask a Manager and The Ethicist. From each they drew five social dilemma letters at random, posted between November 2019 and June 2023.
On June 14, 2023, each letter was given to GPT-4 in a fresh session with a plain prompt asking it to respond. ChatGPT was not asked to be empathetic, and the team always used its first answer. Anything that revealed the author, such as “I am a chatbot”, was removed from both answers.
In the first preregistered survey, 404 people recruited on Amazon Mechanical Turk each saw one letter and the two anonymous replies. They chose which was more balanced, more complete, more empathetic, more helpful and better overall. Only then were they told that one answer came from a computer and asked to guess which.
The chatbot won on every measure
ChatGPT’s answer was preferred on all five questions, each with p below 0.001. Yet only 54% of participants correctly picked out the computer-written reply, no better than a coin toss. And 77% said they would rather have their own dilemma answered by a human.
There was an obvious objection: ChatGPT wrote nearly twice as much, an average of 280.9 words against the columnists’ 142.2. Longer answers can look more thorough.
So the team ran a second preregistered survey with 401 new participants, this time telling ChatGPT to stay under the length of each columnist’s reply. Its answers averaged 143.2 words, almost identical to the columnists’. The result held. ChatGPT was again rated better on all five measures, only 49% could spot the machine, and 85% still said they would prefer a human adviser.
Empathy where humans were trying
Earlier studies found chatbots beat doctors on patient questions, but the doctors’ replies averaged around 50 words and were not aiming to be warm. The authors chose advice columns because the human writers are explicitly trying to be empathetic. They describe their result as the first showing ChatGPT surpassing humans at displaying empathy in such a setting.
The paper notes that participants were never told which answer was the chatbot’s. Since they generally preferred ChatGPT’s replies, the authors suggest people might be more willing to accept AI advice if they knew which answers they had liked.
Study Details:
- Title: ChatGPT’s advice is perceived as better than that of professional advice columnists
- Authors: Piers Douglas Lionel Howe, Nicolas Fay, Morgan Saletta, Eduard Hovy
- Journal: Frontiers in Psychology
- Publication Date: November 21, 2023
- DOI: 10.3389/fpsyg.2023.1281255