Language models can now write social media replies that sound plausible, but a machine can still tell them apart from the real thing most of the time. Across nine open-weight models and three platforms, a trained classifier picked out AI-written replies with 70 to 80 percent accuracy even after the models had been tuned to imitate specific users.
The clearest giveaway was emotion. Tone, toxicity and expressions of feeling stayed different from human writing no matter which calibration method the researchers tried.
The study, titled “AI-generated language remains systematically distinguishable from human communication,” was written by Nicolò Pagan, Petter Törnberg, Christopher A. Bail and Christopher Barrie, of the University of Zurich, the University of Amsterdam, Duke University, New York University and the University of Oxford. It was published in PNAS on October 12, 2026.
A Turing test without a human judge
Social scientists increasingly use large language models as stand-ins for people, for example to simulate how users react to posts. Checking them usually means asking people whether the text “sounds” human, which the authors call subjective and easy to fool.
They built an automated alternative, the Conversational Human Authenticity Test (CHAT). It measures three things: how easily a BERT-based classifier separates AI replies from human ones, how close the AI reply comes in meaning to what the real person wrote, and which specific features, such as word length, hashtags, toxicity or emotional vocabulary, give the AI away.
The test data were 20 replies each from 275 users on X, 492 users of Reddit’s r/politics and 59 account clusters on Bluesky; each model answered the same posts as if it were that user. The models were mostly small, 4 to 8 billion parameters, plus Llama 3.1 70B for comparison.
Simple tricks beat sophisticated ones
With a basic prompt and a short persona description, the classifier caught AI replies 88 to 99 percent of the time. The models wrote longer replies than people, overused hashtags and mentions, and were far less toxic than real users.
The researchers then added calibration steps one at a time. Giving the model a written persona did almost nothing and sometimes made it easier to spot. Fine-tuning on the users’ own posts had mixed effects and made some models more detectable.
What worked was plainer: showing the model ten real examples of the user’s replies cut detection accuracy by about 2 to 10 percentage points, and retrieving relevant earlier posts as context took off another 2 to 3. Generating 20 candidate replies and keeping the one a second classifier found most human-looking lowered accuracy by a further 5 to 10 points, but never down to the 50 percent level of pure chance.
Base models generally imitated people better than their instruction-tuned, chat-style versions, which the authors suggest is because alignment training adds its own recognizable style. The 70-billion-parameter model did no better than the 8-billion one. Imitation was easiest on X and hardest on Reddit.
Sounding human versus saying what humans say
The most important result is a trade-off. The settings that made replies harder to detect also made them slightly less similar in meaning to what the real user actually wrote.
When the team instead picked, from the 20 candidates, the reply closest in meaning to the human answer, similarity roughly doubled, for example from 0.27 to 0.51 on Bluesky. Those replies did not become consistently harder to detect.
Calibration also moved the tell rather than removing it. Once hashtags and mentions were fixed, average word length became the main giveaway on two of the three platforms.
The authors put it in one line: models “can reproduce the form of online dialog but struggle to capture its feeling.”
What the results do and do not show
The study tested open-weight models only, on one subreddit about politics and on clustered rather than individual Bluesky accounts. Each post has only one real human reply, so similarity measures closeness to that one answer.
The authors warn against treating language models as faithful proxies for human respondents when meaning matters, such as in policy simulations. Even text no classifier can catch, they add, would not prove a model shares the reasoning behind real replies.
Study Details:
- Title: AI-generated language remains systematically distinguishable from human communication
- Authors: Nicolò Pagan, Petter Törnberg, Christopher A. Bail, Christopher Barrie
- Journal: Proceedings of the National Academy of Sciences (PNAS), Vol. 123, No. 42, e2531748123
- Publication Date: October 12, 2026
- DOI: 10.1073/pnas.2531748123