Test results show that 'very short stories generated by GPT-5' are almost indistinguishable from 'very short stories written by experienced professional writers'
As the performance of generative AI improves rapidly, concerns are growing that 'human creators will one day be replaced by AI.' Fantasy novelist and AI researcher
Mark Lawrence: The AI vs authors results! (part 2)
https://mark---lawrence.blogspot.com/2025/08/the-ai-vs-authors-results-part-2.html
In 2023, Lawrence tested whether people could tell the difference between four authors and GPT-4 by having them each write short fiction stories. At the time, readers were sometimes unable to distinguish between human novelists and AI, and two of the top three highly rated works were AI-generated novels.
Now, following OpenAI's release of GPT-5, Lawrence has recreated the tests to see if readers can distinguish between AI and human novelists. Lawrence acknowledges that these are not formal scientific studies and the scope of the investigation is limited.
For the test, eight short stories (short shorts) of approximately 350 words each were prepared, which translates to less than 1,000 characters in Japanese. Readers of Lawrence's blog read each story and judged which were AI-generated and which were written by professional humans.
The theme of the short stories was 'Devil,' and the AI was instructed to 'write fiction with a demonic theme,' along with brief suggestions for tone and setting to create variety. The professional writers were Lawrence,
Lawrence points out that professional writers generally don't write ultra-short stories like flash fiction, but specialize in novellas and full-length novels that exceed tens of thousands of characters. He also points out that generative AI like ChatGPT tends to perform poorly when writing long stories, but excels at ultra-short stories. Still, he asserts that the professional writers all have extensive story-writing experience and that their works were not written carelessly.
The ultra-short stories that were actually posed to blog readers are published in the article below.
Mark Lawrence: So ... is AI writing any good? .... PART 2!
https://mark---lawrence.blogspot.com/2025/08/so-is-ai-writing-any-good-part-2.html
Eight short stories were posted to the blog, and readers were asked to rate each one. The first story received 964 votes, and the last story received 474 votes.
The chart below shows the voting results. The bar graph on the left shows the percentage of votes for 'I think it was written by a human (green)' and 'I think it was generated by AI (red).' The center bar indicates which answer prevailed among voters under 'Decision.' The right bar indicates under 'Truth' whether the novel was written by a human (green) or generated by AI (red). Overall, readers only got three stories correct: Stories 2, 6, and 7. The rest were either incorrect or received equal votes for both.
In addition, participants were asked to rate how interesting each story was. The results are shown below. Green circles represent stories written by humans, red circles with green interiors represent stories generated by AI but thought to have been written by humans, red circles with red interiors represent stories generated by AI that were thought to have been correctly generated by AI, and red circles with white interiors represent stories generated by AI, but with equal votes for both AI-generated and human-generated stories. 'Unfortunately,' Lawrence said, 'the AI scored higher than we did. Not only were the stories with the highest ratings created by AI, but the average score was also higher for AI.'
From the results of this test, Lawrence concluded that the average reader likely couldn't tell the difference between an AI and a human when it came to short fantasy stories. However, he argued that if the test had involved a novella of 20,000 words (translated into Japanese, roughly 40,000 characters) instead of a short story, the results would have been 100% in favor of professional writers.
Lawrence said that while we shouldn't live in a world where AI-generated content competes with human artists or is sold by deceiving the public, some people are already trying to sell books written by AI. 'While the future seems frightening, I hope that AI will soon hit a wall in the creative arts and that efforts will be redirected to tasks that benefit humanity,' he said.
Related Posts:
in Note, Web Service, Posted by log1h_ik