Don’t believe any writer who tells you they’re not worried about AI eating their lunch; they’re lying.
Many companies are actively assigning creative writing work to AI bots. ChatGPT has routinely taken over low-level copywriting work that would formerly have gone to freelancers on Fiverr; international ad agency BBDO is experimenting with using generative AI to write briefs.
We’re hearing more often from disgruntled writers complaining about work “drying up without explanation”, as agencies crunch numbers and decide a monthly subscription to an AI copy mill seems to make more financial sense than paying a pricier (and more emotionally fragile) human writer.
But does it really? How well does a content piece generated by a large-language model (LLM) AI hold up when compared to one written by an experienced human writer?
Let’s find out.
The matchup
To test this, we decided to pit a generative AI writing solution against one of our (human) professional writers.
In one corner, we have an experienced journalist with serious research chops. In the other corner, we have a custom GPT from ChatGPT which, as its press material says, “excels in crafting tailored, engaging content with a keen emphasis on quality, relevance, and maintaining a precise word count.”
To ensure a level playing field, both writers were provided with exactly the same brief, written with a significant amount of structural detail.
An excerpt of the brief follows below:
Target Audience: HR professionals, company leaders, and managers tasked with planning and implementing the transition from remote to on-site work.
Tone and approach: Friendly but professional — provide friendly, accessible advice to fellow HR professionals and executives without descending into jargon or taking an adversarial tone.
For this experiment, we assess both writers based on creativity, structure, and emotion:
Creativity and insight. Would either copy contain surprising insights (the stuff of “why didn’t I think of that before?”) and nuanced opinions? Both are just basic expectations of human writers (just ask any editor with over two decades’ experience), but are things that AI usually struggles with.
That’s not surprising, as AI only combines existing information in new ways; it often struggles to produce content that reflects truly novel ideas or personal insights.
Structural consistency, coherence, and flow. Would either writer be able to develop a coherent line of thinking over the length of the article? Human writers are able to maintain a consistent narrative or argumentative thread, making deliberate choices about structure and flow (managing words per paragraph, for instance, to ensure readability).
LLMs, on the other hand, tend to exhibit inconsistencies in style or logic in their writing, leaving the narrative to wander over longer stretches of break-free content.
Emotional depth and empathy. Good writing often reflects a deep understanding of human emotions, experiences, and the capacity for empathy. This depth comes from personal experience and the ability to emotionally connect with readers.
While AI models can mimic emotional expressions to some extent, their writing has lacked the depth of understanding and genuine empathy that comes from actual human experiences and emotions.
How they measured up
The introduction
A stark difference was already apparent within the first two sentences of each introduction.
One traffics in business abstractions, emotionally detached from the human situation and mostly expressed in clichéd turns of phrase. The other centers the narrative immediately in the reader’s personal context.
Read the excerpts from each introduction below: which one is likely to be derived from an AI database, and which one sounds more human?
| AI: “In the wake of global shifts and challenges, the professional landscape has undergone a transformation unlike any before, thrusting remote work into the spotlight and reshaping how organizations operate. As the world gradually adjusts to new norms, many companies are now facing the complex task of transitioning employees from remote to on-site work settings.” | Human: “It’s now become more common to find yourself in the comfort of your home on a workday, while your manager checks in from the office. A hybrid work setting, wherein employees may choose to either work from or in the office, has become a more common set- up for many workplaces post-pandemic.” |
The latter piece also includes hyperlinks to citations, which the former leaves out completely. (Helpful hint: the cutoff date for ChatGPT data is April 2023—any citations it provides may be hopelessly out of date for most current-events content.)
That’s just for the introduction. How do both writers tackle the article’s argument throughout their respective pieces?
Brevity and flow in the body
Comparing the brief with the resulting content offers a telltale sign of the AI hiding in plain sight.
The AI-generated piece on the left slavishly follows the bullet-point structure of the brief, down to copying the talking points per bullet and simply expanding on each point.
The elaborations in each section, too, feel lacking in any insight—flat corporate-style truisms that smack of the obvious and do little beyond expanding the word count.
Not so with the human-created piece, on the right. While still following the rules set by the brief, the human content creates a more streamlined structure, condensing the overall themes into only five sections that are easier to follow.
AI
| Human
|
It’s immediately obvious that the AI is more seriously constrained by the brief. Lacking the agency or the experience to experiment with the structure, the AI piece simply expands on the brief’s talking points, with passively phrased truisms that offer no unique insights into the problem.
Compare that to the human-created piece, where the brief is treated as a jumping-off point to interesting research-based insights.
AI: “Introducing hybrid work schedules offers a compromise that can ease the transition back to on-site work. Allowing employees to split their time between working remotely and in the office can help them gradually adjust to the change. This flexibility acknowledges the benefits of remote work while leveraging the advantages of on-site collaboration, creating a balanced approach that can boost morale and productivity.” | Human: “The return to the office does not need to be abrupt, experts say, as this could damage employee morale. Instead, organizations are advised to encourage autonomy among their employees, which allows them to have the flexibility they once enjoyed in a remote work setup. Gartner says, for example, organizations may choose to allow managers to have a say on how often their team has to go to work on a weekly or even yearly basis so that connections in the office feel more intentional. A project that demands more team meetings than usual may require employees to come to the office for a full week. Other projects, however, may not be as demanding so meetings in the office do not have to be weekly. Gartner says their research showed employees who had more autonomy at work were 2.3x more likely to stay in their organizations. Consulting firm Gallup also says employees are “most engaged” when teams collaboratively determine a hybrid work strategy.” |
The paragraph and section lengths, too, are literally a massive tell: the AI piece is eight sections long, with 130-180 words per section, with paragraph lengths of about 40-80 words each. The human-written piece is only five sections long, with similar word counts per section but more manageable paragraph lengths of 15-60 words each.
Finally: the conclusions for the respective pieces do a good job of summarising the points laid out in the preceding content (AI taking about 400 words more to do the same job compared to our human writer). But the AI ending feels flat and diffused, compared to the human-written piece which neatly summarises its argument regarding the need for patience among HR practitioners.
| AI: “This comprehensive approach to transitioning from remote to on-site work is designed to guide HR professionals, leaders, and managers through the process, providing actionable insights and strategies to make the transition as smooth and positive as possible for everyone involved.” | Human: “Patience, then, is key. HR practitioners must be equipped to craft a long-term strategy to ensure the success of the RTO. This way, both HR leaders and executives would have the time, and opportunity to analyze what setup works best for the organization, moving forward.” |
What we’ve learned
What does our little experiment tell us about the status quo of AI?
Any status quo in the technology sector is fleeting.
GPT-5 is hot on the heels of GPT-4, as are all the generative AI platforms trying to stake a claim in this AI gold rush: Microsoft’s Copilot, Google’s Gemini, as well as content-specific AI apps like Writer and Jasper.
So while we can rest easy that AI lacks the chops to handle long-form content, writers shouldn’t rest on their laurels. AI is already stealing low-intensity, high-output jobs like writing social media copy. Savvy AI developers can, and will, find a way to generate long-form content at lower costs eventually.
Some writers are sadly replaceable. However…
Editors will stay relevant for some time to come.
“Because of generative Al, editing matters more,” explains content marketer and author Ryan Law. “In a world where writing is commoditised, we need people able to structure, refine, and impart their sense of quality on writing.”
Consider the AI piece we generated. While it certainly followed the brief, its lack of insights and floppy structure would doom it as a content piece right out of the gate. An editor might conceivably shape it into something serviceable, adding the human insight and empathy that the AI piece sorely lacked.
Consider the time and effort you can afford to spend. Not including the time spent writing the brief, the AI piece took less than a minute to write. (And, sadly, it shows.) The human-written piece took a few days, but needed little editing to make it ready for primetime.
Low-intensity, high-volume content projects that you need right now? AI can probably do it for you.
Long-form content with unique insights on a technical topic? You need a human for that.
Fast, cheap, good: AI can certainly deliver on the first two, but not all of the three. Expect to pay good money to human writers to create readable, emotionally-impactful and insightful content, for the foreseeable future.
—
200+ industry-leading tech companies in Southeast Asia are happy clients of With Content. Join them and start delivering valuable content to your potential customers today.
