AI Content Quality: How to Tell Good AI Writing From Bad

Here is a test. Read these two passages and decide which is better writing:

Passage A is grammatically flawless. It contains no false claims. By a narrow definition, it is “good AI writing.” By any professional standard, it is nearly worthless — information-free, interchangeable, and incapable of moving a reader toward any useful understanding. Passage B, by contrast, earns its length. This distinction — between technically correct and genuinely useful — is what content quality researchers are actually trying to measure.

Why “AI-Generated” Tells You Nothing About Quality

The phrase “AI-generated content” spans an enormous quality range. A carefully prompted GPT-4o response on a specialized medical topic, reviewed by a physician, may contain more accurate and useful information than a rushed human draft. A hallucinated Claude response on a specific legal question may contain fabricated case citations that are confidently presented and subtly wrong. The origin label is not the quality label.

According to a 2025 content performance study by Semrush, which analyzed over 1.5 million blog posts, human-written content holds an advantage at every position in the top 10 Google search results — but the gap is not uniform. At positions 6–10, AI-assisted content (human-edited AI drafts) performs comparably to human-only content. At position 1–3, the advantage for human-written content widens substantially. Semrush's researchers attributed this primarily to information depth and demonstrable expertise — qualities that correlate with the author's domain knowledge, not the writing tool's sophistication.

This matters for how we define “quality.” Quality in content is not primarily a linguistic property — it is a utility property. Does the reader leave with accurate, specific, actionable information they did not have before? That question requires evaluating content on dimensions that go well beyond grammar.

The Six Dimensions of AI Content Quality

Academic NLP research and practical content evaluation frameworks have converged on a set of core quality dimensions. The Holistic Evaluation of Language Models (HELM) framework from Stanford's Center for Research on Foundation Models evaluates LLM output across several of these. Here is what each dimension means in practice:

1. Fluency: The Dimension AI Has Largely Solved

Fluency covers grammatical correctness, natural phrasing, and the absence of surface-level errors. Modern frontier LLMs — GPT-4o, Claude Sonnet, Gemini 1.5 Pro — achieve 95–98% grammatical accuracy on standard prose evaluation tasks, per the Stanford HELM leaderboard. This is roughly equivalent to a skilled human writer and significantly better than a proficient non-native English writer. Fluency is no longer a meaningful differentiator between good and bad AI writing, which means evaluating quality on fluency alone is useless. A fluent passage can still be empty, wrong, or misleading.

2. Factual Accuracy: The Dimension AI Consistently Struggles With

Factual accuracy — whether specific claims in the content are true — is where AI writing most frequently fails at a professional level. Stanford HAI's 2025 benchmark of AI writing quality, which evaluated outputs on complex queries across 12 knowledge domains, found AI-generated content had a 14.2% factual error rate on complex, domain-specific queries. Experienced human writers covering the same topics produced a 3.8% error rate. The gap was largest in rapidly evolving domains (regulatory policy, recent scientific research, legal precedents) where the model's training cutoff or knowledge retrieval creates gaps between what the model “knows” and what is currently true.

The error distribution matters too. AI factual errors tend to be confident and subtle — a number slightly wrong, a regulation cited from the wrong jurisdiction, a study described with the opposite conclusion — rather than obviously absurd. This makes them more dangerous than the gross hallucinations that appear in simple prompt tests, because they pass casual human review.

3. Coherence: The Long-Form Problem

Coherence refers to whether the content holds together as a logical whole — whether paragraphs connect, arguments develop, and the overall structure serves the reader's understanding. At the paragraph level, AI writing performs reasonably well. At the document level, particularly beyond 1,500 words, coherence degrades in characteristic ways: arguments restart without resolution, evidence is introduced and never synthesized, the conclusion fails to emerge from the body, and thematic threads are abandoned mid-article.

The MIT Computational Cognitive Science Group has studied long-form coherence in LLM output, finding that models demonstrate significantly lower scores on discourse coherence metrics (entity tracking, argument development, thematic consistency) compared to human experts once texts exceed approximately 2,000 words. The practical implication is that AI writing quality falls off faster in long-form content than it does in short-form, making human editorial oversight more critical as content length increases.

4. Information Density and Novelty

Information density measures how much genuinely new or specific information the reader receives per word. This is the dimension that most sharply separates Passage B from Passage A in the opening example. AI models trained on existing text have a well-documented tendency toward what researchers call “recombinative mediocrity” — producing content that correctly assembles known patterns without adding new information, synthesis, or perspective. The output is accurate but informationally flat.

Per a 2024 Axios-commissioned analysis of web content, AI-generated articles produce lower information density scores on average than human-written articles — they use more words to convey less specific content. The difference is most pronounced in explanatory and analytical content, and least pronounced in factual descriptions and summaries. Novelty — whether the content includes perspectives or synthesis not available in training data — is, by definition, something AI cannot easily generate. It can combine perspectives, but originating a truly novel analytical frame requires the kind of first-person experience and cross-domain reasoning that LLMs approximate but do not replicate.

5. Audience Calibration

Good writing is calibrated to its audience's knowledge level and needs. AI models default toward middle-register writing: more specialized than casual conversation, less technical than expert communication. Without explicit prompting, they tend to over-explain concepts that experts already know and under-explain nuances that novices need. Audience calibration in AI writing requires active prompt engineering — specifying the audience's expertise level, role, and goals — rather than emerging naturally from the writing process as it does for experienced human writers who know their readers.

6. Appropriate Epistemic Hedging

Expert writing signals uncertainty appropriately: “the evidence suggests,” “in most but not all cases,” “this remains contested.” AI writing tends toward overconfident assertion — presenting uncertain claims with the same declarative confidence as established facts. OpenAI's own 2024 safety research found that GPT-4 class models significantly underperform on calibration tasks, claiming higher certainty than the evidence warrants across multiple domains. For publishers and educators, this dimension is particularly critical: content that presents contested or uncertain information with false confidence is actively harmful, regardless of grammatical quality.

AI Content Quality Across Major Tools: A Comparative View

Not all AI writing tools perform equally across these dimensions. The following table summarizes independent benchmark performance across major tools, drawing on the Stanford HELM project, OpenAI's evals, and independent research:

These figures are illustrative ranges drawn from available benchmarks; individual performance varies significantly by domain, prompt quality, and task type. The consistent finding across studies is that no current AI tool reliably matches expert human writers on factual accuracy and long-form coherence in specialized domains — the two dimensions that most determine whether content is genuinely valuable rather than merely readable.

The Market Reality: Most “AI Content” Is Low Quality, and Readers Are Starting to Notice

The flood of AI content across the web has not uniformly improved information quality — and there is accumulating evidence that readers and search engines are developing sensitivity to its characteristic flaws. A 2025 content analysis by Graphite found that over 5% of newly published web articles show signatures consistent with AI generation without significant human editing. The same analysis found those articles had 40% lower average time-on-page than comparable human-written content — a proxy for whether readers found the content valuable enough to finish.

According to Semrush's comprehensive 2025 analysis of over 1.5 million articles, AI-generated content without substantial human editing had lower average engagement metrics across all measured dimensions — time on page, scroll depth, social shares, and return visits. Only 14% of the top-ranking pages in competitive search positions were primarily AI-generated, even though AI tools are now used in content creation by the majority of content teams. The implication is that ranking correlates with the human editorial investment in AI-assisted content, not with AI use per se.

Google's public guidance on AI content has consistently focused on the same dimensions researchers measure: it is indifferent to origin and concerned with usefulness, accuracy, and demonstrated expertise. The March 2024 core update specifically targeted low-quality, unhelpful content regardless of how it was produced — and independent analyses by multiple SEO research firms found that AI-heavy content farms were among the most affected properties, losing 60–90% of their organic traffic.

Five Practical Signals That Separate Good AI Writing From Bad

For editors, publishers, and educators evaluating AI-generated content, these are the most reliable diagnostic signals:

Signal 1: Specific Numbers vs. Vague Quantifiers

Low-quality AI writing uses quantifiers without specificity: “many studies show,” “significant growth,” “a large percentage.” Good AI writing — and good writing generally — uses named sources and specific figures: “per the Bureau of Labor Statistics October 2025 report, 73% of employers...” Vague quantification is the most reliable surface signal of low-information content. Use EyeSift's text analyzer to assess information density before publishing.

Signal 2: Paragraph-to-Paragraph Logical Progression

Read the first and last sentences of each paragraph. In coherent writing, each paragraph advances a specific claim or develops a specific thread, and the relationship between adjacent paragraphs is explicit. In low-quality AI writing, paragraphs often restate the same point with different phrasing, or shift topics without transitional logic. Reading paragraph-final sentences in sequence is a fast diagnostic: if they do not build on each other, the document lacks genuine coherence regardless of how fluent individual sentences are.

Signal 3: Presence of Counterarguments and Qualifications

Expert writing engages with complexity. It acknowledges when evidence is mixed, when expert opinion is divided, when the data supports a conclusion “in most but not all cases.” Low-quality AI writing tends toward unambiguous, hedge-free assertion — not because the topic is genuinely simple, but because the model defaults to confident statement. If a 2,000-word article on a genuinely complex topic contains no qualifications, no acknowledgment of competing perspectives, and no explicit discussion of limitations, it has been generated at a quality level insufficient for professional use.

Signal 4: Verifiable Claims

Check five specific factual claims in any AI-generated article against primary sources. This takes five minutes and catches the characteristic AI error mode: subtly wrong specifics that survive surface-level reading. Common failure patterns include: statistics from the right study but with the wrong number, regulations cited from the correct jurisdiction but outdated version, case law described with the outcome reversed. Turnitin's AI research team has documented that factual errors in AI content cluster most heavily in citations, statistics, and descriptions of regulatory or legal frameworks.

Signal 5: The “So What” Test

After reading the conclusion, ask: what does a reader now know or be able to do that they could not before? Low-quality AI writing describes topics without advancing understanding. Good writing — AI-assisted or human — produces a concrete change in the reader's knowledge state. If the answer to “so what” is “I now know this topic exists,” the content has failed the basic quality threshold regardless of its technical execution.

Evaluating AI Writing Quality in Academic Contexts

For educators, AI content quality assessment has an additional layer: evaluating whether AI-generated academic submissions demonstrate the specific learning outcomes the assignment was designed to produce. A perfectly fluent AI essay may be educationally hollow if it bypasses the analytical process the assignment was meant to develop. Turnitin's 2025 academic integrity report found that 22% of academic submissions contained some proportion of AI-generated content — up from 11% in 2023 — and that the proportion was highest in writing-intensive undergraduate courses.

The most educationally concerning AI submissions are not those that hallucinate obviously — those are easy to catch. They are the ones that accurately summarize known content without demonstrating the student's analytical reasoning, integration of course-specific material, or engagement with the assignment's specific intellectual demands. Detecting this requires assignment-specific evaluation: did the student engage with the specific readings assigned? Is there evidence of the individual's reasoning process? Does the argument reflect the course's particular intellectual frameworks? These are questions that require human judgment and cannot be answered by any automated detector.

Our guide for teachers on AI detection covers how to design assignments that are resistant to hollow AI completion and how to evaluate submissions for genuine intellectual engagement.

The Quality Floor Is Rising — But So Are Expectations

It would be wrong to conclude from this analysis that AI writing quality is uniformly low. The best AI-assisted content — carefully prompted, domain-expert reviewed, editorially refined — can approach expert human quality on most dimensions. The practical challenge is that achieving this quality requires significant human investment, which defeats the low-cost assumption that drives much AI content adoption.

The frontier models are improving on factual accuracy, long-form coherence, and epistemic calibration with each generation. But reader and search engine expectations are rising in parallel — and the bar for “good enough” in most competitive content categories is substantially higher than what raw AI output reliably delivers. The organizations getting value from AI writing are those that treat it as a tool for accelerating expert-directed content production, not as a replacement for the expert judgment that determines quality.

For publishers evaluating incoming AI-assisted content, the AI detection accuracy benchmarks provide a useful reference for what current tools can and cannot reliably distinguish — complementing quality assessment with origin assessment where both matter.

Frequently Asked Questions