Skeptic’s Checklist for AI-Generated Sources & Interpretations
A tool for epistemic vigilance in the age of confident machines.
I. Source Integrity
Do the citations actually exist?
‣ Search titles, DOIs, or URLs—are they real, accessible, and relevant?
Are the sources credible for the topic?
‣ Peer-reviewed journal, institutional website, domain expert? Or low-signal content farm?
Are links live and appropriately cited?
‣ Does the link point to the thing it claims to support—or is it adjacent or misleading?
II. Interpretive Fidelity
Is the AI quoting accurately—in spirit as well as in letter?
‣ Watch for epistemic drift: cautious language like “may suggest” becoming “proves,” or nuanced argument flattened into assertive summary.
Is meaning retained across summarization?
‣ Are important limitations, caveats, or conditions preserved—or lost in translation?
Does the interpretation overstate the source’s position?
‣ Are correlation and causation being collapsed? Are normative claims added to descriptive data?
III. Synthesis Validity
Does the answer stitch together disparate sources appropriately?
‣ Look for false coherence—seamless synthesis of sources that weren’t meant to align.
Is the AI drawing conclusions that actually follow from the evidence?
‣ Beware interpretive leaps where the AI connects dots never connected in the sources.
Ask for a reversal: can it steelman the opposite view?
‣ Use a Synthesis Challenge to test depth: “Now argue the opposite using the same sources.” Inability to do so indicates shallow pattern-matching.
IV. Temporal Coherence
Are the sources from compatible time periods?
‣ Is a 2012 study being treated as equivalent to a 2023 meta-analysis?
Is scientific or scholarly evolution acknowledged?
‣ Are outdated paradigms treated as current? Has the field moved on?
Does the synthesis respect intellectual context?
‣ Look for anachronistic flattening—blurring between different stages of an idea’s development.
V. Framing & Prompt Bias
Was the AI primed with a leading or one-sided question?
‣ “Why is X bad?” often elicits cherry-picked, unbalanced responses.
Are opposing views or uncertainties acknowledged?
‣ If the response reads like an essay with one conclusion, ask: Where’s the counterpoint?
Is this a summary, an argument, or a simulation of consensus?
‣ Check for genre confusion—the AI may simulate scholarly tone while delivering marketing logic.
VI. Trust Signals & Camouflage
Is the answer too clean?
‣ When the synthesis feels frictionless, question whether it’s too neat to be true. Truth often resists easy packaging.
Does the tone match the certainty of the evidence?
‣ Confident prose + equivocal sources = red flag.
Are you being lulled into trust?
‣ Ask: Would I double-check this if a junior researcher handed it to me? If not, why not?
VII. Your Meta-Cognition
Did I feel the need to be skeptical at all?
‣ The most dangerous AI answers are the ones you don’t question.
Is my gut telling me something’s off?
‣ Honor that hunch. Treat it as a prompt to slow down and inspect further.
Have I tried flipping the lens?
‣ Ask: “What might this answer be leaving out, softening, or overstating?”