AI Judge Evaluation Methodology

Overview

The pentagonal radar chart scores reflect how well each theory's answers address key questions about consciousness. These are not judgments about which theory is "correct"—they measure explanatory clarity and completeness within the theory's own framework.

The 5 Questions

All 15 theories answer the same 5 standardized questions, designed to be theory-neutral and centered on the hard problem. Each question elicits concrete, testable claims rather than meta-theoretical positioning.

1. The hard problem: why does physical processing give rise to subjective experience?
2. Mechanism: What specific mechanism distinguishes conscious processing from unconscious processing of equal complexity?
3. Empirical dissociations: How does this theory account for known dissociations between consciousness and neural processing—such as anesthesia abolishing consciousness while preserving neural activity, split-brain cases, and blindsight?
4. Boundary conditions: What is the smallest or simplest system this theory predicts could be conscious, and how would that be tested?
5. Falsifiability: What specific, falsifiable predictions does this theory make that could distinguish it from competing theories, and what evidence would falsify it?

The 5 Radar Dimensions

Each theory is scored on 5 dimensions (scale 1–5), visualized as a pentagonal radar chart. Each dimension maps directly to one question, and every score includes a rationale explaining the reasoning.

Scores reflect how well the theory has been explained on the platform, not its inherent quality. A high score means clear, specific answers—not that the theory is correct.

The 15 Evaluated Theories

Scoring Process

1. PDF Extraction: Academic papers are processed using OpenAI gpt-4o-2024-11-20 to extract key claims and mechanisms.
2. Answer Generation: Claude Opus 4.6 generates answers to each question based solely on extracted source material.
3. Metrics Scoring: Claude Opus 4.6 scores each dimension with a 1–5 score and a rationale explaining the reasoning.
4. Calibration Check: Anchor points are verified to prevent score drift.
5. Community Review: All AI-generated content can be challenged, refined, and overridden by the research community.

Calibration & Fairness

Fairness in Content Generation

• Identical prompts for each theory (only the theory name changes)
• Same questions asked of every theory (5 standardized questions)
• Same rubric applied to all evaluations
• No theory-specific modifications to prompts or evaluation criteria
• Source-grounded: All content generated from academic PDF sources

Theory Selection

For details on how we decide which theories to include and exclude, see our Theory Selection Criteria page.

Important Caveats

• Answer Quality, Not Theory Truth: A high score means the theory provides clear, specific answers—not that the theory is correct.
• Source-Grounded: All scores derive from academic source material, not general knowledge.
• Trade-offs Are Real: Some theories intentionally sacrifice one dimension for another.
• Improvable: As source materials are added or answers refined, scores can improve.

Version History