User Guide

4b. Literature-Grounded Reviews (New Feature!)

The system now supports literature-grounded reviews that automatically search for and analyze related papers to ground the assessment in existing research. This feature helps verify novelty claims, identify missing citations, and position the paper within the research landscape.

Pipeline Architecture: Dual-Mode Operation

The literature-enhanced pipeline (run_review_with_dir_literature.py) supports two modes controlled by the --literature-grounding flag:

Key characteristics:

• Uses criteria.yaml for evaluation criteria (same as standard pipeline)
• Literature stages are optional — can be enabled/disabled per run
• Falls back to standard review if literature stages fail
• Literature config in config/literature_sources.yaml

What Is Literature Grounding?

When enabled, literature grounding adds a 4-stage process that goes beyond standard review:

Key Benefits

• Baseline Literature Context: Establishes what's known in the research area by finding relevant papers
• Novelty Rankings: Per-criterion novelty assessment (1-5 scale) comparing the paper against baseline literature
• Research Trajectory: Shows where the paper fits in the evolution of the field
• Fact-Checking: Verifies suspicious claims like "first study" when detected
• Novelty-Adjusted Scoring: Adjusts scores based on actual contribution vs. prior work

How to Enable Literature Grounding

Literature grounding is completely optional and can be enabled in three ways:

python setup_run_literature.py \\
  --run-dir my_literature_review

# Set to false to disable
enabled: true

# Enable literature grounding
python run_review_with_dir_literature.py \\
  --run-dir my_review_run \\
  --literature-grounding

# Standard mode (no literature)
python run_review_with_dir_literature.py \\
  --run-dir my_review_run

Literature Sources Configuration

The system supports multiple literature sources that can be enabled/disabled in config/literature_sources.yaml:

Enabling/Disabling Sources

Edit config/literature_sources.yaml to control which sources are active:

sources:
  semantic_scholar:
    enabled: true   # Primary source, has citations
  arxiv:
    enabled: true    # Good fallback for Semantic Scholar rate limits
  world_bank:
    enabled: false  # Set to true to enable

Semantic Scholar API Setup

Semantic Scholar offers a free-tier API (100 requests/minute). For higher limits, get an API key:

1. Visit https://www.semanticscholar.org/product/api#api-key
2. Sign up for a free API key
3. Add to your global .env file:

   SEMANTIC_SCHOLAR_API_KEY=your_key_here

Arxiv API

Arxiv is a free, open-access archive that serves as an excellent fallback when Semantic Scholar rate limits are reached. When enabled, it provides:

• Preprints and published papers across CS, physics, math, and more
• No API key required
• No strict rate limits (1 request per 3 seconds recommended)
• Open-access PDF availability

Note: Arxiv papers lack citation counts and will rank lower than Semantic Scholar papers in results. Use as a fallback when Semantic Scholar is rate-limited.

World Bank API

The World Bank API is free and requires no authentication. When enabled, it provides:

• Development economics working papers and reports
• Policy documents and publications
• Open-access PDF availability

Note: World Bank papers lack citation counts and will rank lower than Semantic Scholar papers in results.

Literature-Grounded Output

When enabled, reviews include additional sections:

• Research Trajectory: Position of the paper in the literature landscape
• Novelty Assessment: Per-criterion novelty rankings (1-5 scale)
• Fact-Check Summary: Verification of bold or suspicious claims
• Novelty-Adjusted Score: Score adjusted based on actual novelty

Configuration Options

The config/literature_sources.yaml file controls all aspects:

Batch Processing with Literature Grounding

For batch runs with literature grounding:

python setup_batch_runs_literature.py \\
  --master-papers-dir papers_master \\
  --base-run-dir lit_run \\
  --num-runs 5 \\
  --create-batch-script

4c. Standalone Literature Review Pipeline

The standalone literature review pipeline is a dedicated workflow for papers that require comprehensive literature analysis. Unlike the literature-grounded enhancement (Section 4b), this pipeline uses only the literature-grounded agents and does not include a fallback to standard review.

Pipeline Architecture

The standalone pipeline (run_review_literature.py) implements a pure 4-stage literature-grounded workflow:

Key characteristics:

• Configured via literature_sources.yaml (not criteria.yaml)
• Always runs all 4 stages (no fallback to standard review)
• Uses dedicated literature/ directory for baseline papers
• Stage-specific LLM parameters (Librarian, Reader, Fact-Checker, Critic)

Key Differences from Literature-Grounded Enhancement

When to Use the Standalone Pipeline

The standalone pipeline is designed for:

• Systematic Literature Reviews: When you need to position papers within the research landscape
• Novelty Assessment: When evaluating claims of originality or "first study" assertions
• Citation Analysis: When checking for missing prior work or proper attribution
• Research Trajectory: When understanding how a paper advances the field

Getting Started with Standalone Literature Review

Step 1: Set Up the Run Directory

Use the dedicated setup script:

python setup_literature_review.py \
  --run-dir my_literature_review

This creates:

• my_literature_review/papers/ — Place papers to review here
• my_literature_review/literature/ — Place baseline reference papers here (optional)
• my_literature_review/input/literature_sources.yaml — Configure literature sources
• my_literature_review/input/.env — LLM parameters for each stage

Step 2: Configure Literature Sources

Edit literature_sources.yaml to define your baseline literature:

# Literature sources (baseline papers)
sources:
  - id: "baseline_001"
    title: "Foundational Paper in Your Field"
    authors: ["Author One", "Author Two"]
    year: 2024
    venue: "Top Conference"
    topics: ["topic1", "topic2"]
    key_contributions:
      - "Established the methodology"
      - "Introduced key framework"

# Research trajectory definition
research_trajectory:
  starting_point: "earliest_work"
  progression:
    - stage: "initial_concepts"
      description: "Early foundational work"
    - stage: "current_state"
      description: "State-of-the-art approaches"
    - stage: "open_challenges"
      description: "Current limitations"

Step 3: Configure LLM Parameters

The standalone pipeline uses stage-specific LLM settings in .env:

# Stage 1: Librarian (Baseline Reference Creation)
PROVIDER_LIBRARIAN=openai
LIBRARIAN_MODEL=gpt-4o
LIBRARIAN_TEMPERATURE=0.2

# Stage 2: Reader (Novelty Ranking & Extraction)
PROVIDER_READER=openai
READER_MODEL=gpt-4o-mini
READER_TEMPERATURE=0.3

# Stage 3: Fact-Checker (Claim Verification)
PROVIDER_FACT_CHECKER=openai
FACT_CHECKER_MODEL=gpt-4o
FACT_CHECKER_TEMPERATURE=0.1

# Stage 4: Critic (Grounded Synthesis)
PROVIDER_CRITIC=deepseek
CRITIC_MODEL=deepseek-reasoner
CRITIC_TEMPERATURE=0.2

Step 4: Run the Literature Review

Execute the standalone pipeline:

python run_review_literature.py --run-dir my_literature_review

Visual Progress Indicators

The pipeline provides clear visual feedback for each stage:

LITERATURE-GROUNDED REVIEW [1/5]: paper.pdf
============================================================

[Stage 1/4] Librarian: Creating baseline reference...
[Librarian] ✓ Created baseline with 5 papers

[Stage 2/4] Reader: Extracting evidence with novelty ranking...
[Reader] ✓ Completed 8 criterion extractions

[Stage 3/4] Fact-Checker: Running verification checks...
[Fact-Checker] ✓ Completed 3 verification checks
[Fact-Checker] ⚠️  2 claims require further review

[Stage 4/4] Critic: Synthesizing grounded review...
[Critic] ✓ Review synthesized (score: 65.2)

[Summary]
  Overall Score: 65.2
  Recommendation: REVISE AND RESUBMIT
  Avg Novelty: 3.45/5
  Novelty Adjustment: +2.5

Batch Processing with Standalone Pipeline

For processing multiple directories with the standalone pipeline:

python setup_batch_literature_review.py \
  --master-papers-dir papers_to_review \
  --master-literature-dir baseline_literature \
  --base-run-dir literature_run \
  --num-runs 3 \
  --create-batch-script

This creates batch directories with:

• Papers distributed across literature_run1/, literature_run2/, etc.
• Baseline literature papers copied to each run's literature/ directory
• run_batch_literature_review.py script for batch execution

Run the batch:

# Sequential execution
python run_batch_literature_review.py

# Parallel execution
python run_batch_literature_review.py --parallel --max-workers 4

Output Differences

Standalone literature reviews always include:

• Research Trajectory Section: Detailed analysis of how the paper fits into the literature
• Novelty Rankings: Per-criterion novelty scores (1-5 scale)
• Fact-Check Results: Verification of suspicious claims
• Novelty-Adjusted Score: Base score adjusted for actual contribution

Comparison: Which Should You Use?

5a. The Most Important Part: Customizing Your Criteria

This is the system's most powerful feature. You do **not** need to touch any Python code to completely change the review. The *only* file you need to edit is my_review_run/input/criteria.yaml.

The One Big Rule: Weights Must = 100

The system uses a 100-point scale. The weight: value you give to each criterion tells the system how much it "matters." Before you run the script, you **must** ensure all weight values in your file add up to exactly 100.

Anatomy of a Criterion

Let's break down one criterion block from the file:

  - id: empirical_rigor
    name: Empirical Rigor
    description: |
      Assesses the quality of the empirical methods, data,
      and execution. Look for research design, causal
      identification, and statistical analysis.
    weight: 20
    scale:
      type: numeric
      range: [1, 5]
      labels:
        1: "Fundamentally flawed"
        2: "Significant weaknesses"
        3: "Adequate"
        4: "Strong and robust"
        5: "Exceptional / state-of-the-art"

• id: The unique, one-word ID. This is used as the header in the final CSV file (e.g., empirical_rigor_score).
• name: The "pretty" name used in the Markdown report.
• description: This is the most important field. This text is injected directly into the prompt for Agent 1. A clear, specific description will give you a high-quality review. A vague description will give you a vague review.
• weight: How much this score matters out of 100.
• scale: Defines the scoring. The labels are used to tell the AI what the 1-5 scale means.

Example: How to Add a New Criterion

Let's add a new criterion for "Statistical Robustness" with a weight of 10.

Step 1: Add the new block

Copy-paste an existing block and edit it. Now your file looks like this:

criteria:
  - id: theoretical_contribution
    name: Theoretical Contribution
    weight: 15
    # ... (other fields) ...
    
  - id: empirical_rigor
    name: Empirical Rigor
    weight: 20
    # ... (other fields) ...
    
  # ... (all your other criteria) ...
    
  # --- OUR NEW CRITERION ---
  - id: statistical_robustness
    name: Statistical Robustness
    description: |
      Evaluates the quality of statistical tests, power
      analysis, and sensitivity/robustness checks.
    weight: 10
    scale:
      type: numeric
      range: [1, 5]
      labels:
        1: "Statistically flawed"
        2: "Weak / Inappropriate"
        3: "Adequate"
        4: "Robust"
        5: "Exceptional"

Step 2: Adjust the weights to sum to 100

Our total weight is now 110 (the original 100 + our new 10). We must remove 10 points from other criteria.

Let's reduce theoretical_contribution from 15 to 10, and empirical_rigor from 20 to 15.
(15 + 20) = 35. (10 + 15) = 25. We've removed 10 points. Our total is 100 again.

Your file is now valid and ready to run. The system will automatically add a new "Statistical Robustness" section to all reviews and a new statistical_robustness_score column to your CSV.

How to Remove a Criterion

1. Delete the entire block (from - id: ... to the last label: ...).
2. Adjust the remaining weights to add up to 100.

Recommendation Rules Based on Final Score

The system determines the recommendation using the following default thresholds, which can be changed in my_review_run/input/criteria.yaml:

• If the final_score is 85 or higher → Accept
• If the final_score is between 70 and 84 → Accept with Revisions
• If the final_score is between 50 and 69 → Revise and Resubmit
• If the final_score is below 50 → Reject

Specialization Domain

The domain section in the criteria yaml file is what "specializes" the agents. You can reference it in the my_review_run/input/prompts/extractor_system.txt, like: "You are an expert reviewer in the field of {domain}." "Analyze the following paper based on criteria for {domain}." By changing this one line in criteria.yaml, you can pivot your entire review system to a new field (e.g., "machine_learning" or "clinical_psychology") and the extractor agents will adjust their analysis accordingly.

5b. Controlling Agent Behavior with Prompts

The "brain" of your AI agents is controlled by a set of simple text files located in your run directory. By editing these "prompts," you can change the agents' personalities, their analytical focus, and the style of their writing.

The "System" vs. "User" Prompt: A Key Concept

You'll notice that both the Extractor and Synthesizer agents have two prompt files. This is the most important concept to understand:

File-by-File Breakdown

Practical Examples: What Can I Change?

• To make the synthesizer's tone more critical: Open synthesizer_system.txt and add a line to its role: "Be especially critical of methodological flaws and overstatements of findings. Do not be overly complimentary."
• To add a new instruction to the extractor: Open extractor_system.txt and add a new rule: "You must always provide at least one direct quote from the paper to support your 'strengths' and 'weaknesses' rationale."
• To change the final review's summary: Open synthesizer_user.txt and modify the description for executive_summary. For example, you could change it from "3-4 sentence summary..." to "a 3-bullet-point summary of the most critical findings."

📋 The Judge Role: Deep Dive

The Judge component is a sophisticated arbitration system that resolves disagreements between multiple AI models when they produce conflicting reviews of the same paper. Understanding how it works will help you trust its verdicts and use it effectively.

Core Responsibilities

The Judge serves three primary functions in the review system:

• Conflict Detection: Identifies papers where different AI models disagree on recommendations (e.g., one says "Accept", another says "Reject")
• Champion Selection: Intelligently selects the best representative review from each conflicting faction
• Final Adjudication: Makes an authoritative decision based on careful analysis of the original paper and conflicting reviews

The "Champion vs Champion" Strategy

Instead of adjudicating between all possible pairs of reviews, the Judge uses an efficient "champion selection" approach:

{
  "judge_recommendation": "Accept w/ Revisions",
  "judge_rationale": "Review A more accurately captures...",
  "winning_review": "A",
  "winning_review_rationale": "Review A demonstrates deeper analysis..."
}

Scoring System & Weighted Rules

The Judge does not assign scores—that's done by the Extractor and Editor agents. Instead, the Judge uses the existing weighted scoring system to evaluate champion quality:

Configuring the Judge

You can customize which AI model acts as Judge by setting environment variables in your run directory's .env file:

# Default configuration (can be omitted)
JUDGE_PROVIDER=openai
JUDGE_MODEL=gpt-4o

# Alternative: Use Gemini as Judge
JUDGE_PROVIDER=gemini
JUDGE_MODEL=gemini-2.5-pro

# Alternative: Use Claude as Judge
JUDGE_PROVIDER=anthropic
JUDGE_MODEL=claude-sonnet-4-5

Progress Tracking & Cost Management

The Judge system includes sophisticated progress tracking to avoid redundant adjudication:

• Configuration Hashing: Detects changes in Judge configuration and re-adjudicates if settings change
• Progress Files: Stores adjudication results in judge_progress.json
• Resume Capability: Skips already-adjudicated papers when configuration is unchanged
• Cost Tracking: Monitors API usage for budget management

Understanding Judge Verdicts

The final JUDGE_VERDICTS_...csv file contains these key columns:

• paper_filename — The paper being adjudicated
• model_a_recommendation — First model's recommendation
• model_b_recommendation — Second model's recommendation
• judge_recommendation — The Judge's final decision
• winning_review — Which review the Judge preferred ("A", "B", or "Neither")
• judge_rationale — Detailed explanation of the decision

The Judge's rationale is your most valuable asset—it provides a neutral, expert assessment of which review was more accurate and why. This can help you:

• Understand why models disagreed
• Identify systematic biases in specific models
• Make final acceptance/rejection decisions with confidence
• Improve your review criteria over time

Key Strengths of the Judge System

1. Efficiency: Champion selection reduces adjudication complexity from O(n²) to O(n)
2. Transparency: Detailed rationales explain every decision
3. Robustness: Progress tracking prevents redundant work and enables resumption
4. Configurability: Weights, thresholds, and Judge model are all customizable
5. Cost-Effectiveness: Tracks API usage and avoids unnecessary adjudications

The Judge component transforms the multi-agent review system from a simple consensus tool into a sophisticated peer review mechanism—ensuring that final decisions are based on the best available analysis, not just the first available agreement.