Active Projects Extractor

Overview

The extract.py script extracts active/ongoing research projects from institute project listing pages using a two-pass approach with automatic pagination detection and LLM-based extraction.

Workflow

Configuration (.env_extraction)

# Input/Output Files
INPUT_FILE=input/institutes.json
OUTPUT_FILE=output/active_projects.json

# Extraction Limits
MAX_PROJECTS=50

# Maximum pagination pages per institute. 1 = initial page only; values >1 enable pagination and intelligent probing when max_pages exceeds the highest detected page.
MAX_PAGES=8

# LLM Configuration for Crawl4AI
EXTRACTION_PROVIDER=anthropic/claude-sonnet-4-20250514
EXTRACTIONLLM_API_KEY=your_api_key_here

# Skip already processed institutes (set to "true" to enable)
SKIP_EXISTING=false

# Rate limiting settings
LLM_MAX_RETRIES=3              # Max retry attempts for failed LLM calls
LLM_REQUESTS_PER_SECOND=1.0    # Rate limit (1.0 = 1 request per second)

LLM Rate Limiting & Retry Logic

The extractor includes built-in resilience against LLM rate limiting and transient failures. All LLM calls are automatically wrapped with retry logic and rate limiting to ensure reliable extraction.

Features

• Automatic Retries: Failed LLM calls are automatically retried with exponential backoff
• Rate Limit Detection: Detects common rate limit error patterns (429, "rate limit", "quota exceeded", etc.)
• Configurable Rate Limiting: Control the maximum requests per second to stay within API quotas
• Exponential Backoff: Retry delay doubles after each attempt (1s → 2s → 4s → ... → max 60s)

Retry Behavior

Rate Limit Detection

The extractor automatically detects rate limit errors by checking for these patterns in error messages:

• HTTP status code: 429
• Error messages: "rate limit", "too many requests", "quota exceeded", "rate_limit_exceeded"
• Service messages: "try again later", "service unavailable", "temporarily unavailable"

Configuration Recommendations

Example Configurations

# Conservative settings for free tier APIs
LLM_MAX_RETRIES=5              # More retries for stricter limits
LLM_REQUESTS_PER_SECOND=0.5    # 1 request every 2 seconds

# Aggressive settings for paid tier APIs
LLM_MAX_RETRIES=2              # Fewer retries (fail fast)
LLM_REQUESTS_PER_SECOND=5.0    # 5 requests per second

# No rate limiting for local/unlimited LLMs
LLM_MAX_RETRIES=2              # Keep some retries for transient failures
LLM_REQUESTS_PER_SECOND=0      # 0 disables rate limiting

Logging

When retries occur, the extractor logs detailed information:

WARNING - Rate limit detected (attempt 1/4). Retrying in 1.0s... Error: 429 Client Error: Rate Limit Exceeded
WARNING - LLM call failed (attempt 2/4). Retrying in 2.0s... Error: ConnectionError: Connection timed out
ERROR - LLM call failed after 4 attempts. Error: APIError: Server error (500)

Important Notes

• Applies to all LLM calls: Both listing page extraction and project detail extraction use rate limiting
• Per-crawler rate limiting: Rate limiting is tracked per extractor instance, not globally
• No persistence: Retry state is not saved across runs - if extraction is interrupted, start fresh
• Works with all providers: Rate limiting is provider-agnostic and works with any LLM configured via EXTRACTION_PROVIDER

Input Format: institutes.json

The input file contains a list of institutes to process. Two formats are supported:

Array Format

[
  {
    "text": "UNU-CPR",
    "url": "https://unu.edu/cpr/projects",
    "id": 1
  },
  {
    "text": "UNU-BIOLAC",
    "url": "https://unu.edu/biolac/projects",
    "id": 2
  }
]

Object with References

{
  "references": [
    {
      "text": "UNU-CPR",
      "url": "https://unu.edu/cpr/projects",
      "id": 1
    }
  ]
}

Output Format: active_projects.json

An array of active project objects with the following schema:

[
  {
    "institute_name": "string",
    "institute_id": "integer",
    "project_url": "string",
    "title": "string",
    "status": "string",
    "description": "string",
    "objectives": "string | null",
    "duration": "string | null",
    "funding": "string | null",
    "partners": ["string"] | null,
    "contact": "string | null",
    "additional_info": { ... } | null
  }
]

Field Descriptions

Common additional_info Keys

Based on analysis across 87 projects, 91% include additional_info. Here are the information types extracted:

Frequency indicates how many projects contain at least one key from that category.

Debug Output: exclusions_debug.jsonl

All excluded projects are logged to output/exclusions_debug.jsonl (JSONL format, one entry per line):

{
  "timestamp": "2025-12-25T10:30:00.123456",
  "institute_id": 1,
  "institute_name": "UNU-CPR",
  "institute_url": "https://unu.edu/cpr/projects",
  "project_title": "Example Project",
  "project_url": "https://example.com/project",
  "project_brief_description": "Brief description...",
  "project_status_indicator": "Completed",
  "exclusion_trigger": "Completed",
  "exclusion_reason": "completed_keyword"
}

Exclusion Reasons

Exclusion Keywords

The following keywords trigger automatic exclusion (found in title or brief description):

• Job postings: vacancy, hiring, consultant, officer, internship, call for application, apply now, submit cv, job opening, career opportunity
• Page elements: cookie, privacy policy, terms of use, subscribe, newsletter, follow us, rights reserved, skip to content, menu, contact us, about us

Completed Status Keywords

Projects matching these in status indicators are excluded: completed, finished, closed, past projects, archived

Date-Based Expiration

IMPORTANT: Explicit status takes precedence over date checks. If status explicitly says "Active" or "Ongoing", the project is kept regardless of end date. If status says "Completed", the project is excluded. Dates are only checked when status is unknown/ambiguous.

Projects with ambiguous/unknown status are automatically excluded if an end date is detected and the date has passed. The extractor scans:

• Listing page data: status_indicator and brief_description fields
• Full details: duration field from the project page

Date patterns detected:

• "Expected End Date: 31 May 2023"
• "End Date: 31 May 2023"
• "31 May 2023", "May 31, 2023"
• "2023-05-31" (ISO format)
• "31/05/2023", "31-05-2023"

CLI Arguments & Skip-Existing Processing

The extractor supports incremental processing to resume interrupted runs or add new institutes without reprocessing existing ones:

CLI Arguments Reference

Detailed Examples (assuming detected ?page=0 through ?page=4)

Note: "Detected links" count excludes ?page=0 since it duplicates the base page content. The max_detected_page value is the highest page number found (e.g., 4 for ?page=0 through ?page=4).

Precedence Order

When multiple sources specify processing behavior, the following precedence applies (highest to lowest):

1. --force CLI flag - Always process all institutes (overrides everything)
2. --reprocess CLI flag - Reprocess only specified institutes (overrides --skip-existing)
3. --skip-existing CLI flag - Skip institutes with existing projects (overrides env)
4. Environment variable - Used only if no CLI flags provided

# Run with defaults (process all)
python extract.py

# Skip already processed institutes
python extract.py --skip-existing

# Force reprocess all (overrides --skip-existing and --reprocess)
python extract.py --force

# Reprocess specific institutes by ID
python extract.py --reprocess 1,2,3

# Reprocess specific institutes by short name (e.g., UNU-WIDER)
python extract.py --reprocess UNU-WIDER,MERIT

# Reprocess by partial name match (case-insensitive)
python extract.py --reprocess "Biotechnology","Comparative Regional"

# Custom output file
python extract.py -o custom/projects.json

# Override max projects per institute
python extract.py --max-projects 30

# Combining flags (--force takes precedence)
python extract.py --skip-existing --force  # will force reprocess all