Blog Article

Overview

Vanguard-MAS transforms user-defined risk criteria into a structured investigation using specialized AI agents that work in parallel and sequence to produce a vetted, multi-perspective vendor recommendation.

What's New — August 2026

• Entity gate and URL filters: a dedicated identity check rejects findings about same-named products or companies (e.g. a WordPress theme sharing the vendor's name), and deterministic filters drop content merely hosted on the vendor's platform and lookalike domains. Applied to findings, adversarial counter-arguments, references, and the Reference Audit in every mode.
• Smarter URL validation: sources blocked by bot protection (HTTP 403) are no longer discarded — the claim is verified against the page's latest Internet Archive snapshot, and the audit row says exactly what was established.
• Verdict at a Glance: VRM reports open with the recommendation, its full derivation trail (VC score with borderline warnings, PVE level and provenance, any downgrade and its trigger), and what the verdict means in practice.
• Vendor Follow-Up Questions: every criterion that couldn't be verified from public evidence compiles into a ready-to-send due-diligence questionnaire (report section 8).
• PVE hardening: the data question now asks what the vendor can access (not what you intend to send), a capability-containment question prices in unrestricted unused features, and each classification carries a validity statement bound to the assessed use.
• AI supply-chain flag: AI/LLM providers found among the vendor's subprocessors surface as an explicit monitoring condition in the conclusion.
• Search controls in Settings: Perplexity search depth and recency filter, and Claude "max searches per query" (the agentic search loop) are configurable per browser, with explanations inline. DeepSeek deployments on Azure Foundry now perform real web searches.
• Cost transparency: per-search fees are itemized in the cost table, and the Summary provider line is mode-aware ("n/a" for template-generated VRM/PVE reports).

Key Features

Standalone PVE Assessment

The PVE (Product Vulnerability Exposure) mode provides a standalone business impact assessment without the full VRM framework. Available in both CLI and web interface.

What is PVE?

Product Vulnerability Exposure (PVE) measures the potential impact on your organization if an ICT product fails or is compromised—whether due to technical weaknesses in the product itself or the vendor's inability to manage security or operational risks.

PVE focuses on the consequences of failure or security incidents, not the likelihood. It assesses how severe the impact would be on operations, data, recovery capability, and resilience if an incident were to occur.

PVE Classification Levels

Business Case for PVE Standalone

PVE standalone mode provides a calibrated assessment approach that helps organizations focus their due diligence efforts proportionally to the actual business risk. Instead of applying a one-size-fits-all security questionnaire to every vendor, PVE helps you determine the appropriate depth of assessment based on business impact severity.

Choosing the Right Assessment Mode

Vanguard-MAS uses PVE classification to determine the appropriate depth of vendor assessment. After PVE classification, choose the assessment mode that matches your vendor relationship:

Data Handling Scenarios: Why Some VRM Questions May Not Apply

VRM assessment questions are designed for comprehensive SaaS evaluations, but not all questions apply depending on your data relationship with the vendor:

Practical Approach: PVE-Calibrated Assessment Scope

Use your PVE classification to determine the appropriate depth of vendor capability assessment:

What PVE Analyzes

PVE measures business impact through two dimensions:

CLI Usage

# Basic PVE assessment (interactive questions only)
vanguard pve --product "Slack" --domain "slack.com" --intended-use "Team communication"

# With background vendor research (more accurate data sensitivity)
vanguard pve --product "Slack" --domain "slack.com" --intended-use "Team communication" --research

# With custom output path
vanguard pve --product "Zoom" --domain "zoom.us" --intended-use "Video conferencing" --output reports/zoom-pve.md

PVE Output

• Classification: Low / Medium / High
• Impact Score: 1.0 - 5.0
• Detailed explanation: Score breakdown with reasoning

When to Use PVE

PVE → Assess Workflow (Recommended for Most Vendors)

This two-step approach maximizes efficiency while maintaining appropriate due diligence:

1. Step 1: Run PVE Standalone — Classify business impact (Low/Medium/High)
2. Step 2: Run Targeted Assess Mode — Focus on specific vendor capabilities based on PVE level

When to Use Each Mode

Assess Mode

Assess Mode uses a custom risk criteria approach with a dedicated verification phase. Vanguard-MAS has two assessment modes with different agent pipelines:

Agent Roles

Assess Mode Pipeline

┌─────────────────────────────────────────────────────────────────┐
│                   Vanguard Orchestrator                         │
│  - Decomposes criteria into tasks                               │
│  - Manages parallel/sequential execution                        │
└─────────────────────────────────────────────────────────────────┘
                    │
        ┌───────────┴───────────┐
        │   Phase II: Research  │ (Parallel)
        │   InvestigatorAgent   │ → Web searches for each domain
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │  Phase III: Verify    │ (Sequential)
        │    AuditorAgent       │ → URL validation & fact-checking
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │   Phase IV: Analyze   │ (Sequential)
        │  RiskAnalystAgent     │ → Maps findings to requirements
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │    Phase V: Challenge │ (Sequential)
        │ DevilsAdvocateAgent   │ → Finds counter-arguments
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │  Phase VI: Synthesize │
        │   FinalReport         │ → Go/No-Go/Conditional
        └───────────────────────┘

Key Differences Between Modes

VRM Mode

The VRM (Vendor Risk Management) mode provides a structured framework assessment with PVE classification and risk matrix analysis.

VRM Mode Pipeline

┌─────────────────────────────────────────────────────────────────┐
│                     VRM Orchestrator                            │
│  - Manages VRM framework workflow                               │
│  - Maps findings to VRM schema                                  │
└─────────────────────────────────────────────────────────────────┘
                    │
        ┌───────────┴───────────┐
        │  Step 0: Research     │ (Pre-assessment)
        │  Vendor Background    │ → Research vendor capabilities
        │  + Present to User    │ → Show findings before questions
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │  Step 1: Business     │ (Interactive)
        │  Impact Assessment    │ → User answers org-specific questions
        │  - Research detected  │ → Data type, multi-tenant, API keys
        │  - User input         │ → Criticality, fallback, cascade
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │ Phase I: PVE Class    │
        │  Generate PVE Score   │ → Based on business impact
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │  Phase II: Capability │ (Parallel)
        │  InvestigatorAgent    │ → Research 7 maturity domain groups (33 subdomains)
        │  + URL Validation     │ → Mapping to VRM schema
        │  + Confidence Filter  │ → Authoritative source check
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │ Phase III: Risk Matrix│
        │   VC × PVE → Risk     │ → Initial inherent risk
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │ Phase IV: Controls    │
        │  InvestigatorAgent    │ → Identify mitigating controls
        │  + URL Validation     │
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │  Phase V: Challenge   │
        │ DevilsAdvocateAgent   │ → Adversarial analysis
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │ Phase VI: Residual    │
        │  Risk + Breach        │ → Final risk + breach analysis
        │  Mitigation Impact    │
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │ Phase VII: Conclusion │
        │  VRMConclusion        │ → Engage/Conditional/Do Not Engage
        └───────────────────────┘

Processing Pipeline

1. Step 0 - Vendor Background Research: Research vendor capabilities (data types, multi-tenant, API keys, regulatory)
2. Step 1 - Business Impact Assessment: Interactive assessment using hybrid approach:
   • Research-detected factors (automatic): Data type, multi-tenant architecture, regulatory compliance, API keys/OAuth tokens
   • User input factors (organization-specific): Service criticality, fallback availability, cascade effects
3. Phase I - PVE Classification: Generate PVE score based on business impact analysis
4. Phase II - Vendor Capability: Parallel investigation across maturity domains with URL validation
5. Phase III - Initial Risk Matrix: Calculate inherent risk from VC × PVE
6. Phase IV - Mitigating Controls: Identify security controls with URL validation
7. Phase V - Adversarial Analysis: Devil's Advocate challenges findings
8. Phase VI - Residual Risk: Final risk after controls and adversarial findings
9. Phase VII - Conclusion: Final recommendation with breach mitigation impact

Business Impact-Driven PVE Classification

The PVE (Product Vulnerability Exposure) classification uses a two-dimensional impact model:

Dimension 1: Data Sensitivity Impact (Confidentiality)

• What type of data does the vendor handle? (Public → PII → Financial → Credentials/PHI)
• Multi-tenant architecture? (Standard SaaS - minimal risk adjustment)
• What regulatory consequences apply? (GDPR, HIPAA, PCI DSS)
• CRITICAL: API Keys/OAuth Tokens - Does the vendor store credentials that grant access to other systems?
  • Access rights: Read-only vs Read/Write vs Admin/Delegated
  • Validity period: Short-lived (<24h) vs Medium (1-30 days) vs Long-lived (30+ days)
  • Example: Buffer stores OAuth tokens for social media posting (60-day validity = HIGH risk)

Dimension 2: Service Interruption Impact (Availability)

• How critical is this service to your operations? (Peripheral → Core)
• Do you have a fallback or workaround? (Easy switch → No alternative)
• Would failure cause cascade effects? (Affects other systems)

Impact Score = (Data Sensitivity × 50%) + (Service Interruption × 50%)
PVE Level: Low (<0.40) | Medium (0.40-0.74) | High (≥0.75)

Assessment Domains (v2)

The VRM framework now features 7 maturity domain groups with 33 subdomains, each optimized with context-aware search strategies based on signal type analysis.

Domain Categories

Vendor Service Type Classification

Selecting a vendor service type automatically recommends appropriate domains and subdomains:

Context-Aware Search Strategies (v2)

Each subdomain uses an optimal search strategy based on signal type analysis:

Flexible Domain Selection

The v2 framework provides fine-grained control over assessment scope:

• Domain-level: Enable/disable entire maturity domains
• Subdomain-level: Enable/disable individual subdomains within each domain
• Auto-population: Vendor service type automatically recommends domains/subdomains
• User override: Full control to override recommendations
• Default: All domains and subdomains enabled
• Instant validation: An enabled domain must keep at least one subdomain selected — violations are flagged inline and block submission
• Skip-neutral scoring: Deselected subdomains are excluded from both numerator and denominator and reported as “Skipped by configuration” — they never lower a score

Assessment Pipeline (Updated)

1. Vendor Background: Product description and core services
2. PVE Classification: Business impact-driven classification with detailed justification
3. Vendor Capability (v2): Assessment across 7 maturity domain groups with 33 subdomains using context-aware search strategies — AI & Emerging Technology scores as its own section
4. Initial Risk Matrix: VC × PVE → Inherent Risk
5. Mitigating Controls: Security controls that reduce risk
6. Adversarial Analysis: Devil's Advocate challenges findings and identifies hidden risks
7. Residual Risk: Final risk after accounting for controls AND adversarial findings
8. Breach Mitigation Analysis: When breaches are found, analyzes control adequacy and affects recommendation
9. Conclusion: Recommendation with security requirements and complete reference list

How the Vendor Capability (VC) Score Is Computed

All scores are on a 0–5 scale. The same explanation is shown in the web UI's “VRM Framework Overview” card so results are interpretable at a glance.

1. Subdomain score: each selected subdomain is researched via web searches, its evidence validated, and checked against 2–3 scoring criteria. Score = verified criteria ÷ its criteria × 5.
2. Polarity-aware criteria: a criterion verifies only on favorable evidence. Every criterion carries veto keywords, so a vendor admitting a weakness (“systems remain unpatched”, “customers cannot opt out of model training”) counts against the criterion rather than for it. Customer-data model training without an opt-out is both a High-severity counter-risk and a direct scoring penalty.
3. Scoring section score: a criteria-weighted average of its assessed subdomains — a subdomain with 3 criteria weighs more than one with 2. Cross-routed subdomains (“→ scores under …”) contribute to their target section.
4. Skipped ≠ failed: deselected subdomains are excluded from both numerator and denominator — they never lower a score, and 5.0/5.0 stays reachable from a partial selection. The report labels them “Skipped by configuration”, and a fully-deselected group renders as “Skipped by configuration” rather than 0.0/5.0.
5. Overall VC: criteria-weighted mean across the sections that assessed anything, always annotated with its basis (“based on N of M subdomains assessed”) — a thin-coverage score is visibly different from a comprehensive one.
6. VC level: <2.5 → Low, 2.5–3.9 → Medium, ≥4.0 → High. This level enters the VC × PVE risk matrix, so score quality directly drives the recommendation.

Breach Mitigation Impact

When breaches are found, mitigation adequacy affects the final recommendation:

Parallel Search Strategy

VRM Phase II uses a hybrid search strategy that combines site-specific and general web searches for comprehensive vendor intelligence:

Query Generation Per Maturity Domain (v2)

Each of the 7 maturity domain groups is researched in parallel using context-aware search strategies based on signal type:

Standard Mode Query Generation

For each requirement, the system generates 4 queries (when primary_domain is provided):

# General Searches (third-party sources)
"{vendor_name} ({primary_domain}) {requirement}"
"{vendor_name} ({primary_domain})" "{requirement}" latest

# Site-Scoped Searches (official vendor documentation)
site:{primary_domain} {requirement}
site:{primary_domain} "{requirement}"

Example: Technical Maturity - Data Encryption

For a requirement like "Data encryption in transit and at rest" for Slack:

Parallel Execution Across Domains

All selected maturity domain groups are researched simultaneously using async concurrency with context-aware search strategies:

┌─────────────────────────────────────────────────────────────┐
│  Phase II: Vendor Capability (Parallel Execution)           │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
│  │ Operational  │  │  Technical   │  │ Engineering  │  ...  │
│  │  Maturity    │  │  Maturity    │  │  Maturity    │       │
│  │              │  │              │  │              │       │
│  │ • Data       │  │ • API        │  │ • Bug bounty │       │
│  │   Retention  │  │   Security   │  │ • Pen test   │       │
│  │ • Data       │  │ • Encryption │  │ • CVE        │       │
│  │   Residency  │  │ • Access     │  │   assessment │       │
│  │   ...        │  │   Control    │  │ • Ethical AI │       │
│  │              │  │   ...        │  │   ...        │       │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘       │
│         │                 │                 │               │
│         └─────────────────┴─────────────────┘               │
│                           │                                 │
│                    ▼ Vendor Capability Score                │
│              (Operational + Technical +                     │
│               Engineering + Risk Management)                │
└─────────────────────────────────────────────────────────────┘

Efficient Mode (Background Research)

For vendor background research (Step 0), the system uses efficient mode with only 2 queries per requirement:

# Efficient Mode: Reduced query count for faster background research
"{vendor_name} ({primary_domain}) {requirement}"
site:{primary_domain} "{requirement}"

Quality Confidence Adjustment

After search results return, findings are scored and adjusted based on source quality:

Why Both Search Types Are Essential

Using both site-scoped and general searches for the same domain/requirement is critical because each type reveals different aspects of a vendor's security posture:

Run Controls & Performance

Every run — CLI or web — can be tuned for speed and scope without editing config files. All controls are recorded in the report's Settings Audit Trail, and anything skipped is reported as skipped, never silently.

Assessment Depth

Per-Run Phase Toggles

Each phase can be switched off for a single run (defaults persist per browser via localStorage):

• Adversarial analysis (Devil's Advocate) — includes the two sub-toggles below
• Weak signals (news, litigation, leadership, CVE searches)
• Residual risks (post-decision risk profiling)
• Breach mitigation (control-adequacy analysis when breaches are found)

Progressive vs Comprehensive Search (A/B)

One vendor page (e.g. a trust center) often answers several subdomains at once. Progressive search exploits this: it builds an evidence pool first, marks every subdomain whose scoring criteria are all already covered by pooled evidence, and launches investigators only for the rest.

• Comprehensive (default): every selected subdomain gets its own full investigation — maximum coverage, most search calls.
• Progressive: evidence is reused before expanding. VRM seeds the pool from vendor background research (fast-track runs execute a small seed round); Quick/Assess run a first wave, then only search requirements the first wave did not satisfy.
• Conservative by design: a subdomain is only short-circuited when all of its criteria are covered, so skipping never trades away evidence a criterion needed.
• Built for comparison: every run counts its actual search-provider calls, and both the search type (progressive / comprehensive) and call count appear in the report's audit trail and the History page — re-run or clone the same assessment with the other mode to A/B cost vs. coverage.

Criterion-Gap Retry (automatic)

Search results vary run to run — a subdomain's searches can return pages that answer the question without containing the exact evidence a scoring criterion needs. After the first scoring pass, every run automatically performs one bounded round of targeted follow-up searches:

• VRM: each still-unverified criterion on an assessed subdomain gets one targeted query; new validated findings join the evidence pool and the capability is re-scored once.
• Assess/Quick: each requirement with no verified findings after the audit gets one targeted follow-up search; results pass the same Auditor verification bar before merging.
• Bounded by design: the retry runs at Fast depth (one query per gap) and is capped per run — 4 gaps at Fast, 12 at Standard, 20 at Deep. One round only, no convergence loop. Skipped subdomains never trigger retries.
• Fully accounted: retry searches are included in the search-call counters, and the audit trail reports exactly what the round did.

Evidence Review (LLM judge)

After scoring and gap retry, every VRM run makes one batched LLM call that reviews each verified criterion together with the exact quoted sentence and source that verified it: does this evidence genuinely support this criterion for this vendor? Criteria whose evidence is about a different subject (an external standard, another company, a document the vendor merely hosts), states an absence, or is unrelated get demoted — the criterion flips to unverified with the judge's reason in the report, and the capability is re-scored once. The judge fails open: if the LLM call errors, the lexically-scored result stands. Subdomains where nothing could be verified state it plainly: “Insufficient public evidence — an absence of published documentation, not verified weakness.”

The audit trail records all of this per run:

**Run options:** depth=standard | progressive search: on (9/21 subdomains from seed) | search calls: 87 | gap retry: 6 targeted searches for 6 unverified criteria, 4 newly verified | evidence review: 14 criteria reviewed, 3 demoted | skipped phases: weak signals

How Search Results Are Processed and Used

The hybrid search results from both site-scoped and general searches flow through a multi-stage pipeline that transforms raw search data into actionable vendor risk intelligence:

┌─────────────────────────────────────────────────────────────────┐
│              Search Results Processing Pipeline                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  1. PARALLEL SEARCH EXECUTION                                   │
│     ┌──────────────────┐    ┌────────────────┐                  │
│     │ General Search   │    │ Site-Scoped    │                  │
│     │ (4 queries)      │    │ Search (4)     │                  │
│     │ • Third-party    │    │ • Vendor docs  │                  │
│     │ • Breach reports │    │ • Trust center │                  │
│     └────────┬─────────┘    └────────┬───────┘                  │
│              └───────────────────────┘                          │
│                            ▼                                    │
│  2. FINDING EXTRACTION (LLM)                                    │
│     • Search snippet → Structured claim                         │
│     • Extract quotable text for verification                    │
│     • Initial confidence score (0.0-1.0)                        │
│                            ▼                                    │
│  3. QUALITY CONFIDENCE ADJUSTMENT                               │
│     • Vendor domain: +0.2 (definitive source)                   │
│     • Security news: +0.1 (credible incidents)                  │
│     • No vendor mention: -0.3 (irrelevant)                      │
│     • Similarly-named: -0.5 (wrong company)                     │
│     • Consulting firms: -0.3 (compliance searches only)         │
│                            ▼                                    │
│  4. URL VALIDATION                                              │
│     • HTTP 200 → Include                                        │
│     • 404/410/Timeout → Reject                                  │
│     • Follow redirects (max 5 hops)                             │
│                            ▼                                    │
│  5. CROSS-REFERENCE & MAPPING                                   │
│     ├── VRM Mode: Findings → Maturity Domain Scores (1-5)       │
│     ├── Assess Mode: Findings → Requirements                    │
│     └── Both: Vendor claims vs third-party verification         │
│                            ▼                                    │
│  6. ADVERSARIAL CHALLENGE                                       │
│     • Devil's Advocate generates counter-queries                │
│     • SOC2 claims → Search for breaches/compliance exceptions   │
│     • Encryption claims → Search for vulnerabilities            │
│     └── High-severity counter-arguments → Risk adjustment       │
│                            ▼                                    │
│  7. FINAL OUTPUT                                                │
│     ├── VRM: VC × PVE → Risk Matrix → Recommendation            │
│     ├── Assess: Confidence-based → Go/No-Go/Conditional         │
│     └── Reference audit table with all sources                  │
└─────────────────────────────────────────────────────────────────┘

Processing Stages Explained

Stage 1: Parallel Search Execution

The InvestigatorAgent ([investigator.py:200-280](src/vagent/agents/investigator.py)) generates 4 queries per requirement:

• Query 1: "{vendor_name} ({primary_domain}) {requirement}" — General search for third-party assessments
• Query 2: "{vendor_name} ({primary_domain})" "{requirement}" latest — Recent content from tech blogs/news
• Query 3: site:{primary_domain} {requirement} — Official vendor documentation
• Query 4: site:{primary_domain} "{requirement}" — Exact policy matches

Stage 2: Finding Extraction

Raw search results are processed by an LLM to extract structured findings:

• Claim: Clear summary of the finding (e.g., "Slack uses AES-256 encryption for data at rest...")
• Quotable Text: Direct quote from source (for verification)
• Confidence Score: Initial score based on source quality
• Tags: Category labels (domain, requirement)

Stage 3: Quality Confidence Adjustment

The _apply_quality_confidence_adjustment() method ([investigator.py:573-850](src/vagent/agents/investigator.py)) adjusts confidence based on multiple factors:

Stage 4: URL Validation

Before inclusion, all URLs are validated:

• HTTP 200: URL accessible → proceeds to content verification
• 404/410: URL not found → finding rejected
• Timeout: URL unreachable → finding rejected
• Redirects: Followed (max 5 hops) to reach final destination

Stage 5: Cross-Reference & Mapping

Findings are mapped to the appropriate framework:

• VRM Mode: Findings map to 7 maturity domain groups (Operational, Technical, Engineering, Risk Management & Supply Chain, Governance, Assurance, AI & Emerging Technology)
  • Each domain receives a score (1-5) based on finding quality and quantity
  • Scores combine into overall Vendor Capability (VC) score
• Assess Mode: Findings map to user-defined requirements
  • Each requirement is assessed against findings
  • Confidence-based filtering determines inclusion
• Cross-Validation: Vendor claims (site-scoped) are validated against third-party sources (general search)
  • Example: If vendor claims "SOC 2 certified" but general search finds no independent verification → flag for review
  • Discrepancies between marketing claims and actual security posture are highlighted

Stage 6: Adversarial Challenge

The DevilsAdvocateAgent ([advocate.py](src/vagent/agents/advocate.py)) challenges findings:

• Topic-specific negative queries: Each finding type generates targeted counter-queries
  • SOC2 → "vendor" security breach, "vendor" compliance exception
  • Data retention → "vendor" data retention issues, "vendor" privacy violation
  • Financial → "vendor" layoffs, "vendor" debt
• Weak signal search: Separate searches for emerging risks
  • Lawsuits & litigation (court cases, regulatory actions)
  • News & controversies (negative press, service outages)
  • CVEs & breaches (published vulnerabilities)
• Risk adjustment: 3+ high-severity counter-arguments increase residual risk

Stage 7: Final Output

Processed findings produce the final recommendation:

How Verification and Audit Works

The AuditorAgent validates findings through a multi-stage verification process:

- ✅ Certification & Infrastructure Validations — https://vendor.com/security ("Vendor holds ISO 27001 and SOC 2 Type 2...")
- ❌ Patching & Remediation — no supporting evidence found
- ❌ Customer-Data Training Safeguards — contradicted: "cannot opt out" (https://example.com/review)

Settings Audit Trail

All reports include a Settings Audit Trail section at the beginning that documents the exact configuration used for the assessment. This provides full transparency and reproducibility.

What's Included

Example Audit Trail

## Assessment Settings

**Mode:** ASSESS | **LLM:** anthropic/claude-sonnet-4-6 | **Search:** perplexity/sonar-pro | **Summary:** openai

**Agents:** parallel=4, searches=3, validation=Moderate

**Run options:** depth=standard | progressive search: off | search calls: 42 | skipped phases: all phases enabled

**Estimated run cost:** $0.1710

| Model | Calls | Input tokens | Output tokens | Est. cost (USD) |
|-------|-------|--------------|---------------|-----------------|
| gpt-4o | 8 | 57,176 | 2,803 | $0.1710 |
| **Total** | **8** | **57,176** | **2,803** | **$0.1710** |

*Generated: 2025-03-08T15:30:45+09:00*

Per-Run Token Usage & Cost Estimation

Every token-consuming call in a run is metered — agent LLM calls (all providers, including streaming Anthropic), web-search calls (Perplexity, OpenAI, Claude, Azure variants), background research, criterion-gap retries, and the evidence-review judge. At completion the run renders a per-model cost table into the report and stores the summary on the assessment record.

• Editable pricing table: rates live in src/vagent/model_pricing.json as USD per 1 million tokens (input / output), matched case-insensitively — exact model name first, then longest prefix (dated Azure deployments like gpt-5.4-pro-2026-03-05 match gpt-5.4-pro)
• Per-request fees: an optional per_call field adds a flat fee per API call (e.g. Perplexity's $5/1k searches)
• Unpriced models are never guessed: entries with null rates appear with cost “n/a” and are named in the headline — paste your negotiated/Azure contract rates into the JSON to price them (takes effect on the next run, no restart needed)
• Non-invasive by design: recording failures never break a run, and concurrent runs track independently

Why This Matters

• Reproducibility: Know exactly which providers and models were used
• Cost Tracking: Every run reports its actual token usage and estimated spend, so comparing provider/model combinations is data, not guesswork
• Quality Assurance: Verify settings match your intended configuration
• Audit Compliance: Demonstrate compliance with verification standards

Context-Aware Filtering Strategy

Vanguard-MAS uses a two-tier filtering approach that adapts based on the search context to ensure high-quality, relevant sources for each type of investigation.

How Adversarial Analysis Works

The Devil's Advocate uses topic-specific negative queries to systematically challenge each verified finding and identify hidden risks that could impact the vendor relationship.

Web Interface

Vanguard-MAS includes a React-based web interface for running assessments through a browser with real-time progress tracking.

Interface Screenshots

Homepage — Mode selection and quick access to all assessment types

VRM Mode — Interactive business impact assessment with real-time progress tracking

Features

Running the Web Interface

1. Start the API Server

# Activate the Python environment
source env/bin/activate

# Install with API dependencies
pip install -e .

# Start the API server
uvicorn vagent.api_server:app --reload --port 8000

2. Start the Frontend (in a new terminal)

cd web-frontend
npm install
npm run dev

3. Open in Browser

Navigate to http://localhost:5173

Domain Selection by Mode

Quick Assessment: Free-Text Risk Domains

Quick mode allows you to enter any risk domains as free text. The system provides suggestions but you can customize:

• Default domains: Cybersecurity, Financial Stability (pre-selected)
• Common suggestions: Cybersecurity, Financial Stability, Data Privacy, Compliance
• Fully customizable: Add or remove any domains (e.g., "Business Continuity", "Customer Support", "SLA Review")

Example Quick Assessment domains:
• Cybersecurity
• Financial Stability
• Data Privacy
• Compliance
• Business Continuity
• Customer Support

VRM Assessment: Structured Maturity Domains

VRM mode offers selective assessment across 7 maturity domain groups with 33 subdomains and vendor service type classification:

Assess Mode: Custom Risk Domains

Assess mode provides a form-based interface for fully customizable risk domains. You can add any number of risk domains with custom requirements:

• Vendor Profile: Enter vendor name, primary domain, industry sector, and tier
• Risk Domains: Add custom domains (e.g., "Cybersecurity", "Financial Stability") with priority levels
• Requirements: Add specific requirements for each domain to guide the assessment

Agent Parameters: Search provider, LLM provider, validation threshold, and parallel investigators are configured in Settings, not per assessment.

Note: For CLI usage, Assess mode can also accept a config.json file for pre-configured assessments. See the Configuration section for details.

REST API Endpoints

The FastAPI backend (vagent.api_server) exposes the full assessment lifecycle at http://localhost:8000 (interactive OpenAPI docs at /docs).

Run Assessments

Manage Assessment History

Configuration & Utilities

Installation

Requirements

• Python 3.10 or higher
• API keys for your chosen LLM and search providers

Setup

1. Clone the repository:

   git clone <repository-url>
   cd vanguard-mas

2. Create a virtual environment:

   python -m venv env
   source env/bin/activate  # On Windows: env\Scripts\activate

3. Install dependencies:

   pip install -e .

4. Configure environment variables:

   cp .env.example .env
   # Edit .env with your API keys

Configuration

Environment Variables

Create a .env file with the following:

# LLM Provider (for agents: Investigator, Analyst, Devil's Advocate, Auditor)
LLM_PROVIDER=anthropic  # Options: anthropic, openai, deepseek, azure_openai, azure_anthropic
ANTHROPIC_API_KEY=your_key_here
ANTHROPIC_API_BASE_URL=https://api.anthropic.com/v1
ANTHROPIC_MODEL=claude-sonnet-4-6

# OpenAI API (for LLM provider)
# Used when LLM_PROVIDER=openai for agent reasoning/analysis
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o
# Reasoning Effort (for OpenAI reasoning models like gpt-5-mini)
# Controls depth of reasoning for agent analysis tasks
# Options: low, medium, high
# NOTE: Only applies to reasoning models (gpt-5*, o1, o3, etc.)
OPENAI_REASONING_EFFORT=low

# DeepSeek API (for LLM provider)
DEEPSEEK_API_KEY=your_deepseek_api_key_here
DEEPSEEK_BASE_URL=https://api.deepseek.com/v1
DEEPSEEK_MODEL=deepseek-chat

# Summary/Report LLM role — powers the report-writing analyst in Assess/Quick
# mode (VRM/PVE reports are template-generated and don't use it).
# EMPTY (recommended) = same provider as LLM_PROVIDER. Per-provider models:
# SUMMARY_ANTHROPIC_MODEL, SUMMARY_OPENAI_MODEL, etc.
SUMMARY_LLM_PROVIDER=  # Options: anthropic, openai, deepseek, azure_openai, azure_anthropic

# Search Provider (for web search)
# Note: DeepSeek-direct has no web search, but DeepSeek deployments on Azure
# Foundry DO search (SEARCH_PROVIDER=azure_openai + a DeepSeek deployment name)
SEARCH_PROVIDER=perplexity  # Options: perplexity, openai, anthropic, azure_openai, azure_anthropic
PERPLEXITY_API_KEY=your_key_here
PERPLEXITY_MODEL=sonar-pro

# OpenAI Search (for web search functionality)
# Recommended: gpt-4o-mini (most economical, fast)
# Alternative: gpt-5-search-api (agentic, may include chain-of-thought)
OPENAI_SEARCH_MODEL=gpt-4o-mini
# Reasoning Effort (OpenAI reasoning models only)
# Controls depth of reasoning for supported models (gpt-5*, o1, o3, chatgpt-4, etc.)
# Options: low, medium, high
# NOTE: Only applies to reasoning models. Ignored for gpt-4o-mini
OPENAI_SEARCH_REASONING_EFFORT=low

# Anthropic Claude Web Search (requires Claude with web search tool)
# Uses ANTHROPIC_API_KEY from above
ANTHROPIC_SEARCH_MODEL=claude-sonnet-4-6

# Azure AI Foundry (for azure_openai / azure_anthropic as LLM, search, or summary provider)
AZURE_API_KEY=your_azure_key_here
AZURE_API_VERSION=2024-10-21
AZURE_ENDPOINT=https://your-resource.openai.azure.com          # Azure OpenAI endpoint
AZURE_ANTHROPIC_ENDPOINT=https://your-resource.services.ai.azure.com  # Azure Anthropic (Foundry) endpoint
AZURE_OPENAI_MODEL=gpt-5                # Deployment for agent analysis
AZURE_OPENAI_SEARCH_MODEL=gpt-5         # Deployment for web search
AZURE_ANTHROPIC_MODEL=claude-sonnet-4-6 # Deployment for agent analysis
AZURE_ANTHROPIC_SEARCH_MODEL=claude-sonnet-4-6  # Deployment for web search

# CourtListener (optional - court-record search for lawsuit detection)
# Free API key from courtlistener.com (Free Law Project); without it,
# litigation signals fall back to LLM web search automatically
COURTLISTENER_API=your_courtlistener_key_here

# Agent Configuration
MAX_PARALLEL_INVESTIGATORS=4

# Validation Threshold (Strict/Moderate/Relaxed)
# Controls how strictly findings are validated during audit (Assess mode only)
VALIDATION_THRESHOLD=Moderate

Launch Configuration

Create a config.json file:

{
  "project_metadata": {
    "report_id": "VNG-2026-0001",
    "requester": "Security Team",
    "timestamp": "2026-02-23T00:00:00Z"
  },
  "vendor_profile": {
    "entity_name": "Acme Corp",
    "primary_domain": "acme.com",
    "industry_sector": "SaaS",
    "vendor_tier": "Standard"
  },
  "risk_domains": [
    {
      "domain": "Cybersecurity",
      "priority": "High",
      "requirements": [
        "Verify SOC2 Type II compliance",
        "Check for recent data breaches"
      ]
    }
  ],
  "agent_parameters": {
    "search_provider": "anthropic",
    "llm_provider": "openai",
    "validation_threshold": "Moderate",
    "max_parallel_investigators": 4
  }
}

Validation Thresholds

The validation_threshold setting controls how strictly findings are validated during the Auditor phase:

When to use each:

• Strict: Final due diligence - only accept fully verified claims
• Moderate: Standard assessment - balance thoroughness with inclusiveness
• Relaxed: Initial discovery - include all accessible URLs for review

Note: Compliance claims (SOC2, ISO 27001, GDPR, etc.) ALWAYS require authoritative sources (vendor domain or official certification databases) regardless of threshold.

Usage

CLI Commands

Vanguard-MAS provides multiple CLI commands for different assessment types:

Full Assessment

Run with a configuration file:

vanguard assess config.json

With custom output:

vanguard assess config.json --output reports/vendor-x.md

Quick Assessment

vanguard quick --vendor "Acme Corp" --domain Cybersecurity --domain Financial

VRM Framework Assessment

# Required: --domain is essential for disambiguation
vanguard vrm --product "Buffer" --domain "buffer.com" --intended-use "Social media scheduling for marketing team"

# With pre-configured PVE classification
vanguard vrm --product "Slack" --domain "slack.com" --intended-use "Team communication platform" --pve High

# With custom output path
vanguard vrm --product "Zoom" --domain "zoom.us" --intended-use "Video conferencing" --output reports/zoom-vrm.md

Standalone PVE Assessment

Run a standalone Product Vulnerability Exposure analysis:

# Basic PVE assessment (interactive questions only)
vanguard pve --product "Slack" --domain "slack.com" --intended-use "Team communication"

# With background vendor research (more accurate data sensitivity)
vanguard pve --product "Slack" --domain "slack.com" --intended-use "Team communication" --research

# With custom output path
vanguard pve --product "Zoom" --domain "zoom.us" --intended-use "Video conferencing" --output reports/zoom-pve.md

Utility Commands

• Generate example config: vanguard init-config my-config.json
• View environment template: vanguard env-example

PVE Classification Options

Other Commands

• Generate example config: vanguard init-config my-config.json
• View environment template: vanguard env-example

Report Structure

Standard Assessment Report

1. Executive Summary: Go/No-Go/Conditional status with confidence
2. Assessment by Requirement: Requirement-by-requirement findings with evidence
3. Adversarial Risks: Counter-arguments from the Devil's Advocate
4. Weak Signals: Lawsuits, litigation, news, leadership changes
5. Residual Risk Profile: Risks regardless of decision
6. Breach Mitigation Analysis: Control adequacy when breaches found
7. Reference Audit: URL validation status table
8. Recommendation: Final decision with reasoning

VRM Framework Report

1. Vendor Background: Product description and core services
2. PVE Classification: Business impact-driven classification
3. Vendor Capability Assessment (v2): Evaluation across 7 maturity domain groups with 33 subdomains — AI & Emerging Technology scores as its own section, every section is coverage-annotated, and skipped subdomains render as “Skipped by configuration”
4. Initial Risk Matrix: VC × PVE → Inherent Risk
5. Mitigating Controls: Security controls that reduce risk
6. Adversarial Analysis: Devil's Advocate findings
7. Lawsuits & Litigation: Legal proceedings with case details
8. Residual Risk: Final risk after controls and adversarial findings
9. Conclusion: Recommendation with security requirements

Development

Project Structure

src/vagent/
├── __init__.py
├── config.py              # Configuration management
├── orchestrator.py        # Main workflow orchestration
├── main.py                # CLI interface
├── schemas/
│   └── agent_state.py     # Data models and schemas
├── agents/
│   ├── base.py            # Base agent class
│   ├── investigator.py    # Research agents
│   ├── auditor.py         # Fact-checking agent
│   ├── analyst.py         # Risk analysis agent
│   └── advocate.py        # Devil's Advocate agent
├── vrm/
│   ├── schema.py          # VRM framework data models
│   └── orchestrator.py    # VRM-specific workflow
└── tools/
    ├── search.py          # Search provider abstraction
    └── url_validator.py   # URL validation