VVaaniEvalOpen app

Open-source voice AI evaluation

Find the production calls your voice agent team needs to fix.

VaaniEval turns real voice-agent conversations into reviewable evidence, quality scores, and clear engineering feedback.

Apply for the free pilotView on GitHub

Self-hosted · MIT licensed · Built for ElevenLabs and Vapi workflows

Conversation review
Task completion86%
Quality trend+12%

Customer I need to reschedule my appointment.

Agent I can help with that. What day works best?

Evaluation findingResolved with a clear next step
Built for production workflows usingElevenLabsVapiOpenAIAnthropic

A tighter QA loop

Move from call volume to specific, reviewable failures.

01

Conversation evidence

Review transcripts, audio, metadata, and evaluation results in one focused workspace.

02

Actionable evaluation

Score task completion, unsupported claims, fallback behavior, and operational quality.

03

Production visibility

Track quality trends by agent and move from aggregate signals back to individual calls.

04

Provider flexibility

Import conversations from ElevenLabs or Vapi and choose how evaluator models are configured.

How it works

From provider import to verified fix.

Keep the evidence attached to every score so reviewers can understand what happened, not only whether a metric passed.

  1. 1
    Import production calls

    Normalize transcripts, audio, and metadata from supported voice platforms.

  2. 2
    Evaluate consistent behaviors

    Run repeatable scorecards with rationales and conversation evidence.

  3. 3
    Find patterns and fix agents

    Compare quality trends and inspect the exact calls behind a regression.

Why Self-Hosted Open Source

Evaluate calls without exposing sensitive customer data.

Closed-source SaaS observability platforms require routing raw call recordings and transcripts to third-party databases, presenting immediate compliance risks. VaaniEval keeps all evaluation traces inside your secure perimeter.

🔒

Data Residency & Privacy

Keep all high-fidelity recordings, transcripts, and evaluation runs inside your own VPC (AWS, GCP, or Azure). Fully comply with GDPR and local data residency laws.

🏥

HIPAA & PCI Guardrails

Route evaluations directly to your enterprise-approved Azure OpenAI or local models. Eliminate third-party SaaS middleware, hidden sub-processors, and data training leaks.

🔎

White-Box Auditability

Inspect 100% of the FastAPI and React codebase. Verify that credentials encrypted using your CREDENTIAL_ENCRYPTION_KEY stay secure, and customize ingestion filters to mask PII.

Field notes

Practical guidance for voice-agent teams.

Read all articles →
Voice AI QA

A practical QA process for production voice agents

How to replace random call sampling with a repeatable conversation review and evaluation workflow.

7 min readEvaluation

Voice AI evaluation metrics that lead to actionable fixes

A focused metric framework for task success, hallucinations, fallback behavior, and operational quality.

6 min readCompliance & Security

Self-Hosted vs. Closed-Source Voice AI Evaluations: A Compliance and Privacy Analysis

How closed-source SaaS evaluation tools expose sensitive call data, and how a self-hosted open-source framework like VaaniEval secures HIPAA, GDPR, and PCI compliance.

10 min read

Design partner program

Build a QA process around your real production calls.

We are working with voice-AI teams to shape the next version of VaaniEval.

Apply for the pilot