AI Interview Platform
A real-time AI voice interview platform for technical and language assessments.
Project Snapshot
Project Overview
The AI Interview Platform is a real-time voice-first assessment tool that automates the screening process for technical and language recruiting. Instead of text-based multiple-choice questionnaires, candidates participate in a spoken dialogue with a low-latency voice agent. The system dynamically tailors questions based on the candidate's resume, evaluates responses for depth and correctness, and produces a structured evaluation report including a full transcript and rubric scores. This bridges the gap between impersonal coding screens and labor-intensive first-round phone calls.
The Problem
Initial recruitment screens suffer from high friction and operational overhead. Scheduling hundreds of first-round phone calls is slow, prone to interviewer bias, and difficult to manage across global timezones. On the other hand, traditional automated screening tests (like static multiple-choice questions or non-interactive code assessments) suffer from high drop-off rates, invite plagiarism, and fail to measure vital communications skills or reasoning depth. The challenge was to construct a screening experience that felt like talking to a human recruiter, operated at low latency (under 1.5 seconds of conversational turn-around), and could reliably grade responses across technical domains.
The Solution
We built a voice interface using Next.js, Vapi (for low-latency WebRTC speech-to-text and text-to-speech routing), and Gemini (for reasoning, question planning, and evaluations). ChromaDB acts as a vector database for semantic search, allowing the system to reference job rubrics and candidate resumes in real time during the dialogue. A Node.js backend handles session state transitions, saving data to a PostgreSQL database. By orchestrating WebRTC streams and LLM context management, we achieved natural voice interactions that adapt to candidates' answers while keeping them focused on the assessment criteria.
Architecture & Data Flow
The platform utilizes a low-latency WebRTC voice loop coordinated via a Next.js orchestration backend. Vapi acts as the telephony/voice gateway, handling speech-to-text (STT) and text-to-speech (TTS). Audio updates are pushed via WebSockets/Webhooks to the Next.js backend, which retrieves contextual evaluation guidelines from ChromaDB, constructs the prompt context for Gemini, and returns streaming responses.
Candidate Browser
WebRTC Audio Stream & UI
Next.js App Router
Session State & Prompt Router
Vapi Voice Gateway
Low-latency STT / TTS
Gemini 1.5 Pro
LLM Agent & Conversation Logic
ChromaDB Vector Store
RAG for Resume & Rubrics
PostgreSQL DB
Session Logs & Transcripts
Data Flow Connections
WebRTC Audio Stream
JSON Webhook (Speech Transcripts)
Semantic Lookup (Skills & Criteria)
Dynamic Context & Prompts
Response Instruction Stream
Send Answer text
Log Transcripts & Scores
Key Features
Core capabilities implemented to fulfill business requirements and user needs.
Real-time Conversational Voice
Engage candidates in natural spoken dialogue via WebRTC with latency under 1.5 seconds.
Resume-Aware RAG Integration
ChromaDB searches the candidate's resume and the job description to ask contextually relevant follow-ups.
Dynamic Question Sequencing
The AI interviewer transitions smoothly from introductions to resume checks, technical scenarios, and final wrap-up.
Multidimensional AI Grading
Grading algorithms evaluate technical correctness, communication clarity, problem-solving structure, and role fit.
Interactive Candidate Dashboard
Secure portal for candidates to test mic/audio inputs, start interviews, and view application status.
Recruiter Admin Dashboard
Full pipeline view with candidate transcripts, audio playbacks, automated feedback summaries, and grading rubrics.
Technical Implementation
Deep dive into code execution, state orchestrations, and API integrations for core modules.
1. Voice State Orchestration with Vapi Webhooks
To prevent the conversational agent from losing context, the Next.js API acts as a session manager. Every time Vapi detects speech completion, it sends a payload to our webhook. The backend inspects the active session state (Introduction, Resume Check, Technical Phase, Closing) and retrieves relevant guidelines.
import { NextRequest, NextResponse } from "next/server";
import { getSessionState, updateSessionState } from "@/lib/db";
import { generateNextQuestion } from "@/lib/gemini";
export async function POST(req: NextRequest) {
const payload = await req.json();
if (payload.message.type === "assistant-request") {
const { sessionId, transcript } = payload.message;
const session = await getSessionState(sessionId);
// Retrieve resume context and rubrics from ChromaDB RAG
const ragContext = await getRAGContext(session.candidateId, transcript);
// Generate next question using Gemini
const { responseText, nextState } = await generateNextQuestion({
currentState: session.state,
history: session.history,
userResponse: transcript,
context: ragContext
});
await updateSessionState(sessionId, {
state: nextState,
history: [...session.history, { role: "user", text: transcript }, { role: "assistant", text: responseText }]
});
return NextResponse.json({
response: responseText
});
}
return NextResponse.json({ status: "ignored" });
}2. RAG Retrieval via ChromaDB
To make the interview highly relevant, candidates' resumes and job descriptions are parsed and stored as vector embeddings. During the interview, semantic searches fetch specific technical topics corresponding to candidate responses, preventing generic or irrelevant questioning.
import { ChromaClient } from "chromadb";
import { getGeminiEmbeddings } from "./gemini";
const chroma = new ChromaClient();
export async function getRAGContext(candidateId: string, transcript: string) {
const collection = await chroma.getCollection({ name: `candidate-${candidateId}-data` });
const queryEmbedding = await getGeminiEmbeddings(transcript);
const results = await collection.query({
queryEmbeddings: [queryEmbedding],
nResults: 3
});
return results.documents.flat().join("\n");
}Technical Challenges & Solutions
Deep engineering obstacles solved during development, detailing specific logic, tradeoffs, and outcomes.
Challenge
Conversational Latency and Response Delays
Resolution
Using streaming API loops and optimized prompts.
Initial attempts with standard REST loops caused turn-around latency of over 3.5 seconds, making the conversation feel sluggish. We refactored the speech gateway to stream responses directly to Vapi. By running pre-compiled prompts, caching system instructions, and utilizing Gemini 1.5 Flash for sub-queries, we brought overall latency down to ~1.3 seconds.
Challenge
Managing Interruption Handling
Resolution
WebRTC connection monitoring and Vapi interrupt callbacks.
If a candidate starts speaking while the AI is talking, the AI must stop immediately to feel natural. We integrated Vapi's WebRTC interrupt thresholding, configuring the Next.js session manager to immediately cancel active LLM generation streams when an interruption event payload is received, preserving the dialogue rhythm.
Challenge
Grading Reliability and Consistency
Resolution
Structured output parsing with detailed grading schemas.
LLMs can be inconsistent when grading technical responses. To resolve this, we configured the post-interview evaluation script to run double-pass validations. The first pass extracts exact factual points mentioned. The second pass maps these points against a strict rubric JSON schema. If the evaluations diverge, the script highlights the discrepancies for manual recruiter review.
Results & Engineering Outcomes
Concrete metrics and operational upgrades delivered through the completed project.
Average Latency
Achieved sub-1.5 second turnaround time for speech synthesis and response generation, ensuring conversation flows naturally.
Scheduling Load Reduction
Recruiters bypassed first-round scheduling, relying entirely on AI voice screens to filter top-tier talent.
High Candidate Completion Rate
Interactive voice screens recorded a 92% completion rate compared to traditional asynchronous coding screens.
Interface Gallery & Screenshots
Visual interface layout and user dashboards designed for the system.
Candidate Screen
Real-time Voice Screening Session
"I noticed you mentioned working with real-time stream processing on your resume. Could you explain how you optimized WebSocket connections to minimize data lag?"
"Yes, in my last project we configured a Redis-backed message adapter and throttled payload pushes. This allowed us to preserve real-time updates while avoiding queue overflow."
UserTechnology Stack
Detailed breakdown of the architectural choices and engineering considerations behind each technology.
| Technology | Architectural Role | Selection Rationale |
|---|---|---|
| Next.js | Frontend & APIs | Provides Server Components for fast initial dashboard loads, and API routes to handle Webhook payloads from the voice gateway. |
| Gemini Pro | Conversational Brain | Utilized for deep reasoning capabilities, parsing resumes, dynamic conversation management, and scoring rubrics. |
| Vapi | Voice Layer | Acts as our WebRTC gateway, converting candidates' audio to text in real-time and synthesizing AI responses back into spoken audio. |
| ChromaDB | RAG Database | Stores resume chunks and grading benchmarks to fetch candidate-specific experience details dynamically. |
Frequently Asked Questions
Answers to technical details, implementation scopes, and operations related to the project.
Related Solutions
Need a similar platform?
Explore how I build, optimize, and launch technical solutions across these service verticals.
Have a similar project in mind?
I can help design, build, and deploy custom AI solutions, SaaS portals, and full-stack web platforms. Tell me about your requirements — no sales calls required.