AI DEVELOPMENT • VOICE AI

AI Interview Platform

A real-time AI voice interview platform for technical and language assessments.

Next.jsGeminiVapiChromaDBNode.jsPostgreSQL
A

Project Snapshot

IndustryHuman Resources / EdTech
Project TypeAI Voice Platform & RAG System
RoleFull-Stack & AI Engineer
Timeline8 Weeks
TeamSolo Developer
PlatformWeb Application (Desktop & Mobile)
Core Stack
Next.jsGemini ProVapiChromaDBNode.jsPostgreSQLTailwind CSS

Project Overview

The AI Interview Platform is a real-time voice-first assessment tool that automates the screening process for technical and language recruiting. Instead of text-based multiple-choice questionnaires, candidates participate in a spoken dialogue with a low-latency voice agent. The system dynamically tailors questions based on the candidate's resume, evaluates responses for depth and correctness, and produces a structured evaluation report including a full transcript and rubric scores. This bridges the gap between impersonal coding screens and labor-intensive first-round phone calls.

The Problem

Initial recruitment screens suffer from high friction and operational overhead. Scheduling hundreds of first-round phone calls is slow, prone to interviewer bias, and difficult to manage across global timezones. On the other hand, traditional automated screening tests (like static multiple-choice questions or non-interactive code assessments) suffer from high drop-off rates, invite plagiarism, and fail to measure vital communications skills or reasoning depth. The challenge was to construct a screening experience that felt like talking to a human recruiter, operated at low latency (under 1.5 seconds of conversational turn-around), and could reliably grade responses across technical domains.

The Solution

We built a voice interface using Next.js, Vapi (for low-latency WebRTC speech-to-text and text-to-speech routing), and Gemini (for reasoning, question planning, and evaluations). ChromaDB acts as a vector database for semantic search, allowing the system to reference job rubrics and candidate resumes in real time during the dialogue. A Node.js backend handles session state transitions, saving data to a PostgreSQL database. By orchestrating WebRTC streams and LLM context management, we achieved natural voice interactions that adapt to candidates' answers while keeping them focused on the assessment criteria.

Architecture & Data Flow

The platform utilizes a low-latency WebRTC voice loop coordinated via a Next.js orchestration backend. Vapi acts as the telephony/voice gateway, handling speech-to-text (STT) and text-to-speech (TTS). Audio updates are pushed via WebSockets/Webhooks to the Next.js backend, which retrieves contextual evaluation guidelines from ChromaDB, constructs the prompt context for Gemini, and returns streaming responses.

Layer 1: Entry & Client

Candidate Browser

WebRTC Audio Stream & UI

Layer 2: Orchestration & Logic

Next.js App Router

Session State & Prompt Router

Layer 3: Storage & Services

Vapi Voice Gateway

Low-latency STT / TTS

Gemini 1.5 Pro

LLM Agent & Conversation Logic

ChromaDB Vector Store

RAG for Resume & Rubrics

PostgreSQL DB

Session Logs & Transcripts

Data Flow Connections

1
Candidate BrowserVapi Voice Gateway

WebRTC Audio Stream

2
Vapi Voice GatewayNext.js App Router

JSON Webhook (Speech Transcripts)

3
Next.js App RouterChromaDB Vector Store

Semantic Lookup (Skills & Criteria)

4
Next.js App RouterGemini 1.5 Pro

Dynamic Context & Prompts

5
Gemini 1.5 ProNext.js App Router

Response Instruction Stream

6
Next.js App RouterVapi Voice Gateway

Send Answer text

7
Next.js App RouterPostgreSQL DB

Log Transcripts & Scores

Key Features

Core capabilities implemented to fulfill business requirements and user needs.

Real-time Conversational Voice

Engage candidates in natural spoken dialogue via WebRTC with latency under 1.5 seconds.

Resume-Aware RAG Integration

ChromaDB searches the candidate's resume and the job description to ask contextually relevant follow-ups.

Dynamic Question Sequencing

The AI interviewer transitions smoothly from introductions to resume checks, technical scenarios, and final wrap-up.

Multidimensional AI Grading

Grading algorithms evaluate technical correctness, communication clarity, problem-solving structure, and role fit.

Interactive Candidate Dashboard

Secure portal for candidates to test mic/audio inputs, start interviews, and view application status.

Recruiter Admin Dashboard

Full pipeline view with candidate transcripts, audio playbacks, automated feedback summaries, and grading rubrics.

Technical Implementation

Deep dive into code execution, state orchestrations, and API integrations for core modules.

1. Voice State Orchestration with Vapi Webhooks

To prevent the conversational agent from losing context, the Next.js API acts as a session manager. Every time Vapi detects speech completion, it sends a payload to our webhook. The backend inspects the active session state (Introduction, Resume Check, Technical Phase, Closing) and retrieves relevant guidelines.

app/api/vapi-webhook/route.tstypescript
import { NextRequest, NextResponse } from "next/server";
import { getSessionState, updateSessionState } from "@/lib/db";
import { generateNextQuestion } from "@/lib/gemini";

export async function POST(req: NextRequest) {
  const payload = await req.json();
  
  if (payload.message.type === "assistant-request") {
    const { sessionId, transcript } = payload.message;
    const session = await getSessionState(sessionId);
    
    // Retrieve resume context and rubrics from ChromaDB RAG
    const ragContext = await getRAGContext(session.candidateId, transcript);
    
    // Generate next question using Gemini
    const { responseText, nextState } = await generateNextQuestion({
      currentState: session.state,
      history: session.history,
      userResponse: transcript,
      context: ragContext
    });
    
    await updateSessionState(sessionId, {
      state: nextState,
      history: [...session.history, { role: "user", text: transcript }, { role: "assistant", text: responseText }]
    });
    
    return NextResponse.json({
      response: responseText
    });
  }
  
  return NextResponse.json({ status: "ignored" });
}

2. RAG Retrieval via ChromaDB

To make the interview highly relevant, candidates' resumes and job descriptions are parsed and stored as vector embeddings. During the interview, semantic searches fetch specific technical topics corresponding to candidate responses, preventing generic or irrelevant questioning.

lib/rag.tstypescript
import { ChromaClient } from "chromadb";
import { getGeminiEmbeddings } from "./gemini";

const chroma = new ChromaClient();

export async function getRAGContext(candidateId: string, transcript: string) {
  const collection = await chroma.getCollection({ name: `candidate-${candidateId}-data` });
  const queryEmbedding = await getGeminiEmbeddings(transcript);
  
  const results = await collection.query({
    queryEmbeddings: [queryEmbedding],
    nResults: 3
  });
  
  return results.documents.flat().join("\n");
}

Technical Challenges & Solutions

Deep engineering obstacles solved during development, detailing specific logic, tradeoffs, and outcomes.

C1

Challenge

Conversational Latency and Response Delays

Resolution

Using streaming API loops and optimized prompts.

Initial attempts with standard REST loops caused turn-around latency of over 3.5 seconds, making the conversation feel sluggish. We refactored the speech gateway to stream responses directly to Vapi. By running pre-compiled prompts, caching system instructions, and utilizing Gemini 1.5 Flash for sub-queries, we brought overall latency down to ~1.3 seconds.

C2

Challenge

Managing Interruption Handling

Resolution

WebRTC connection monitoring and Vapi interrupt callbacks.

If a candidate starts speaking while the AI is talking, the AI must stop immediately to feel natural. We integrated Vapi's WebRTC interrupt thresholding, configuring the Next.js session manager to immediately cancel active LLM generation streams when an interruption event payload is received, preserving the dialogue rhythm.

C3

Challenge

Grading Reliability and Consistency

Resolution

Structured output parsing with detailed grading schemas.

LLMs can be inconsistent when grading technical responses. To resolve this, we configured the post-interview evaluation script to run double-pass validations. The first pass extracts exact factual points mentioned. The second pass maps these points against a strict rubric JSON schema. If the evaluations diverge, the script highlights the discrepancies for manual recruiter review.

Results & Engineering Outcomes

Concrete metrics and operational upgrades delivered through the completed project.

1.3s

Average Latency

Achieved sub-1.5 second turnaround time for speech synthesis and response generation, ensuring conversation flows naturally.

80%

Scheduling Load Reduction

Recruiters bypassed first-round scheduling, relying entirely on AI voice screens to filter top-tier talent.

High Candidate Completion Rate

Interactive voice screens recorded a 92% completion rate compared to traditional asynchronous coding screens.

Interface Gallery & Screenshots

Visual interface layout and user dashboards designed for the system.

Candidate Screen

Real-time Voice Screening Session

Live Connection
AI

"I noticed you mentioned working with real-time stream processing on your resume. Could you explain how you optimized WebSocket connections to minimize data lag?"

"Yes, in my last project we configured a Redis-backed message adapter and throttled payload pushes. This allowed us to preserve real-time updates while avoiding queue overflow."

User

Technology Stack

Detailed breakdown of the architectural choices and engineering considerations behind each technology.

TechnologyArchitectural RoleSelection Rationale
Next.jsFrontend & APIsProvides Server Components for fast initial dashboard loads, and API routes to handle Webhook payloads from the voice gateway.
Gemini ProConversational BrainUtilized for deep reasoning capabilities, parsing resumes, dynamic conversation management, and scoring rubrics.
VapiVoice LayerActs as our WebRTC gateway, converting candidates' audio to text in real-time and synthesizing AI responses back into spoken audio.
ChromaDBRAG DatabaseStores resume chunks and grading benchmarks to fetch candidate-specific experience details dynamically.

Frequently Asked Questions

Answers to technical details, implementation scopes, and operations related to the project.

The speech-to-text parser runs on top of noise-canceling WebRTC streaming. Under the hood, Vapi is configured with advanced acoustic filters, making the translation layer highly resilient to background murmurs, hums, and diverse regional accents.
Yes, candidates can interrupt the AI interviewer at any point. When speech is detected while the AI is speaking, the gateway stops playing the current audio stream immediately and sends an interruption hook to the backend.
We implement guardrails by validating candidate input length, running safety filters, and restricting the system prompt state machine. Because the conversation is voice-first and dynamically references the resume, standard copy-paste cheating is neutralized.

Related Solutions

Need a similar platform?

Explore how I build, optimize, and launch technical solutions across these service verticals.

Work with me

Have a similar project in mind?

I can help design, build, and deploy custom AI solutions, SaaS portals, and full-stack web platforms. Tell me about your requirements — no sales calls required.

Discuss Your Project →