Sebastian Gomez

Jun 24, 2026

Updated: Jun 25, 2026

Building a Real Time AI Race Coach: The Technical Deep Dive

Introduction

During the Gemini Field Test I helped build something that until recently felt like science fiction: an AI powered race coach able to analyze telemetry in real time and give instructions to the driver while they drive at more than 200 km/h on the Thunderhill Raceway circuit, in Sacramento, CA.

Vikram Tiwari already told the race day narrative in "The Race for Real Time", and Matt Thompson together with Ajeet Mirwani detailed the high level split brain architecture in "Beyond the Chatbot: A Blueprint for Trustable AI". This is the third article in the series, and it is the one that goes straight to the code.

Here we are going to take apart, piece by piece, how this system was built. From the SSE server that streams telemetry at 10Hz, through lap detection with Haversine and the friction circle for driving analysis, all the way to the integration with Gemini Nano running directly in the browser. Everything is organized into a 7 step progressive codelab that any developer can follow.

General Architecture

The system follows a decoupled architecture with four main layers:

• Telemetry server: Express.js serving CSV data via Server Sent Events at 10Hz.
• Frontend: React 19 + TypeScript consuming the stream via EventSource.
• 3D visualization: Three.js through React Three Fiber to render the circuit and the car.
• Multi level AI: a coaching pipeline that goes from deterministic logic, through Gemini Nano (on device via the Chrome Prompt API), up to Gemini Flash and Pro for audio generation.

The key to the design is what the companion articles call the "split brain architecture": the deterministic code handles the critical safety decisions (you do not need an LLM to say "BRAKE!"), while Gemini Nano enriches the messages with natural context. Each level has its role and its latency.

Steps 1-2: The Telemetry Server and Data Parsing

It all starts with data. The file scripts/telemetry-server.js is a minimalist Express server that reads a CSV of real telemetry captured at Thunderhill and streams it as an SSE stream.

app.get("/events", (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
  });

  let index = 0;
  const interval = setInterval(() => {
    if (index >= frames.length) {
      index = 0; // Loop the data
    }
    const frame = frames[index];
    res.write(`data: ${JSON.stringify(frame)}\n\n`);
    index++;
  }, 100); // 10Hz

  req.on("close", () => {
    clearInterval(interval);
  });
});

The SSE headers are critical: text/event-stream tells the browser to keep the connection open, no-cache prevents a proxy from interrupting the stream, and keep-alive keeps the TCP connection active. The 100ms interval gives us a 10Hz update frequency, enough for real time telemetry without saturating the browser.

Parsing DMS Coordinates

The GPS data in the CSV comes in DMS format (degrees, minutes, seconds), not in decimals. The parser uses a regex to extract the components and convert them:

function parseDMS(value: string): number {
  const match = value.match(/(\d+)°([\d.]+)\s*([NSEW])/);
  if (!match) return parseFloat(value) || 0;
  const degrees = parseInt(match[1], 10);
  const minutes = parseFloat(match[2]);
  const direction = match[3];
  let decimal = degrees + minutes / 60;
  if (direction === "S" || direction === "W") decimal = -decimal;
  return decimal;
}

The resulting TelemetryFrame has more than 23 fields: GPS position, speed, RPM, throttle and brake position, lateral and longitudinal G forces, oil and coolant temperature, oil pressure, battery voltage, terrain gradient, and more. PapaParse handles the CSV parsing in the frontend.

Step 3: Real Time Consumption with useRealtimeTelemetry

The useRealtimeTelemetry hook is the heart of the data system in the frontend. It connects to the SSE stream using EventSource and implements a reconnection pattern with exponential backoff:

const delay = Math.min(1000 * Math.pow(2, failedAttemptsRef.current), 30000);

This guarantees that if the server goes down, the client retries with growing intervals (1s, 2s, 4s, 8s...) up to a maximum of 30 seconds, avoiding bombarding a server that might be recovering.

The most interesting pattern is the dual frequency update. Not every component needs the data at the same speed:

const HISTORY_UPDATE_INTERVAL_MS = 200; // 5Hz for the map

// Full frequency for the gauges:
setCurrentFrame(frame);

// Throttled for the history:
if (now - lastHistoryUpdate.current > HISTORY_UPDATE_INTERVAL_MS) {
  setData(prev => [...prev, frame]);
  lastHistoryUpdate.current = now;
}

The speed, RPM, and G force indicators update at the full 10Hz from the server; they need that immediacy. But the map trail (which accumulates points in a growing array) is limited to 5Hz. Without this separation, the history array would grow at twice the speed and the map re render would consume unnecessary resources. The hook also supports the GPSD protocol for connecting to real GPS hardware.

Step 4: Lap Detection and the Ideal Lap

Lap detection requires knowing when the car crosses the finish line. Without a physical sensor, we use GPS and the Haversine formula to calculate distances between two points on the Earth's sphere:

function calcDistance(
  f1: {latitude: number; longitude: number},
  f2: {latitude: number; longitude: number}
) {
  const R = 6371e3;
  const dLat = ((f2.latitude - f1.latitude) * Math.PI) / 180;
  const dLon = ((f2.longitude - f1.longitude) * Math.PI) / 180;
  const a = Math.sin(dLat/2) * Math.sin(dLat/2) +
    Math.cos((f1.latitude * Math.PI)/180) *
    Math.cos((f2.latitude * Math.PI)/180) *
    Math.sin(dLon/2) * Math.sin(dLon/2);
  return R * 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
}

A finish line crossing is detected when the car is within 20 meters of the start point and at least 30 seconds have passed since the last crossing (to avoid false positives at the start of the lap). The heuristic search for the start point evaluates candidate positions every 5 seconds during the first 3 minutes of data.

The Ideal Lap

The resampleLap() function takes a raw lap (irregular points in time) and interpolates it linearly at regular intervals of 5 meters, creating a uniform representation of the trajectory. Then, calculateIdealLap() implements a micro sectors algorithm: it divides the circuit into 50 meter sectors and, for each sector, selects the fastest fragment among all completed laps. The result is a "frankenstein" lap that represents the best the driver achieved in each section of the circuit, not necessarily from the same lap.

Step 5: The Driving Analysis Engine (useDrivingAnalysis)

The useDrivingAnalysis hook analyzes a sliding window of 60 frames (6 seconds at 10Hz) to generate insights about the driving style.

Phase Detection

Each point is classified into a phase of the circuit based on the lateral G forces and the speed delta:

• Entry (corner entry): |latG| > 0.3 with decreasing speed.
• Apex (point of maximum turn): |latG| > 0.3 with stable speed.
• Exit (corner exit): |latG| > 0.3 with increasing speed.
• Straight: |latG| <= 0.3.

The Friction Circle

This is possibly the most revealing calculation in the whole system. The friction circle combines the lateral and longitudinal G forces to determine how much of the tire's available grip is being used:

const comboG = Math.sqrt(latG * latG + longG * longG);
const MAX_TIRE_G = 1.3;
const tireUsagePct = (comboG / MAX_TIRE_G) * 100;

if (tireUsagePct < 60) status = "COLD";
else if (tireUsagePct < 85) status = "UNDER_DRIVING";
else if (tireUsagePct < 105) status = "AT_LIMIT";
else status = "OVER_DRIVING";

The value MAX_TIRE_G = 1.3 represents the theoretical grip limit of the tires. An expert driver operates consistently between 85% and 105%. Below 60%, the tires are not generating enough temperature. Above 105%, the car is sliding, which is slower and dangerous. The engine also detects safety flags such as COASTING_DETECTED (throttle released without braking) and PANIC_BRAKE_IN_TURN (harsh braking in a corner).

Step 6: 3D Visualization with Three.js

The 3D visualization in TrackMap3D.tsx transforms GPS data into an interactive three dimensional scene using React Three Fiber.

Coordinate Projection

The GPS coordinates are projected to flat meters using the local approximation:

• latScale = 111000 (meters per degree of latitude, constant).
• lonScale = 111000 * cos(centerLat) (meters per degree of longitude, which varies with latitude).

The points are stored in a Float32Array to send them directly to the GPU without intermediate conversions. The car model includes arrow helpers that visualize the G force vector in real time.

Cameras and Car Dynamics

The system offers chase (third person) and bumper (first person) camera modes, both smoothed with linear interpolation (lerp). The car's orientation is calculated like this:

• Yaw (horizontal turn): atan2 between consecutive points.
• Pitch: terrain slope + gForceLong * 0.08.
• Roll: gForceLat * 0.15.

The factors 0.08 and 0.15 are empirical values that produce a visually convincing animation without exaggerating the movements.

Step 7: AI Coaching with Gemini Nano

This is where it all comes to life. Gemini Nano runs directly in the browser through the Chrome Prompt API (window.LanguageModel), eliminating network latency for the coaching responses.

Note: verify the state of the API before publishing or copying this code. The Chrome Prompt API is an experimental surface (origin trial) and has changed between Chrome versions: the window.LanguageModel global, the shape of the create() options, and outputLanguage have all shifted. The snippet below reflects the API as it stood in early 2026; pin the Chrome version you use and check the official documentation, because it may break in later versions.

const session = await window.LanguageModel!.create({
  outputLanguage: 'en',
  initialPrompts: [{
    role: "system",
    content: `You are a Race Spotter.
Check the "flags" in the input JSON. Priority is Top to Bottom.
PRIORITY 1: SAFETY
- If safety_status is "UNSTABLE" -> "Smooth it out! Reset."
PRIORITY 2: CRITICAL ERRORS
- If error_type is "LATE_BRAKE_T9" -> "BRAKE! Crest approaching!"
PRIORITY 3: PACE
- If opportunity is "UNDER_DRIVING_T5" -> "Trust the compression. Full throttle."`
  }]
});

The Priority Matrix

The coaching logic follows a strict hierarchy: SAFETY > CRITICAL ERRORS > PACE. This is fundamental. It does not matter if the driver could go faster in corner 5 if right now they are losing control in corner 9. The payload sent to Nano has this structure:

{
  "context": {
    "location": "T5_ENTRY",
    "speed": 145,
    "tire_status": "AT_LIMIT",
    "grip_pct": 92
  },
  "flags": {
    "safety_status": "STABLE",
    "error_type": null,
    "opportunity": "UNDER_DRIVING_T5"
  },
  "delta": -0.3
}

Deterministic Triggers

The system does not query Nano on every frame. It uses deterministic triggers with an 8 second cooldown between messages:

• Pace deviation: speed >10 km/h below the ideal lap for more than 3 seconds.
• Elevation change: terrain gradient >3% (relevant at Thunderhill, which has significant hills).
• New best lap: immediate celebration.
• Location change: new section of the circuit.

Only when a trigger fires does the code package the context and query Nano to enrich the message with natural language.

Predictive Coaching

The usePredictiveCoaching module implements a feature that takes coaching beyond the reactive: it predicts mistakes before they happen.

The system analyzes previous laps to detect "mistake zones", places where the driver lost more than 15 km/h relative to the ideal lap. Then, during the current lap, it calculates an anticipation point:

const speed = Math.max(currentFrame.speed, 50);
const lookaheadSeconds = 8;
const lookaheadMeters = (speed / 3.6) * lookaheadSeconds;
const targetDist = currentDist + lookaheadMeters;
const upcomingMistake = mistakeZones.find(z =>
  z.startDist >= targetDist - 30 && z.startDist <= targetDist + 30
);

At 150 km/h, the lookahead is about 333 meters: 8 seconds before reaching a problem zone, the driver gets a warning. The minimum speed is fixed at 50 km/h to avoid lookaheads that are too short in slow zones. The mistake zones are mapped to circuit points with a tolerance of 150 meters.

Text to Speech System

Coaching is only useful if the driver can hear it. The TTS system in useTTS.ts supports 4 providers:

• Browser SpeechSynthesis: native API, no network latency.
• Google Cloud TTS: high quality, requires an API key.
• Gemini Flash (Live WebSocket): audio streaming via WebSocket.
• Gemini Pro (generateContentStream): streaming audio generation. It is worth clarifying that this does not use a dedicated TTS model, but rather audio generated as model output.

For the providers that return raw PCM audio (without headers), the system builds a 44 byte WAV header:

function createWavHeader(dataLength: number, options: WavConversionOptions) {
  const { numChannels, sampleRate, bitsPerSample } = options;
  const buffer = new ArrayBuffer(44);
  const view = new DataView(buffer);
  writeString(0, "RIFF");
  view.setUint32(4, 36 + dataLength, true);
  writeString(8, "WAVE");
  writeString(12, "fmt ");
  // ... PCM format, sample rate, bit depth
  writeString(36, "data");
  view.setUint32(40, dataLength, true);
  return new Uint8Array(buffer);
}

Urgency Modulation

Not every message has the same priority, and the voice reflects it:

const urgencySettings = {
  URGENT: { rate: 1.5, pitch: 1.1, volume: 1.0 },
  CALM:   { rate: 0.9, pitch: 0.9, volume: 0.8 },
  STANDARD: { rate: 1.1, pitch: 1.0, volume: 0.9 }
};

A "BRAKE!" plays fast and loud. A "good pace, keep it up" comes through more relaxed. An isFetchingRef acts as a serialization lock to prevent multiple messages from overlapping.

Codelab Structure

The project is organized into 7 incremental branches (step-01 to step-07), each one building on the previous one: https://github.com/seagomezar/real-time-coach-codelab.

Note: verify that the public repository and the `step-01` to `step-07` branches exist and are reachable before following the codelab, since all the instructions depend on being able to clone it.

• Step 1, `step-01`: base project, React + Vite setup.
• Step 2, `step-02`: telemetry SSE server and data parsing.
• Step 3, `step-03`: useRealtimeTelemetry hook with backoff and dual frequency.
• Step 4, `step-04`: lap detection, Haversine, ideal lap.
• Step 5, `step-05`: analysis engine: friction circle, phases, flags.
• Step 6, `step-06`: 3D visualization with Three.js / React Three Fiber.
• Step 7, `step-07`: AI coaching with Gemini Nano, prediction, and TTS.

Each step has a self contained set of changes. A developer can run git checkout step-03, run npm install && npm run dev, and have a working system with real time telemetry and a 2D map, without needing AI or 3D yet.

Conclusion

Building a real time AI race coach left us with several clear lessons:

• Deterministic first, AI second: the critical safety decisions never depend on a language model. The AI enriches, it does not decide.
• Frequency matters: not everything needs to update at 10Hz. The dual frequency pattern applies to any system that combines real time data with expensive visualizations.
• On device AI changes the rules: Gemini Nano running in the browser eliminates network latency for the coaching responses. In a car at 200 km/h, every millisecond counts.
• Micro sectors are powerful: the idea of building an ideal lap by stitching together the best fragments of each lap is simple and effective.

For the complete race day narrative, read Vikram Tiwari's article "The Race for Real Time". To understand the philosophy of the split brain architecture and why separating deterministic logic from generative AI matters, Matt Thompson and Ajeet Mirwani's article "Beyond the Chatbot: A Blueprint for Trustable AI" is required reading.

This codelab is available as a public repository with the 7 progressive steps. Clone the repo, check out step-01, and build your own race coach.