Video-to-3D scene compiler with live AI-driven agent simulation. Upload a short video of any indoor space and watch it come alive as an interactive 3D scene populated with autonomous agents.

Video Upload → Gemini Analysis → ADK Compile Pipeline → CompiledScenePackage JSON → Three.js Simulation

1. Upload a short video (up to 20s) of an indoor space
2. Gemini 3.1 Pro analyzes the video — detects objects, people, zones, spatial layout
3. ADK pipeline compiles the analysis into a structured 3D scene with agents, furniture, navigation graphs
4. Three.js frontend renders the scene and runs a live simulation with utility-based AI agents

The server is stateless at runtime — the browser owns all live world state.

• Node.js 20+
• A Google AI API key (paid account)

# Clone and install
git clone <repo-url> && cd next-state
npm install

# Configure environment
cp .env.example .env  # or create .env manually

Add your API key to .env:

GEMINI_API_KEY=<your-key>
PORT=3001
NODE_ENV=development
CORS_ORIGIN=http://localhost:5173

# Terminal 1 — Backend (port 3001)
cd server && npm run dev

# Terminal 2 — Frontend (port 5173)
cd client && npm run dev

Open http://localhost:5173 and upload a video.

client/          React + Three.js frontend
  src/
    scene/       3D rendering (agents, environment, furniture, labels)
    store/       Zustand state management
    components/  UI panels (inspector, debug overlay, upload)
server/          Node.js API server
  src/
    adk/         Gemini ADK compile pipeline
      agents/    Structuring, style extraction
      prompts/   LLM prompt templates
shared/          Shared TypeScript types and Zod schemas

• Procedural 3D agents with animated walking, sitting, talking, fidgeting
• Agent props — laptops, phones, cups rendered on agent bodies
• Thought bubbles — pop up when agents change their intent
• Click-to-inspect — click any agent to see their mind state, traits, goals
• Furniture inference — deterministic infill adds chairs, tables, and venue-appropriate objects
• Density-aware population — scenes auto-populate with synthetic agents based on crowd density
• Style extraction — colors, materials, and lighting matched from the source video
• Utility-based AI — agents use softmax sampling over utility scores for natural behavior

cd server && npm run test
cd client && npm run test

Private — all rights reserved.