How AI Reverse Scripts Work: Turning Raw Footage into Structured Video Narratives

1. What a reverse script is (and what it is not)

A reverse script is a machine-generated narrative layer that describes what actually happened in the source footage, organized by scene, speaker, and story beat. It is not a transcript. It is not a shot list. It is not a final editing script. It is the intermediate output of a video understanding pipeline — a bridge between raw footage and an editable timeline. Think of it as the AI watching your videos and writing down what it sees, in story order, with timecodes, character labels, and suggested groupings.

• It captures scene boundaries, dialogue segments, character appearances, and narrative arcs.
• It is structured enough to feed into a timeline editor, but flexible enough for human review.
• It works across multiple source files, maintaining character and story continuity.

2. The video understanding pipeline that creates a reverse script

Building a reverse script requires multiple AI models working in sequence. The pipeline begins with raw footage and ends with a readable narrative. Each stage produces structured data that feeds into the next, creating layers of understanding that compound in value.

• Scene detection splits the video into coherent segments using models like TransNetV2.
• ASR transcribes speech with speaker diarization — labeling who said what and when.
• Entity recognition identifies faces and objects, then clusters recurring identities across scenes.
• Narrative composition stitches everything into scene-by-scene story beats.

3. Scene detection: finding where the story changes

Before AI can understand what happens in a video, it needs to know where one scene ends and the next begins. Scene detection models analyze visual transitions — cuts, fades, and shot changes — to produce scene boundaries with frame-accurate timestamps. Each scene segment becomes a container for the dialogue, entities, and narrative summaries that follow. Without reliable scene boundaries, the reverse script would lose its structural backbone.

4. ASR and speaker diarization: knowing who said what

Speech recognition transcribes every spoken word, but a transcript alone is just walls of text. Speaker diarization assigns each utterance to a specific speaker, labeling voices as Speaker A, Speaker B, and so on. When combined with entity recognition, these labels can be mapped to named characters. The result is a time-aligned transcript where you can see not just what was said, but who said it and when — essential for dialogue-heavy edits like interviews, debates, and story recaps.

5. Entity recognition: building a project-level character and object library

A single character may appear in dozens of scenes with different lighting, angles, and expressions. Entity recognition uses visual embeddings — vector representations of detected faces and objects — to cluster similar appearances together. HDBSCAN clustering groups these embeddings into identity clusters, so the system knows when the same person appears across multiple scenes. This identity library persists at the project level, meaning when you add more footage to the same project, new appearances of known characters are automatically recognized.

6. Narrative composition: from structured data to readable script

The final stage takes all the structured outputs — scene boundaries, time-aligned transcripts with speaker labels, entity clusters, and key frames — and feeds them to a large language model. The model processes segments of approximately 200 seconds at a time, producing scene-by-scene narrative summaries with dialogue references, character descriptions, and suggested groupings. This is the reverse script: a human-readable, structured document that reads like a story outline but is backed by frame-accurate metadata.

7. Using the reverse script for editing

Once the reverse script is generated, it becomes the editing map. The script planner agent reads the entire reverse script, understands the full narrative arc, and can respond to editing requests like 'make an emotional highlight reel' or 'focus on character interactions.' It selects relevant clips, writes narration text, and produces a structured timeline JSON. The reverse script bridges the gap between raw understanding and creative editing decisions.