Developer note: Series 3, Episode 5 deep dive for Paragon AI MAG. How does AI video generation work? The mechanism: a diffusion transformer (DiT) denoises latent representations — video compressed into latent space, chopped into 3D spacetime patches that act like text tokens, denoised over 20 to 50 passes with text conditioning, native audio generated in the same pass as the pixels. We also cover why clips stay under 30 seconds (compute grows with every frame, small errors compound into physical drift), the 2026 leaders (Veo 3.1, Kling 4, Seedance 2.5, Runway Gen-4.5), Veo 3.1 API rates and Flow credits, and the 6-element prompt formula. Canonical URL: https://paragonaimag.substack.com/p/how-ai-video-generation-works
By Paragon AI MAG — Series 3, Episode 5, "How the AI Tools You Use Actually Work" — 17 November 2026
Watch the video: https://youtu.be/Ew7yFjZX92w
In February 2024, filmmaker Tyler Perry watched a demonstration of OpenAI's Sora, an AI system that could turn a written sentence into realistic video footage. Days later, Perry announced he was halting an $800 million expansion of his Atlanta studio — a four-year plan to add 12 soundstages across his 330-acre campus at Fort McPherson. "All of that is currently and indefinitely on hold because of Sora and what I'm seeing," he told The Hollywood Reporter, voicing direct concern for the jobs of editors, visual-effects artists, and animators.
Yet by late 2026, the tool that halted that studio build no longer exists. OpenAI shut down the Sora app on April 26, 2026, and retired the Sora API on September 24, 2026. The technology itself has not disappeared. But understanding how generative video actually works explains both why Hollywood paused — and why the medium is still bound to short clips, shaky physics, and vendors that vanish overnight.
The Fall of Sora: A Two-Year Timeline
Sora was revealed on February 15, 2024, generating minute-long clips that looked filmed rather than computed. It opened to the public in December 2024. A second version, Sora 2, arrived on September 30, 2025, with synchronized audio and an invite-only iOS app. Then, in April 2026, the consumer app closed its doors. By July, ByteDance had launched Seedance 2.5. In September, the Sora API shut down, leaving enterprise users stranded. One month later, in October, Kling 4 launched to fill the void. The lesson from that timeline is the first thing to carry into any production plan: never build a pipeline on a single proprietary model. If a giant like Sora can vanish overnight, literally any vendor can. Stay agile.
The Engine: Sculpting Static
A video AI never films anything. It does not pull clips from a stock library or stitch together recorded footage. It begins with pure random noise — the digital equivalent of television static — and systematically strips that noise away across 20 to 50 passes, guided by a text prompt, until a coherent scene resolves. The process is called diffusion. During training, neural networks study millions of video clips paired with descriptive captions. Engineers gradually corrupt those clips by adding noise until the frames are indistinguishable from static. The model learns to reverse that corruption, step by step. Across vast datasets, it builds an internal map of lighting, fabric, camera moves, and spatial continuity. At generation time, the sequence runs in reverse, in three parts. First, latent-space compression: raw video is too data-heavy to diffuse pixel by pixel, so an encoder squeezes the footage into a compact internal representation — latent space — where the heavy computation happens, before a decoder turns it back into frames. Second, text conditioning: a text encoder turns your prompt into mathematical embeddings that steer each pass. Third, iterative denoising: the network starts with random static across all frames and settles the global composition in the early passes, then fine-tunes details like reflections and edges in the later ones.
Spacetime Patches: The Diffusion Transformer
Early video models processed footage frame by frame with convolutional networks, so background objects warped, faces morphed, and clothing colors flickered from one frame to the next. Modern systems pair diffusion with transformer architectures, forming a diffusion transformer. The model divides the video into small 3D units called spacetime patches — blocks that capture a patch of pixels across a brief span of time. Spacetime patches are to video what tokens are to text. Just as a language model weighs the relationships between words in a sentence, a diffusion transformer uses self-attention to cross-reference every patch across space and time at once. A patch showing a subject's face in the opening frames attends directly to the face in later frames, keeping the person visually consistent from beginning to end. That is the whole trick: the model predicts the next patch the way a language model predicts the next word.
Why the Physics Breaks — and Why Clips Stay Short
A video AI does not simulate the real world. It has no programmed understanding of gravity, fluid mechanics, or mass. It relies entirely on statistical associations in its training data. So physical interactions glitch: a hand reaching for a coffee cup can melt into the porcelain; liquid poured into a glass can float or clip through the rim; background text morphs into meaningless alien runes as the camera moves past it, because the model understands the shape of a sign, not the alphabet. Clips stay short for three reasons. First, the transformer compares patches across the entire clip to keep things consistent, so the compute required grows steeply with every added frame. Second, small errors compound: the model starts to forget what happened in frame one by the time it reaches frame 1,000, so by second 40 a face can shift, a logo can warp, or a jacket can change shade. This is called physical drift. Third, pricing bills by the second — a two-minute clip that comes out warped and unusable is an expensive mistake for both the user and the server.
The 2026 Leaders
With Sora retired, four models lead commercial production in late 2026: Google Veo 3.1 is the all-rounder: the best overall quality and prompt accuracy, native audio, and 4K video. Dialogue, sound effects, and ambient noise are denoised in the same pass as the visuals. Its clips max out around 8 seconds. Kling 4 is the cinematic and social specialist from Kuaishou: native 4K at 60fps, clips up to 30 seconds, and multi-shot camera sequences. ByteDance Seedance 2.5 is the cost-and-realism leader: clips up to 30 seconds with joint audio-video generation at about $0.047 per second. Runway Gen-4.5 is the creative control workstation: reference-image controls, character consistency, and the best setup for a production team that needs to mask, edit, and version videos together. The quick verdict for 2026, by job: use Veo 3.1 for the best overall quality and prompt accuracy; Kling 4 for long cinematic talking heads; Seedance 2.5 for high-volume budget testing; Runway Gen-4.5 if you need strict visual consistency across multiple shots. Most commercial teams end up on a two-model strategy — drafting bulk footage cheaply on Seedance 2.5, reserving Veo 3.1 or Runway Gen-4.5 for hero shots.
What It Actually Costs
The number that changes the economics is 4.7 cents. Seedance 2.5 drops the floor to just under 5 cents a second, which fundamentally changes how performance marketers can test ad variants: when volume and cost per clip matter more than maximum cinematic beauty, Seedance wins that war every time. Veo 3.1 works differently. On Google Flow, every account gets 50 free credits a day; paid plans start at $4.99 a month for Google AI Plus (200 monthly credits) and $19.99 a month for Google AI Pro (1,000 monthly credits). Generating a clip costs 10 credits on Lite, 20 on Fast, and 100 on Quality. On the Gemini API, at 720p with audio, Lite is $0.05 a second, Fast is $0.10, and Standard is $0.40. An 8-second 1080p clip with audio runs about $0.64 on Lite, $0.96 on Fast, and $3.20 on Standard — and you pay only for clips that are successfully generated.
The 6-Element Prompt Formula
A mediocre prompt on the best model will always lose to a great prompt on an average one. If you are a brand trying to keep a specific sneaker or soda can looking identical across 20 shots, text-to-video alone will hallucinate a slightly different logo every time. The fix is image-to-video: give the model a fixed starting frame, and you are no longer asking it to invent a product from scratch — you are asking it to animate what you have already locked in. The formula itself is simple. Treat the model like a literal film crew and explicitly dictate six elements: Subject + Setting + Camera Framing + Lighting + Motion + Sound. For example: "A baker in a flour-dusted apron slides a tray of croissants onto the counter and says 'Last batch of the day!' Soft rain on the glass, a bell rings, warm light, handheld camera, 10 seconds." Leave any element blank and you force the model to guess — and its guesses will change every time you hit generate, burning through your credits and ruining your consistency. And whenever a specific face, product, or wardrobe item must stay consistent, provide a reference image. A single clear reference anchors details far more reliably than an extra paragraph of text. Budget for 3 to 5 generation attempts to land one usable clip.
How to Spot an AI Video in Your Feed
Three tells still break the illusion. First, hands and physical contact: fingers merging into cups, cutlery fusing, objects clipping through each other. Second, background text: letters that morph into meaningless runes as the camera moves. Third, clip length: most AI-generated clips in social feeds are still under 10 seconds, because of the compute and consistency limits above. Two things that used to be tells no longer are. Watermarks still exist — Google embeds an imperceptible SynthID watermark in every Veo frame, and Sora carried a visible watermark plus C2PA metadata — but you cannot see SynthID with your eyes. And bad audio is no longer a tell at all. AI video used to be silent, with sound dubbed on later. Today, engines like Veo 3.1 and Seedance 2.5 generate audio natively, in the exact same pass as the video pixels: the ambient hiss of a cafe and the sharp clink of a cup arrive in lockstep with the visual motion, with flawless lip sync and perfectly timed sound effects.
Verdict: Watching
AI video generation is genuine and commercially functional — for short clips, social cutaways, storyboards, and background b-roll. But the market moves so fast that the tool that defined it is already gone, and a 10-second clip is not a movie yet. Tyler Perry was sensible to pause when faced with exponential technology. The underlying lesson is not that AI video eliminates production craft. It shifts the advantage to creators who master directing, prompting, and assembling the output. As the reviewers at RGBA put it: "The model is no longer the bottleneck. The workflow is." Stop stressing over which single video generator is objectively the best — they change every single month. Instead, build an agile workflow with brilliant hooks, strict image references, and smart prompt structures. The tools are just the stone. You are the sculptor. Get to work.
Paragon AI MAG — one video and one podcast episode every day. Subscribe to the Paragon AI MAG YouTube channel and follow the show on RedCircle so you never miss an episode. Everything we publish is backed by sources, and every fact is checked several times before release.



