Auto reframe in video editors is usually sold as AI subject tracking. For the most common case, a person moving in front of a background that stays still, a much simpler method finds the same framing: compare each frame with the one before and see where the picture changed. The reframe tool in the osFoundry desktop app works this way, in Rust and ffmpeg. Below is the same method in about 60 lines of Python, run on a 7.5-second 1920×1080 clip of three shots. It lands on the same crops as the app, to the pixel.

1. Shrink the video to 64×36 grey frames

There is no need to look at full-size frames to find where something moved. ffmpeg can hand over the video as raw bytes, four frames a second, scaled down to 64×36 and converted to one grey byte per pixel:

import math, re, subprocess, sys FFMPEG, SRC = sys.argv[1], sys.argv[2] W, H, FPS = 64, 36, 4 raw = subprocess.run( [FFMPEG, "-loglevel", "error", "-i", SRC, "-vf", f"fps={FPS},scale={W}:{H}", "-pix_fmt", "gray", "-f", "rawvideo", "-"], capture_output=True, check=True).stdout frames = [raw[i:i + W * H] for i in range(0, len(raw) - W * H + 1, W * H)]

Each frame is 2,304 bytes. The 7.5-second clip gives 30 frames and so 29 pairs to compare. Scaling down also averages away most of the grain, which matters for the next step.

2. Find the centre of what changed

For each pair of frames, take the difference of every cell, ignore small differences, and compute the centre of mass of the rest, weighted by how much each cell changed:

def centroid(prev, curr, floor=12): total = sx = sy = 0 for i, (a, b) in enumerate(zip(prev, curr)): d = abs(b - a) if d < floor: continue total += d sx += d * (i % W) sy += d * (i // W) if total < W * H * 0.5: return None return sx / total / W, sy / total / H samples = [centroid(a, b) for a, b in zip(frames, frames[1:])]

A 64×36 grey frame from the third shot (left) and its difference from the frame before (right): only the walking woman lights up

The two constants are what make this usable. The noise floor of 12 (out of 255) drops changes that come from the sensor and the compression. On this clip, with the floor at 12 an average of 381 of the 2,304 cells count as changed. With the floor at 1, 1,855 do, which is most of the frame, and the centre gets pulled toward the middle of the picture: the first sample lands at 0.44 of the width instead of 0.40.

The second check returns None when the total change is tiny. When nothing moved, the "centre of the change" is the centre of noise, and following it makes the frame drift for no reason. None means "keep the previous position".

3. Limit how fast the frame may move

A raw centroid jumps whenever a second person enters the shot. Capping the step at 5% of the frame per sample keeps it from whipping across:

def smooth(samples, step=0.05): cur = next((s for s in samples if s), (0.5, 0.5)) out = [] for s in samples: if s: cur = tuple(min(1, max(0, c + max(-step, min(step, t - c)))) for c, t in zip(cur, s)) out.append(cur) return out track = smooth(samples)

It starts from the first real observation, so a clip that opens on a still frame doesn't pan in from the centre.

4. Find the cuts with ffmpeg's scene score

ffmpeg's select filter gives every frame a scene score between 0 and 1, and showinfo prints the frames that pass. The timestamps go to stderr, not stdout:

log = subprocess.run( [FFMPEG, "-hide_banner", "-i", SRC, "-filter_complex", "select='gt(scene,0.35)',showinfo", "-f", "null", "-"], capture_output=True, text=True).stderr cuts = [round(float(t) * 1000) for t in re.findall(r"pts_time:([\d.]+)", log)]

On this clip that finds cuts at 1,767 ms and 4,733 ms. Printing the score of every frame (select='gte(scene,0)',metadata=print) shows how wide the margin is: the two cuts score 0.616 and 0.432, and no other frame of the 225 scores more than 0.068.

5. One crop per shot

Finally, average the track inside each shot and put a 9:16 window around that point. 1080 × 9/16 is 607.5; the width is rounded down to an even 606 because H.264 needs even dimensions.

duration = len(frames) * 1000 // FPS per = duration / len(track) bounds = [0] + [c for c in cuts if 0 < c < duration] + [duration] cw = int(1080 * 9 / 16) // 2 * 2 for start, end in zip(bounds, bounds[1:]): a = int(start // per) part = track[a:max(a + 1, math.ceil(end / per))] bx = sum(p[0] for p in part) / len(part) x = min(max(round(1920 * bx - cw / 2), 0), 1920 - cw) print(f"{start / 1000:.2f}-{end / 1000:.2f}s crop={cw}:1080:{x}:0") 
0.00-1.77s crop=606:1080:369:0 1.77-4.73s crop=606:1080:668:0 4.73-7.50s crop=606:1080:932:0

These are the same three crops the app produced for this clip. The whole script took 3.5 seconds on this Mac, most of it in the pure-Python loop.

The cuts matter more than they look. Averaged over the whole clip, the subject sits at x = 707, just 50 pixels from a plain center crop (657), because she walks on the left in one shot and on the right in another. Only per shot does the average mean anything.

Putting the shots back together

The crops become one filter_complex: trim each shot, crop it, and join the parts. This is the form the app builds:

[0:v]trim=start=0.000:end=1.767,setpts=PTS-STARTPTS,crop=606:1080:369:0[v0]; [0:v]trim=start=1.767:end=4.733,setpts=PTS-STARTPTS,crop=606:1080:668:0[v1]; [0:v]trim=start=4.733:end=7.500,setpts=PTS-STARTPTS,crop=606:1080:932:0[v2]; [v0][v1][v2]concat=n=3:v=1:a=0[out]

(Drop the line breaks when you pass it to -filter_complex.) The output is 606×1080 and the same length as the input.

Top row: a center crop of each shot. Bottom row: the crops found above

The app's Rust explains why it doesn't animate the crop inside a shot, even though ffmpeg could:

/// 🔴 Per scene, not per frame. ffmpeg can animate a crop, but a crop that moves continuously is /// a pan the editor never asked for; cutting between fixed framings is what an editor would /// actually do, and it is also what stays honest when the tracking is wrong — a bad framing /// lasts one scene rather than smearing across the whole clip. pub fn reframe_segments( cuts: &[i64], duration_ms: i64, track: &[(f64, f64)], width: u32, height: u32, target: f64, ) -> Vec<ReframeSegment> {

Where frame differencing fails

The method finds what moves, not who matters. It works when the subject is the main thing moving and the camera stays still. It goes wrong when:

  • the camera pans, so every cell changes and the centre lands near the middle;
  • the background is busy, such as traffic or a crowd;
  • two people move on opposite sides, and the average ends up between them.

Reflections are a milder case. In the first shot of this clip the woman's reflection in a window moves too, but it stays close to her, so the frame still finds her. A face or person detector handles the hard cases better, at the price of a model. The app's answer for those cases is a manual override: the user drags a point onto the subject, and the analysis is skipped.

The desktop app used here is free from the osFoundry download page.