If a small client job asks for talking-head clips with the background cut out, I'd want to know before quoting whether a free FFmpeg chroma key is enough or whether I need an AI matting model. So on 2026-11-03 I built a small test, scored both, and got a result that changed how I read my own numbers: the AI matte had a lower error than the chroma key on every one of 90 frames, and it also flickered more on 89 of the 89 frame-to-frame steps.
The test I could afford
I don't have hand-pulled ground-truth mattes and won't pay anyone to make them, so I went cheap. I took a public-domain green-screen clip from Wikimedia Commons, kept the three seconds where the presenter waves (frames 90 to 179), and used a model's transparent export of those frames as the "known" alpha. Then I composited that foreground onto two generated backgrounds at 540×960: a flat green with a little per-frame sensor-style noise, and a busy pattern of shapes that slides left by one pixel per frame. Since I made the composites, I know the alpha every method should return.
That shortcut has a cost I can't measure away. The ground truth is a model's output, not a careful manual matte, so "correct" here only means "what that model said". It came from RVM ResNet50, and the AI method I scored is RVM MobileNetV3 from the same family, so the AI side may look better than it would against a hand-made matte. For the AI side I ran ImgIng https://imging.ai/ on its compatible tier (still Beta, run in a desktop browser). The chroma key was FFmpeg's chromakey with similarity 0.10 and blend 0.
Two scores that disagree
For per-frame error I used SAD inside an edge band, the trimap unknown region, which averaged about 15.8% of the frame. On the green background the AI matte averaged 2.89 there and the chroma key 3.88, and the AI matte was lower on all 90 frames. Gradient error, which tracks edge softness, was 0.53 against 1.04, since the chroma key at blend 0 outputs pure 0 or 1.
For flicker I took the pixels whose ground-truth alpha barely changed between two frames (under 0.5/255, about 93.3% of pixels) and measured how much the predicted alpha moved there:
still = np.abs(gt_cur - gt_last) < 0.5 / 255 jitter = np.abs(pred_cur - pred_last)[still].mean() * 1000 diff = (pred_cur - pred_last) - (gt_cur - gt_last) dtssd = np.sqrt((diff ** 2).sum() / diff.size) * 100 Restricted to the edge band, the AI matte scored 2.205 and the chroma key 0.198, roughly 11 times lower, and the chroma key was steadier on 89 of 89 frame pairs. My guess: a hard 0/1 key with small noise rarely tips over its threshold, while soft model output drifts a little every frame. A control made of the ground truth plus random per-frame noise (σ=0.06 in the edge band) had the best per-frame SAD in the table, 1.53, and one of the worst flicker scores, 11.844. The usual dtSSD still ranked that noisy control best at 1.93, since it averages over the whole frame and mixes missed motion with jitter, so I keep the static-pixel number separate.
Sample: VOA Learning English, Wikimedia Commons, public domain; trimmed, composited, modified. What to look at: in the top chart the orange constructed-noise line sits lowest and the red AI green-background line stays under the blue chroma key line the whole way; in the bottom chart the order flips, the blue line hugs zero and the orange and green lines bounce between roughly 8 and 23.
What changed in how I pick
The busy background moved the numbers more than the choice of method did. The same AI matte went from SAD 2.89 to 7.44 and from flicker 2.205 to 11.984 when only the background changed, and its per-frame SAD swung from 4.90 to 10.41, peaking around frames 60 to 70. A spot check on one early frame and one between frames 60 and 70 would tell two different stories. So for client work the plan is to test on the background the client actually shoots against, not on a clean green I generated.
The other change is that I don't report one averaged score anymore. I keep two columns, per-frame error and static-pixel flicker, and next to each I write how many frames or frame pairs one method won, like 90/90 and 89/89. A mean of 2.89 versus 3.88 sounds mild; "90 of 90" says it's consistent, and "lost 89 of 89 on flicker" says there's a second question open. These numbers only hold for this clip and my code, not against paper tables.
What I still don't know is whether a flicker of 2.205 (an average change of about 0.002 in alpha per frame) is something a client would notice on a phone, or only something my script notices. Until I check, my working assumption is AI matting where there's soft hair and the chroma key for flat, even green where a hard edge is fine.
Source clip: "Filming with the green screen" by VOA Learning English, Wikimedia Commons, public domain, trimmed and composited by me. The two backgrounds are images I generated.
