By Jay Wang / Curify. Adapted from our engineering account of experiments recorded through October 12, 2026. This describes historical research, not a new evaluation of today's production model.

A fluent English sentence can make an ASL translation demo look convincing. It can also hide the fact that the model did not understand the signing.

While building Curify's ASL video translator, we encountered several ways an apparently successful result could mislead us. The useful lesson was to make each prediction inspectable, and to keep the claim narrower than the evidence.

First check whether the answer is already in the input

An early demo appeared to translate a signed clip well. Inspecting it showed that the model was reading captions burned into the frames. We retracted that result as evidence of sign-language understanding.

Video input does not make an evaluation independent. Visible text can leak the answer. So can a prompt or a reference translation created without a documented annotation method.

We separated diagnostic clips from independent evaluation material. A diagnostic example is useful for finding a problem. It does not automatically belong in an accuracy report.

Inspect the input from the saved run

Our frame inspector aligns sampled frames, batch boundaries, reference cues and saved predictions on a timeline. That lets us ask a concrete question: did the recognizer receive enough of the visual event to support the English it produced?

The inspector must match the actual run. Resampling the same clip with a newer configuration would show different evidence. We preserve the sampling configuration alongside predictions; a historical 96-frame sheet is not interchangeable with later duration-dependent sampling.

A minimal inspection record should connect:

  • The input identifier and permission scope.
  • Model, prompt and preprocessing versions.
  • Sampled timestamps and batch boundaries.
  • The saved prediction and the provenance of its reference.
  • A review decision, with the reason and unresolved questions.

The first four items support reproducibility. The final item still needs appropriate ASL expertise.

More frames and better text metrics were not sufficient

We tried denser sampling, uniform sampling and overlapping batches with shared context. In our 12-clip diagnostic study, none of these frame-based variants recovered correct meaning. That is a finding about this setup and sample, not evidence that temporal modeling cannot help.

In one batching comparison, word error rate improved while character overlap fell and outputs became shorter. A single metric could have rewarded the system for saying less. Lowering generation temperature can likewise improve repeatability without improving interpretation.

Our controlled training work raised another distinction. Increasing training data at fixed initialization and compute did not produce a sustained BLEU gain on a frozen 147-clip evaluation. A separate representation experiment improved text-overlap metrics, but differences in pretraining and preprocessing limited the conclusions we could draw. Those numbers were not customer-video accuracy.

The full engineering account includes the experiment table, sample sizes and limitations. These are internally recorded findings, not an independently replicated benchmark.

The review question is whether meaning survived

We want expert feedback on an evaluation rubric that distinguishes:

  1. Omissions, additions and changes in who did what to whom.
  2. Fingerspelling, names and numbers supported by visible evidence versus guesses.
  3. Nonmanual cues and spatial references that affect interpretation.
  4. Legitimate signing variation, which should not automatically be treated as an error.
  5. Cases where the output should express uncertainty or ask for clearer input.

This is a proposed review framework, not a validated ASL scoring instrument. Agreement about rubric anchors should come before claims about accuracy. A research dataset result also needs a separate check on the intended input distribution before it can support a product claim.

What we would like to learn next

For an ASL educator, researcher or fluent reviewer: which of these dimensions would you change, and what would a bounded first review need to establish?

We are looking for a discussion of the evaluation method. We are not asking readers to endorse the tool, share student data, or send private signer videos. Any later sample review would need an agreed scope and permission-cleared material.

You can read the original methods and limitations, or contact Jay at team@curify-ai.com. The ASL video-to-English tool is a separate product surface: its current price and limitations apply, and a successful export does not establish translation correctness.

Research context: How2Sign and Uni-Sign. We credit the upstream work; our integration experiments are not a claim to have reproduced every upstream result.