What if a person could navigate a communication board and speak short messages using only eye movement and an ordinary webcam?

That question led me to build Eye Voice, an open-source, browser-based assistive communication prototype.

It combines MediaPipe face and iris landmarks, personal gaze calibration, dwell-based selection, speech recognition, and text-to-speech—without requiring dedicated eye-tracking hardware or a backend server.

Eye Voice in Action

Here's a quick demonstration of controlling the interface using eye movement.

Eye Voice is an early prototype, not a certified medical device, and should not be used as the sole communication method in safety-critical situations.

Why build it in the browser?

Commercial infrared eye-gaze devices can provide excellent accuracy, but they can also be expensive and difficult to access.

Eye Voice explores a more accessible tradeoff:

  • Use a standard RGB webcam
  • Run gaze processing locally in the browser
  • Avoid installation and build steps
  • Keep camera frames on the user's device
  • Provide keyboard and touch fallbacks
  • Make the project easy to inspect, fork, and improve

The goal is not to replace clinical hardware. It is to create an open prototype that can be tested, studied, and improved by the community.

How the interaction works

The core pipeline is:

Webcam → MediaPipe Face Landmarker → Iris position estimation → Personal calibration → Direction classification → Dwell selection → Speech output

1. Face and iris landmarks

MediaPipe Face Landmarker runs on the webcam feed and provides face and iris landmarks.

Eye Voice calculates each iris position relative to the eye corners, averages both eyes, and applies exponential smoothing to reduce frame-to-frame noise.

2. Nine-point calibration

Every face, camera angle, screen, and lighting setup is different.

The calibration flow collects gaze samples from nine points and fits an affine transformation that maps the raw gaze vector to screen coordinates.

The model is stored locally in the browser so the user does not need to recalibrate on every visit.

3. Coarse directional control

A normal webcam is less precise than a dedicated infrared tracker, so Eye Voice does not attempt pixel-perfect cursor control.

Instead, it classifies gaze into five coarse zones:

  • Up
  • Down
  • Left
  • Right
  • Center

This makes the interaction more tolerant of webcam noise and natural eye movement.

4. Dwell-to-select

Holding the gaze in one direction fills a visible dwell indicator.

When the configured dwell time completes, the related action is triggered:

  • Look up to start or stop listening
  • Look down to select and speak
  • Look left or right to move between quick phrases

The dwell time and sensitivity can be adjusted for different users.

Privacy choices

Camera processing happens locally in the browser. Video frames are not recorded or uploaded.

If the optional Gemini-powered suggestions are enabled, only the transcribed text is sent to the API. The camera feed and audio are not sent.

The project also works with local keyword-based suggestions when no Gemini key is provided.

Try it

Live demo: https://mostafamedhat19.github.io/EyeTracking/

Source code: https://github.com/MostafaMedhat19/EyeTracking

For the best result, use Chrome or Edge, allow camera access, keep your face clearly lit, and avoid moving the camera during calibration.

Open-source contributors are welcome

The repository includes an MIT license, contribution guide, public roadmap, issue templates, governance policy, and automated quality checks.

Some areas where help would be especially valuable:

  • Arabic localization and RTL support
  • Accessibility and browser testing
  • Configurable custom word banks
  • Automated tests for calibration and gaze estimation
  • Feedback from people familiar with AAC and assistive technology

Open issues: https://github.com/MostafaMedhat19/EyeTracking/issues

I would especially appreciate honest feedback about calibration reliability, accessibility, privacy, and the contribution experience.

What would you test or improve first? Please leave your questions, accessibility concerns, or implementation ideas in the comments. I would love to discuss them.

This article was written by Mostafa Medhat with AI assistance for editing and English phrasing. The technical details and project decisions were reviewed by the author.