Beyond Words: Building Synaphi, an AI Medical Assistant for Everyone

This is a submission for the MLH × DEV Writing Challenge, featuring Synaphi, built at EmoryHacks 2026.

What I Built

Imagine understanding every word your doctor says, but being unable to answer a single question.

"Where does it hurt?"

For someone recovering from a stroke, living with severe paralysis, or temporarily unable to speak, that simple question can become an enormous communication barrier.

And the problem isn't necessarily understanding. It's having a reliable way to respond.

At EmoryHacks 2026, we built Synaphi — an AI-assisted patient–clinician communication platform designed around a simple principle: everyone deserves to be heard.

Synaphi brings clinicians and patients into a synchronized communication experience. The clinician asks questions, and the patient responds through an accessible interface designed to support different physical and communication abilities.

The patient can use:

• Large visual and pictogram-based choices
• Keyboard and single-switch scanning
• Dwell-based selection
• Touch controls
• Browser-supported speech input
• Experimental webcam-based blink and head gestures

The platform also supports five languages and can help patients communicate pain location, intensity, and quality.

But we made one architectural decision that defines the entire project:

AI can help decide which choices to offer. It cannot decide what the patient means.

The patient selects the information. The patient confirms the final message. The clinician remains responsible for clinical decisions.

Why not just build another medical chatbot?

Because a chatbot that guesses what a person wants to say can introduce a serious problem: it might express something the person never intended.

For example, if someone indicates pain, AI should not invent additional symptoms or make statements about consent.

Our solution separates AI-powered interpretation from deterministic message construction.

The intelligence assists communication. It does not replace the communicator.

How Synaphi works

The system follows a six-stage interaction:

1. A clinician asks a question, such as "Where does it hurt?"
2. Google Gemini interprets the question into structured intent, response type, and search terms.
3. Typesense retrieves relevant choices from a curated communication library.
4. Synaphi displays large, accessible response cards.
5. The patient selects and reviews a response using their preferred controls.
6. Only after confirmation is the message spoken aloud and sent to the clinician.

Our frontend uses React and TypeScript. A Python FastAPI backend handles communication and AI orchestration, with WebSockets synchronizing clinician and patient views.

We designed the experience to support two devices or a split-screen demonstration.

The hardest accessibility problem: one button

A patient with very limited mobility may be able to activate only one switch.

That means ordinary buttons, dropdowns, and text entry become difficult to use.

We implemented two-level switch scanning.

Instead of cycling through every possible button individually, the interface highlights groups of actions first, then individual choices within the selected group.

The patient can navigate and respond using the Space key as a switch.

We tested a complete question-answering flow using only Space-key input.

Our emergency Help action remains available independently of Gemini and Typesense. Even when the AI services fail, the patient can still trigger the prototype's local help notification.

Experimental facial controls

Synaphi also implements facial-landmark detection using MediaPipe inside the browser.

For intentional-blink selection, the detection logic includes a calibrated threshold, a deliberate hold period, hysteresis, cooldown, and a requirement to reopen the eyes before another selection.

These safeguards are intended to reduce accidental activations.

The detection logic has unit tests, and the MediaPipe model has been loaded in automated browser testing. However, reliable blink/head operation on a real webcam still requires human validation. We are not presenting it as clinically tested.

What makes Synaphi different?

It isn't one individual feature.

It's the combination of accessible physical interaction, context-aware search, real-time communication, patient-controlled message composition, and data-driven personalization.

The long-term goal is to create technology that adapts to the ways people can communicate rather than expecting everyone to use the same interface.

Demo

GitHub: Synaphi — EmoryHacks 2026

Application: Synaphi demo

The linked application is a hackathon prototype. Availability and configured integrations may vary by deployment.

A sample interaction

Our fictional patient, Elena, cannot use conventional speech or hand-based input.

A clinician asks where she feels pain.

Synaphi interprets the question and presents accessible choices.

Elena selects a body region, a more precise location, an intensity, and a pain description.

The system constructs a statement entirely from her choices.

She reviews it and selects Confirm and Speak.

The clinician receives her confirmed response in the synchronized dashboard.

No AI model invents her symptoms or sends a response without her confirmation.

Measured engineering results

We instrumented development tests to measure the communication pipeline.

These are development-environment measurements, not clinical performance benchmarks.

The latency tests used 10–12 requests per measured path. We also implemented 16 backend tests, 12 frontend unit tests, and two Playwright end-to-end flows.

The system demonstrates technical feasibility, but these results do not establish improved clinical outcomes or usability for people with disabilities.

Our testing report documents the methods and remaining limitations.

Partner Technologies

We wanted every technology in Synaphi to solve a real engineering problem.

Google Gemini — Understanding questions without speaking for patients

Gemini interprets clinician questions into structured information that guides the interface.

We use structured outputs and schema validation so the backend can work with predictable fields rather than loosely formatted chatbot text.

Gemini also supports translation, multimodal medical-document extraction, and clinician-facing draft assistance.

For example, in development testing, Gemini extracted nine expected fields from a fictional prescription image.

The extracted information remains unverified until reviewed by a clinician. The system retains source excerpts and provenance rather than treating model output as confirmed medical history.

We encountered a major integration challenge: some requested Gemini models returned timeouts or were unavailable to our API key.

Instead of pretending that every AI capability worked, we implemented model fallbacks, timeouts, and visibly labelled rule-based fallback behavior.

One thing this taught us:

A dependable AI application must be designed for the moment when the AI is unavailable.

Typesense — Search that becomes a communication interface

Typesense is central to how Synaphi selects useful response options.

We created a collection containing 138 curated communication phrases across 12 categories.

The search implementation combines keyword and vector retrieval with typo tolerance, synonyms, filters, and application-side reranking.

For example, the phrases "my belly hurts," "stomach pain," and "abdominal discomfort" may need to retrieve overlapping communication options despite different wording.

Gemini supplies structured intent, while Typesense retrieves the relevant phrases.

Our scoring process also considers category matches, patient preferences, and safety priorities.

We built a Search Inspector to expose the actual query, filters, retrieved results, and ranking factors.

This makes search more than a convenience feature. It helps determine which accessible options appear when a patient needs them.

Tiger Data — Measuring interaction friction over time

An accessible interface shouldn't assume that the same settings work for everyone.

We use Tiger Data's PostgreSQL and TimescaleDB capabilities to store session records and time-stamped interaction events.

The system records information such as:

• Input method and selection events
• Selection duration and missed scanning cycles
• Corrections and unsuccessful attempts
• Scanning configuration and calibration version

A TimescaleDB hypertable stores the events, while a five-minute continuous aggregate supports the analytics dashboard.

This enables Synaphi to suggest accessibility adjustments based on recorded interaction behavior.

For example, repeated missed scan cycles may suggest that the scanning interval should be increased.

Crucially, the system does not silently change someone's controls. Suggested adjustments require explicit approval.

Entire — Understanding AI-assisted development

We used Claude Code during development and installed the Entire CLI to support coding-session tracking and future checkpoint review.

This mattered because the project evolved quickly across accessibility controls, backend services, and multiple integrations.

Entire was configured locally with additional attention to credential redaction. Some early development sessions predated hook activation, and publishing checkpoints to the remote service was not completed at the time of our documentation.

The larger lesson was about maintaining a reviewable engineering process while working with coding agents rather than treating generated code as automatically trustworthy.

Vultr — Preparing for cloud deployment

We prepared a containerized backend using Docker, Docker Compose, and Caddy for HTTPS.

The production serving configuration passed local end-to-end tests.

Vultr deployment was planned, but the cloud instance had not been provisioned at the time of our recorded testing. We don't claim a completed Vultr deployment without verifying it.

The infrastructure is designed to move the backend beyond the development machine while keeping application secrets server-side.

Hackathon Experience

EmoryHacks 2026 challenged us to think beyond what AI could generate and focus on what technology could make possible for someone else.

We wanted to build something that could have a meaningful purpose outside a hackathon.

That ambition also made scope control difficult.

We were dealing with real-time messaging, accessibility input methods, multimodal AI, hybrid search, persistent records, and cloud deployment—all within a short competition window.

The bug that reminded us why testing matters

One of our most interesting failures appeared when we switched from local SQLite to remote Tiger Data.

Our automated demo intermittently showed the patient as disconnected, even though the WebSocket connection was active.

The problem was an asynchronous race condition.

The clinician dashboard's initial data fetch could finish after the WebSocket presence event and overwrite newer state with stale information.

A small ordering bug created a major user-facing failure.

After correcting it, the affected end-to-end flow passed five consecutive runs.

That experience reinforced a lesson: real-time systems need to be tested as systems, not just as isolated components.

The decision I'm most proud of

We deliberately made patient confirmation independent of AI generation.

It would have been easier to allow an LLM to compose the patient's final statement.

But easier wasn't necessarily better.

Instead, we used AI to interpret the clinician's question and retrieve appropriate choices, then relied on deterministic composition for the patient's confirmed message.

This preserves a clearer boundary between what an AI infers and what a human intentionally communicates.

What we still need to prove

Synaphi is a working hackathon prototype, not a medical device or a clinically validated communication aid.

The next steps include real-webcam testing, screen-reader evaluation, translation review with native speakers, and usability studies involving people with relevant disabilities and assistive-communication professionals.

Most importantly, we need evidence that it helps intended users complete communication tasks more effectively than their existing options.

A prototype can prove that an interaction works technically. It cannot, by itself, prove that the interaction is appropriate for everyone who might need it.

What I learned

Building Synaphi changed how I think about accessibility engineering.

The challenge isn't merely adding AI, large buttons, or speech recognition.

It's understanding who controls the interaction, what happens when technology fails, and whether the system respects the person's intended meaning.

AI is most valuable here when it makes meaningful human communication easier—not when it replaces the person communicating.

That is the direction we want to keep exploring.

Our goal is not to build an AI that speaks for everyone.

It's to build technology that helps everyone have a voice of their own.

Explore the project: Synaphi on GitHub

If you work in accessible design, assistive technology, clinical communication, or real-time AI systems, I'd love to hear what you would improve and which limitations you think we should investigate first.