At UniHack 2026 our team built respirAI: a low-cost digital stethoscope that guides you through recording your own lung sounds, screens them with a neural network on your phone, and tells you whether they sound normal or are worth showing to a doctor. It is a screening aid built by students, not a medical device.

This post is how we built it, step by step, and the bug that cost us the most sleep.

The problem

A stethoscope is only useful when a trained ear is on the other end. We wanted the opposite: a device that anyone can hold on their own chest, with the phone doing the guiding and the first round of listening.

Step 1: the device

No custom board. A regular stethoscope chest piece sits on an INMP441 MEMS microphone, wired to a Seeed XIAO ESP32-S3 with a small LiPo battery. The firmware captures audio at 16 kHz, downsamples it to 8 kHz and streams it over Bluetooth Low Energy in 10 ms packets: a 2-byte sequence number followed by 160 bytes of 16-bit mono PCM.

The sequence number matters more than it looks. BLE drops packets, and a lung recording with silent holes in it is worse than no recording, so the app counts what is missing and refuses a recording that has too many gaps.

Step 2: the app

The app is plain JavaScript, HTML and CSS with no framework. In a browser it talks to the device with Web Bluetooth; the same code is wrapped with Capacitor into an Android app.

A check goes through the chest positions one at a time. For each one, a body diagram shows where the chest piece goes, then a breathing circle takes over the whole screen and paces two slow breaths while the device records for 12 seconds. The app runs in five languages.

Two decisions we would make again:

  • A practice device. A fake Bluetooth device that plays stored recordings let us build and test the whole flow without hardware in hand, and it is why the browser tests can run on every push.
  • Fixed interfaces first. We wrote down the BLE packet format and the app-to-AI contract on day one, so firmware, app and AI were built in parallel on separate branches and merged at the end.

Step 3: the AI

The model is a convolutional network trained on the ICBHI 2017 respiratory sound database (920 recordings from 126 participants). It looks at a log-mel spectrogram of each recording and flags crackles and wheezes.

Two rules we held to. We trained on sound labels, not disease labels, because the disease labels are dominated by one condition and a model can score well by always guessing it. And we split by patient, never randomly, because a random split puts the same person in training and test and inflates every number.

Step 4: the voice guide

Here is the thing we missed at first: the person's hands are busy. They are holding a chest piece on their own back. They cannot tap "start".

So we added a voice guide on top of the flow, using an ElevenLabs conversational agent. It says where to put the chest piece, hears "ready", starts the recording through a client tool, answers questions like "where exactly?", and reads the result. The timer, the breathing circle and every button stay in our code; the agent only speaks and presses two buttons. Offline, the same switch plays pre-recorded instructions instead.

One rule is absolute: while the stethoscope records, nobody talks. The phone microphone is muted and the agent's volume is zero, otherwise the stethoscope records the voices too.

The hardest bug: the agent kept getting cut off

The flow looked simple:

  1. The person says "ready".
  2. The agent calls our start_recording tool.
  3. The tool answers: "tell the person to breathe slowly and stay silent".
  4. We wait until the agent has finished saying that.
  5. We mute everything and record.

Step 4 was the bug. The agent would start "Now breathe slowly, in and out with the ci" and go silent. Every time.

Our first version of "wait until it finished" was the obvious one: wait for the agent to start speaking, then wait for it to stop.

// first attempt while (mode !== "speaking") await sleep(100); while (mode === "speaking") await sleep(100); // ...mute and record

It took us far too long to see why this fails. A real agent does not just call the tool. It says "Great!" first, calls the tool while that word is still playing, pauses to think about the tool's answer, and only then says the real sentence. Our code saw "Great!" start and stop, decided the agent was done, and muted it exactly as the real sentence began.

The fix has two parts. First, only speech that begins after the tool was called counts as the answer. Second, "finished" means a full second with nothing audible, because the pause between two sentences is shorter than that.

// since = the moment the tool was called async spoken(since) { // 1. the answer has begun: speech that started after `since` while (!(spokeAt >= since || (saidAt >= since && audible()))) await sleep(80); // 2. and it is over: nothing audible for a full second for (let quiet = 0; ; await sleep(80)) { if (audible()) quiet = 0; else if (!quiet) quiet = performance.now(); else if (performance.now() - quiet >= 1000) break; } }

(The real version also has time limits, so an agent that says nothing cannot hold the check up.)

The bug had a twin. In a loud hackathon hall, the agent kept stopping mid-sentence for a different reason: the microphone heard the room, or the agent's own voice from the speaker, and treated it as the person interrupting. We made the conversation half-duplex: while the agent speaks, the microphone is off.

conversation.setMicMuted(recording || userMutedIt || mode === "speaking");

You lose the ability to talk over the agent. In exchange it finishes its sentences, which for a device that tells people how to breathe is the right trade.

What we learned

  • Test with a stand-in that misbehaves like the real thing. Our fake agent answered instantly and politely, so every test passed while the real one was being cut off. The test that finally caught it has the fake say a word before its tool call.
  • A voice agent is a layer, not the app. Every step still works by screen and button. When the agent cannot be reached, the app says so once and carries on with recorded instructions.
  • Agree on interfaces before writing code. It is the only reason four people could merge hardware, firmware, app and AI on the last morning.

If you want to try it, the practice mode in the web app needs no hardware: https://denis-furtuna.github.io/respirAI/