_| | | | |_

Wearable Hearing Assistance System Conveying Speaker and Paralinguistic Information

Abstract

Weave turns live speech into expressive captions for Deaf and hard-of-hearing viewers by automating Caption with Intention: colour for who is speaking, motion for when each word is spoken, a variable font for how it was said. A microphone array on a headset adds where the voice came from, delivered as a pulse beside the ear and as an arc on a dial. English or Korean, entirely offline. Built for the 15th Creative Design Fair of the SNU College of Engineering, June to August 2026; the whole project, with the motion running, is on the Project Page.

Motivation

Real-time captioning has reached phones and embedded devices, and it stops at transcribing what was said. In a room with several speakers it loses who is talking and from which side, and it flattens loudness, pitch and emphasis; higher recognition accuracy does not bring any of that back. Prior work splits three ways — expressive captions, speaker and direction guidance, haptic aids — and the most expressive of them, Caption with Intention, is a design system for pre-produced video edited by hand. Nothing yet satisfies real time and expressiveness at once, and direction and touch have been studied apart from expressive captions altogether.

Demo

The caption contract

Caption with Intention is a design system from the Chicago Hearing Society, validated over eleven months of community testing. It names three failures of ordinary captions and answers each with a channel. Captions do not say who is speaking, so each speaker gets a colour. They arrive as blocks disconnected from when the words were said, so each word turns to its speaker's colour at its onset. And they flatten how something was said, so a variable font carries the voice: louder is larger, 3–12% of screen height with 5% at normal speech, and lower voices are heavier, from the heaviest weight at 80 Hz to the lightest at 250 Hz.

Weave keeps that mapping and supplies what the design system leaves open. It gives no motion timing at all, so the reference film and the After Effects template became the clock, measured glyph by glyph. And which word the film enlarges is a transcriber's judgement that does not correlate with the audio, so Weave matches shape, timing and amplitude and never fits per-word size.

Not everything in the literature agrees. Caption Royale (CHI '24) found colour the best channel for emotional valence; Caption with Intention spends colour on the speaker. Both cannot have it. Weave keeps colour on the speaker, because in a live room with several people not knowing who is talking is the harder failure — Haptic Captioning's finding and Speech Compass's survey premise, arriving from two other directions. Intensity rides on size and weight instead, and no emotion is inferred or labelled anywhere. Everything the viewer sees is a measurement of the voice, not a claim about the speaker's state.

Pipeline

Four stages run independently on the same audio and are joined per word by timestamp. Recognition uses models that give per-word onsets: for English, Nemotron streaming with Parakeet verifying at each endpoint; for Korean, a 174M causal Zipformer. Diarization is NVIDIA's Streaming Sortformer, run in Core ML through a Swift helper, with a full-turn voice embedding — ERes2Net for English, CAM++ for Korean — confirming identity at each endpoint. Prosody is loudness and pitch measured inside each word's span. Adaptive input gain feeds only the recognizer's copy of the signal, so caption size reflects true loudness.

The join is CaptionSpec, a versioned schema that carries not just the values but the rules for drawing them, so captions and the haptic node read the same information and a different program reproduces the same picture. Loudness is normalized within each speaker's own range; pitch is absolute, as the design system specifies, which is what makes one voice visibly heavier than another. The analysis program serves the web page itself over server-sent events. There is no server, and the network is used once, to download the models and fonts.

Read-ahead

A live recognizer cannot produce text early: a word arrives a median 0.62 s after it was spoken. But the design system wants the line on screen before it is spoken, uncoloured, so the reader sees the words and then watches the colour arrive. Weave gets this without predicting anything by presenting captions from a clock 1.75 s behind the acoustic one, with a per-word floor of 420 ms. A word appears in read-ahead grey and turns to its speaker's colour when the playhead reaches its onset. One animation delay drives the turn, the pop and the voice phase, so the browser schedules everything; there is no reveal queue and no timer.

The load-bearing invariant: text may be revised only ahead of the playhead. Once the colour turn passes a word it is frozen, spelling included, because a late correction to a word already read would be the system rewriting what someone just read. Corrections from the endpoint verifier land in the read-ahead zone, where they are invisible.

Durable speaker attribution arrives much later than text — a median 4.48 s — so the turn is never gated on it. A word turns on time in whatever colour is known, or it does not turn on time at all.

Speaker, direction, touch

Short turns and similar voices cannot be attributed at once, so every word carries one of four states: unknown, provisional, stable, corrected. An undecided speaker is neutral, never a guess, because a wrong colour is a false claim a viewer cannot check, and only stable and corrected may signal a speaker change. Haptics fire only on confirmed changes and strong emphasis, never on every word; continuous vibration is what Tactile Emotions (CHI '25) measured as distracting.

Direction comes from the array and is evidence, not identity. Geometry is nearly error-free when speakers are apart and it is available immediately, where a voice model needs seconds. But two people on the same bearing are one speaker to geometry, and a person who moves is two. So a bearing can fill in an undecided word and contest a weak decision, never a settled one, and a bearing matching nobody is evidence of a new speaker rather than the nearest old one. Each word carries its bearing, drawn on the Voice Compass as an arc on a ring; the arc's width is the spread of the observations.

Motion

Five channels per word, each with one owner and one input, each returning to rest:

• Colour turn — the word's onset, wiped across the word letter by letter.
• Pop — a constant 1.15× on every word at its onset, the design system's synchronization cue.
• Crest — font size from loudness, 0.72× to 1.62×. It overshoots, settles to 0.70 of the peak, sustains, then releases, measured off the film's "louder"; a rise time and a fall time cannot express it.
• Weight — 340 to 900, from the speaker's running median pitch plus prominence. Weight is a property of the voice, not the word: a shout doubles F0, so a per-word mapping would render the angriest voice as the thinnest text.
• Hold — a lift of up to 0.525 em for a word isolated by silence on both sides. A word that swells does not leave the line.

Width rides the word's own pitch as texture, and a character wave runs under the colour turn. Everything is frozen at first sight: duration, axes, sweep and turn moment are computed once per word and survive remounts, because recomputing any of them under a running animation is the bug this project re-committed most. Non-speech sounds are bracketed in four categories: vocal, reaction, music, ambient.

Hardware

The wearable is a 3D-printed headset that wraps the back of the neck, with a ReSpeaker XVF3800 four-mic array — echo cancellation, beamforming and direction of arrival on the board — and a coin vibration motor on each arm beside the ear. A Raspberry Pi Zero 2 W has 512 MB and cannot caption, so it captures and actuates and the computer decides; the node analyses nothing, by rule. One TCP connection carries audio, direction and the optional cues. Mounting the motors and the array on the head aligns left and right, and the zero of the bearing, with the wearer's body.

Two motors, because the wearer only needs to know which way to turn, and the skin's spatial resolution for vibration is low enough that more motors would not add distinguishable directions. The ring is driven on the Pi by default: it polls the array about every 120 ms and pulses 100 ms in every 300 toward the bearing, going silent the moment the board stops reporting speech. The salience lane, speaker changes and emphasis from the caption contract, is opt-in, because it rides durable words and so waits on the endpoint verifier.

Korean

Korean runs on the same pipeline with its own recognizer and embedding model, and a variable Noto Sans KR carries the intonation, since Roboto Flex has no Hangul outlines. Measured against English, Korean renders equivalently on every channel except width, which the Korean font does not have. Caption Royale's limitations note that affective captions in Hangul behave similarly to Latin ones but not identically, with a smaller legibility drop, which is the closest thing to evidence that the Korean path needs its own preference pass rather than a translation of the English one.

Limits

Read-ahead costs 1.75 s of latency; the reference renderer is exact only because it knows the text in advance. Attribution at the edges, short turns and similar voices, stays ambiguous. A session is locked to one language, or to a bilingual profile that holds both models at once. The sound-effect and music caption rules, and the ring on the live lane, are not implemented.