Offer speech models for several languages, not just English

The macOS app asks the operating system which languages it can transcribe and
offers those. Windows cannot be asked the same question usefully — its own
recogniser covers English, French, German, Japanese, Mandarin and Spanish and
nothing else — so copying that architecture would mean fewer languages, not
more. The registry plays that role here instead.

Adds English, German, French, Spanish, Mandarin, and a single model covering
Arabic, Indonesian, Japanese, Russian, Thai and Vietnamese. The small English
model stays the default: it is a third the size of the rest and the matcher
absorbs the accuracy difference.

The panel now picks a language and can delete a downloaded model, which also
drops Follow mode — that mode needs a model, and without one it would start
and never move.

Turkish is deliberately absent rather than forgotten. Kroko publishes a Turkish
community streaming model, but only in their own packaging; nobody has
converted it to the ONNX layout sherpa-onnx loads. The README says so.
This commit is contained in:
CRTkafa 2026-08-26 14:53:24 +07:00
parent 6a7bc695ed
commit 0d98341699
3 changed files with 230 additions and 50 deletions

View file

@ -16,7 +16,8 @@
--- ---
> **Status: early development.** All three guidance modes work. Speech > **Status: early development.** All three guidance modes work. Speech
> recognition is English-only so far — see [Speech recognition](#speech-recognition). > recognition covers several languages but not Turkish yet — see
> [Languages](#languages).
## The original ## The original
@ -50,7 +51,7 @@ What carries over is the product and the algorithms, reimplemented in Rust.
| Cue syntax | `[stage directions]` shown but never spoken | same | | Cue syntax | `[stage directions]` shown but never spoken | same |
| Privacy | fully on-device | same, deliberately | | Privacy | fully on-device | same, deliberately |
| Overlay home | Dynamic Island under the notch | pill on the top edge — see below | | Overlay home | Dynamic Island under the notch | pill on the top edge — see below |
| Speech engine | Apple's recogniser, dozens of languages | sherpa-onnx, English so far | | Speech engine | Apple's recogniser, dozens of languages | sherpa-onnx, a handful of downloadable models |
| Extras | Sidecar, PowerPoint import, remote view, mirror output | out of scope — see below | | Extras | Sidecar, PowerPoint import, remote view, mirror output | out of scope — see below |
## Design decisions ## Design decisions
@ -87,13 +88,17 @@ src-tauri/ Windows shell
├── audio.rs Microphone capture and the speech worker ├── audio.rs Microphone capture and the speech worker
├── speech.rs Streaming recogniser over the sherpa-onnx C API ├── speech.rs Streaming recogniser over the sherpa-onnx C API
├── model.rs Model registry and first-run download ├── model.rs Model registry and first-run download
├── backdrop.rs Mica or blur behind the frameless window
├── settings.rs Persisted preferences
├── session.rs Live session state and the webview-facing DTOs ├── session.rs Live session state and the webview-facing DTOs
└── lib.rs Commands, events, tray icon └── lib.rs Commands, events, tray icon
src/ Svelte UI src/ Svelte UI
├── App.svelte Editor and controls ├── App.svelte Editor, settings panel and transport dock
├── Chrome.svelte Title bar and resize grips for the frameless window
├── LiquidStart.svelte The voice-reactive transport control
├── Overlay.svelte The prompter pill ├── Overlay.svelte The prompter pill
└── lib/ Typed command wrappers, microphone metering └── lib/ Typed command wrappers, spring integrator
``` ```
The engine deliberately knows nothing about Windows. That keeps it testable The engine deliberately knows nothing about Windows. That keeps it testable
@ -120,8 +125,8 @@ without a microphone and without launching the app.
## Speech recognition ## Speech recognition
Word Tracking runs a streaming Zipformer transducer through sherpa-onnx. The Word Tracking runs a streaming Zipformer transducer through sherpa-onnx. The
model is downloaded on first use (about 42 MB) into the app data directory and model is downloaded on first use into the app data directory and never ships in
never ships in the installer. the installer.
`sherpa-rs` only wraps sherpa-onnx's *offline* recogniser, which decodes a `sherpa-rs` only wraps sherpa-onnx's *offline* recogniser, which decodes a
finished buffer. A teleprompter cannot wait for the presenter to stop talking, finished buffer. A teleprompter cannot wait for the presenter to stop talking,
@ -132,12 +137,32 @@ PCM, and opening the same input device twice — once for WebAudio metering, onc
for the recogniser — fails on exclusive-mode hardware. One capture path feeds for the recogniser — fails on exclusive-mode hardware. One capture path feeds
both the level meter and the transcriber. both the level meter and the transcriber.
**Only English is available today.** The macOS app gets dozens of languages free ### Languages
from Apple's on-device recogniser; there is no equivalent on Windows worth
using, so each language needs its own model in the registry. Adding one is a The macOS app asks the operating system which languages it can transcribe and
few lines in [`model.rs`](src-tauri/src/model.rs) plus a published streaming offers those, which on a Mac is dozens — Apple ships them. Windows cannot be
model to point at — the limit is which languages sherpa-onnx publishes streaming asked the same question usefully: its own recogniser covers English, French,
models for, not the app. German, Japanese, Mandarin and Spanish, and nothing else. Copying the
architecture would mean *fewer* languages, not more.
So the model registry plays that role instead, and a language is available
exactly when somebody has published a streaming model for it:
| | Size |
|---|---|
| English (small) — fastest, the default | 42 MB |
| English, German, French | 71 MB each |
| Spanish | 156 MB |
| Chinese (Mandarin) | 77 MB |
| Arabic, Indonesian, Japanese, Russian, Thai, Vietnamese (one model) | 339 MB |
**Turkish is not available yet.** A model exists — Kroko publishes a Turkish
community streaming model — but only in their own packaging, and nobody has
converted it to the ONNX layout sherpa-onnx loads. That conversion is the whole
job; the app already handles everything after it.
Adding a language is a few lines in [`model.rs`](src-tauri/src/model.rs) and a
published model to point at.
## Building ## Building
@ -187,6 +212,8 @@ cargo test --workspace
- [x] System tray - [x] System tray
- [x] Classic and Voice-Activated modes end to end - [x] Classic and Voice-Activated modes end to end
- [x] On-device streaming speech recognition — Word Tracking works - [x] On-device streaming speech recognition — Word Tracking works
- [x] A model registry covering several languages
- [ ] Turkish, once a streaming model is converted for it
- [ ] Settings that persist across launches - [ ] Settings that persist across launches
- [ ] Font, size, colour and opacity - [ ] Font, size, colour and opacity
- [ ] Pause, resume, and mute from the overlay - [ ] Pause, resume, and mute from the overlay
@ -194,7 +221,6 @@ cargo test --workspace
- [ ] `.textream` files - [ ] `.textream` files
- [ ] Global shortcuts - [ ] Global shortcuts
- [ ] Update check - [ ] Update check
- [ ] More speech languages in the model registry
### Deliberately out of scope ### Deliberately out of scope

View file

@ -28,10 +28,17 @@ pub struct SpeechModel {
/// Models offered in the UI. /// Models offered in the UI.
/// ///
/// Only the quantised variants are listed. The float encoder for this model is /// The macOS app asks the operating system which languages it can transcribe
/// 85 MB against 41 MB for int8, and on a task where a fuzzy matcher absorbs /// and offers those. Windows has no equivalent worth using — its own recogniser
/// recognition error, the extra 44 MB buys nothing a presenter would notice. /// covers only English, French, German, Japanese, Mandarin and Spanish — so
pub const MODELS: &[SpeechModel] = &[SpeechModel { /// this registry plays that role, and a language is available exactly when
/// somebody has published a streaming model for it.
///
/// Quantised weights where the publisher provides them. A float encoder is
/// typically twice the size, and on a task where a fuzzy matcher absorbs
/// recognition error it buys nothing a presenter would notice.
pub const MODELS: &[SpeechModel] = &[
SpeechModel {
id: "en-20m", id: "en-20m",
label: "English (small)", label: "English (small)",
language: "en", language: "en",
@ -43,7 +50,74 @@ pub const MODELS: &[SpeechModel] = &[SpeechModel {
tokens: "tokens.txt", tokens: "tokens.txt",
}, },
download_bytes: 43_600_000, download_bytes: 43_600_000,
}]; },
SpeechModel {
id: "en",
label: "English",
language: "en",
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-en-kroko-2025-08-06",
files: KROKO_FILES,
download_bytes: 71_300_000,
},
SpeechModel {
id: "de",
label: "German",
language: "de",
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-de-kroko-2025-08-06",
files: KROKO_FILES,
download_bytes: 71_300_000,
},
SpeechModel {
id: "fr",
label: "French",
language: "fr",
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-fr-kroko-2025-08-06",
files: KROKO_FILES,
download_bytes: 71_300_000,
},
SpeechModel {
id: "es",
label: "Spanish",
language: "es",
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-es-kroko-2025-08-06",
files: KROKO_FILES,
download_bytes: 156_200_000,
},
SpeechModel {
id: "zh",
label: "Chinese (Mandarin)",
language: "zh",
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-multi-zh-hans-int8-2023-12-13",
files: ModelFiles {
encoder: "encoder-epoch-20-avg-1-chunk-16-left-128.int8.onnx",
decoder: "decoder-epoch-20-avg-1-chunk-16-left-128.onnx",
joiner: "joiner-epoch-20-avg-1-chunk-16-left-128.int8.onnx",
tokens: "tokens.txt",
},
download_bytes: 76_500_000,
},
SpeechModel {
id: "multi",
label: "Arabic, Indonesian, Japanese, Russian, Thai, Vietnamese",
language: "multi",
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-ar_en_id_ja_ru_th_vi_zh-2025-02-10",
files: ModelFiles {
encoder: "encoder-epoch-75-avg-11-chunk-16-left-128.int8.onnx",
decoder: "decoder-epoch-75-avg-11-chunk-16-left-128.onnx",
joiner: "joiner-epoch-75-avg-11-chunk-16-left-128.int8.onnx",
tokens: "tokens.txt",
},
download_bytes: 338_700_000,
},
];
/// The Kroko releases all use the same plain file names.
const KROKO_FILES: ModelFiles = ModelFiles {
encoder: "encoder.onnx",
decoder: "decoder.onnx",
joiner: "joiner.onnx",
tokens: "tokens.txt",
};
/// The default model, used when the user has expressed no preference. /// The default model, used when the user has expressed no preference.
pub fn default_model() -> &'static SpeechModel { pub fn default_model() -> &'static SpeechModel {
@ -220,4 +294,40 @@ mod tests {
fn every_status_is_reported() { fn every_status_is_reported() {
assert_eq!(statuses(Path::new("C:/nowhere")).len(), MODELS.len()); assert_eq!(statuses(Path::new("C:/nowhere")).len(), MODELS.len());
} }
#[test]
fn every_model_is_completely_described() {
for model in MODELS {
assert!(!model.label.is_empty(), "{} has no label", model.id);
assert!(!model.language.is_empty(), "{} has no language", model.id);
assert!(
model.repo.contains('/'),
"{} needs an owner/name repository",
model.id
);
assert!(
model.download_bytes > 1_000_000,
"{} reports an implausible size",
model.id
);
for name in [
model.files.encoder,
model.files.decoder,
model.files.joiner,
model.files.tokens,
] {
assert!(!name.is_empty(), "{} is missing a file name", model.id);
}
}
}
#[test]
fn models_land_in_separate_directories() {
let root = Path::new("C:/appdata/textream");
let mut directories: Vec<PathBuf> = MODELS.iter().map(|m| directory(root, m)).collect();
directories.sort();
let count = directories.len();
directories.dedup();
assert_eq!(directories.len(), count, "two models share a directory");
}
} }

View file

@ -187,6 +187,19 @@ Today we are shipping something I have wanted for a long time.
} }
} }
async function removeModel() {
if (!selectedModel || downloading || running) return;
try {
const updated = await api.removeSpeechModel(selectedModel.id);
models = models.map((m) => (m.id === updated.id ? updated : m));
// Follow mode needs a model; without one it would start and never move.
if (settings.mode === "wordTracking") await pushMode("classic");
say(`${updated.label} removed.`);
} catch (error) {
say(`Could not remove the model: ${error}`, "warn");
}
}
async function toggleRun() { async function toggleRun() {
if (running) await stop(); if (running) await stop();
else await start(); else await start();
@ -477,6 +490,23 @@ Today we are shipping something I have wanted for a long time.
<section> <section>
<h2>Speech</h2> <h2>Speech</h2>
{#if models.length > 0}
<label class="field">
<span class="label">Language</span>
<select
bind:value={settings.modelId}
disabled={downloading}
onchange={persist}
>
{#each models as model (model.id)}
<option value={model.id}>
{model.label}
{model.installed ? "· ready" : `· ${megabytes(model.downloadBytes)}`}
</option>
{/each}
</select>
</label>
{#if selectedModel} {#if selectedModel}
<div class="model"> <div class="model">
<div> <div>
@ -487,7 +517,16 @@ Today we are shipping something I have wanted for a long time.
: megabytes(selectedModel.downloadBytes)}</span : megabytes(selectedModel.downloadBytes)}</span
> >
</div> </div>
{#if !selectedModel.installed} {#if selectedModel.installed}
<button
class="action"
disabled={downloading || running}
title={running
? "Stop the prompter before removing a model"
: "Delete the downloaded files"}
onclick={removeModel}>Remove</button
>
{:else}
<button <button
class="action" class="action"
disabled={downloading} disabled={downloading}
@ -502,6 +541,7 @@ Today we are shipping something I have wanted for a long time.
<span style="width:{downloadPercent}%"></span> <span style="width:{downloadPercent}%"></span>
</div> </div>
{/if} {/if}
{/if}
{:else} {:else}
<p class="note">No speech models are registered.</p> <p class="note">No speech models are registered.</p>
{/if} {/if}
@ -509,6 +549,10 @@ Today we are shipping something I have wanted for a long time.
Recognition runs on this machine. Nothing is uploaded, and there is no Recognition runs on this machine. Nothing is uploaded, and there is no
account. account.
</p> </p>
<p class="note">
Turkish is not here yet — no streaming model has been published for it
in a format this engine can load.
</p>
</section> </section>
</div> </div>
</aside> </aside>