Offer speech models for several languages, not just English
The macOS app asks the operating system which languages it can transcribe and offers those. Windows cannot be asked the same question usefully — its own recogniser covers English, French, German, Japanese, Mandarin and Spanish and nothing else — so copying that architecture would mean fewer languages, not more. The registry plays that role here instead. Adds English, German, French, Spanish, Mandarin, and a single model covering Arabic, Indonesian, Japanese, Russian, Thai and Vietnamese. The small English model stays the default: it is a third the size of the rest and the matcher absorbs the accuracy difference. The panel now picks a language and can delete a downloaded model, which also drops Follow mode — that mode needs a model, and without one it would start and never move. Turkish is deliberately absent rather than forgotten. Kroko publishes a Turkish community streaming model, but only in their own packaging; nobody has converted it to the ONNX layout sherpa-onnx loads. The README says so.
This commit is contained in:
parent
6a7bc695ed
commit
0d98341699
3 changed files with 230 additions and 50 deletions
52
README.md
52
README.md
|
|
@ -16,7 +16,8 @@
|
||||||
---
|
---
|
||||||
|
|
||||||
> **Status: early development.** All three guidance modes work. Speech
|
> **Status: early development.** All three guidance modes work. Speech
|
||||||
> recognition is English-only so far — see [Speech recognition](#speech-recognition).
|
> recognition covers several languages but not Turkish yet — see
|
||||||
|
> [Languages](#languages).
|
||||||
|
|
||||||
## The original
|
## The original
|
||||||
|
|
||||||
|
|
@ -50,7 +51,7 @@ What carries over is the product and the algorithms, reimplemented in Rust.
|
||||||
| Cue syntax | `[stage directions]` shown but never spoken | same |
|
| Cue syntax | `[stage directions]` shown but never spoken | same |
|
||||||
| Privacy | fully on-device | same, deliberately |
|
| Privacy | fully on-device | same, deliberately |
|
||||||
| Overlay home | Dynamic Island under the notch | pill on the top edge — see below |
|
| Overlay home | Dynamic Island under the notch | pill on the top edge — see below |
|
||||||
| Speech engine | Apple's recogniser, dozens of languages | sherpa-onnx, English so far |
|
| Speech engine | Apple's recogniser, dozens of languages | sherpa-onnx, a handful of downloadable models |
|
||||||
| Extras | Sidecar, PowerPoint import, remote view, mirror output | out of scope — see below |
|
| Extras | Sidecar, PowerPoint import, remote view, mirror output | out of scope — see below |
|
||||||
|
|
||||||
## Design decisions
|
## Design decisions
|
||||||
|
|
@ -87,13 +88,17 @@ src-tauri/ Windows shell
|
||||||
├── audio.rs Microphone capture and the speech worker
|
├── audio.rs Microphone capture and the speech worker
|
||||||
├── speech.rs Streaming recogniser over the sherpa-onnx C API
|
├── speech.rs Streaming recogniser over the sherpa-onnx C API
|
||||||
├── model.rs Model registry and first-run download
|
├── model.rs Model registry and first-run download
|
||||||
|
├── backdrop.rs Mica or blur behind the frameless window
|
||||||
|
├── settings.rs Persisted preferences
|
||||||
├── session.rs Live session state and the webview-facing DTOs
|
├── session.rs Live session state and the webview-facing DTOs
|
||||||
└── lib.rs Commands, events, tray icon
|
└── lib.rs Commands, events, tray icon
|
||||||
|
|
||||||
src/ Svelte UI
|
src/ Svelte UI
|
||||||
├── App.svelte Editor and controls
|
├── App.svelte Editor, settings panel and transport dock
|
||||||
|
├── Chrome.svelte Title bar and resize grips for the frameless window
|
||||||
|
├── LiquidStart.svelte The voice-reactive transport control
|
||||||
├── Overlay.svelte The prompter pill
|
├── Overlay.svelte The prompter pill
|
||||||
└── lib/ Typed command wrappers, microphone metering
|
└── lib/ Typed command wrappers, spring integrator
|
||||||
```
|
```
|
||||||
|
|
||||||
The engine deliberately knows nothing about Windows. That keeps it testable
|
The engine deliberately knows nothing about Windows. That keeps it testable
|
||||||
|
|
@ -120,8 +125,8 @@ without a microphone and without launching the app.
|
||||||
## Speech recognition
|
## Speech recognition
|
||||||
|
|
||||||
Word Tracking runs a streaming Zipformer transducer through sherpa-onnx. The
|
Word Tracking runs a streaming Zipformer transducer through sherpa-onnx. The
|
||||||
model is downloaded on first use (about 42 MB) into the app data directory and
|
model is downloaded on first use into the app data directory and never ships in
|
||||||
never ships in the installer.
|
the installer.
|
||||||
|
|
||||||
`sherpa-rs` only wraps sherpa-onnx's *offline* recogniser, which decodes a
|
`sherpa-rs` only wraps sherpa-onnx's *offline* recogniser, which decodes a
|
||||||
finished buffer. A teleprompter cannot wait for the presenter to stop talking,
|
finished buffer. A teleprompter cannot wait for the presenter to stop talking,
|
||||||
|
|
@ -132,12 +137,32 @@ PCM, and opening the same input device twice — once for WebAudio metering, onc
|
||||||
for the recogniser — fails on exclusive-mode hardware. One capture path feeds
|
for the recogniser — fails on exclusive-mode hardware. One capture path feeds
|
||||||
both the level meter and the transcriber.
|
both the level meter and the transcriber.
|
||||||
|
|
||||||
**Only English is available today.** The macOS app gets dozens of languages free
|
### Languages
|
||||||
from Apple's on-device recogniser; there is no equivalent on Windows worth
|
|
||||||
using, so each language needs its own model in the registry. Adding one is a
|
The macOS app asks the operating system which languages it can transcribe and
|
||||||
few lines in [`model.rs`](src-tauri/src/model.rs) plus a published streaming
|
offers those, which on a Mac is dozens — Apple ships them. Windows cannot be
|
||||||
model to point at — the limit is which languages sherpa-onnx publishes streaming
|
asked the same question usefully: its own recogniser covers English, French,
|
||||||
models for, not the app.
|
German, Japanese, Mandarin and Spanish, and nothing else. Copying the
|
||||||
|
architecture would mean *fewer* languages, not more.
|
||||||
|
|
||||||
|
So the model registry plays that role instead, and a language is available
|
||||||
|
exactly when somebody has published a streaming model for it:
|
||||||
|
|
||||||
|
| | Size |
|
||||||
|
|---|---|
|
||||||
|
| English (small) — fastest, the default | 42 MB |
|
||||||
|
| English, German, French | 71 MB each |
|
||||||
|
| Spanish | 156 MB |
|
||||||
|
| Chinese (Mandarin) | 77 MB |
|
||||||
|
| Arabic, Indonesian, Japanese, Russian, Thai, Vietnamese (one model) | 339 MB |
|
||||||
|
|
||||||
|
**Turkish is not available yet.** A model exists — Kroko publishes a Turkish
|
||||||
|
community streaming model — but only in their own packaging, and nobody has
|
||||||
|
converted it to the ONNX layout sherpa-onnx loads. That conversion is the whole
|
||||||
|
job; the app already handles everything after it.
|
||||||
|
|
||||||
|
Adding a language is a few lines in [`model.rs`](src-tauri/src/model.rs) and a
|
||||||
|
published model to point at.
|
||||||
|
|
||||||
## Building
|
## Building
|
||||||
|
|
||||||
|
|
@ -187,6 +212,8 @@ cargo test --workspace
|
||||||
- [x] System tray
|
- [x] System tray
|
||||||
- [x] Classic and Voice-Activated modes end to end
|
- [x] Classic and Voice-Activated modes end to end
|
||||||
- [x] On-device streaming speech recognition — Word Tracking works
|
- [x] On-device streaming speech recognition — Word Tracking works
|
||||||
|
- [x] A model registry covering several languages
|
||||||
|
- [ ] Turkish, once a streaming model is converted for it
|
||||||
- [ ] Settings that persist across launches
|
- [ ] Settings that persist across launches
|
||||||
- [ ] Font, size, colour and opacity
|
- [ ] Font, size, colour and opacity
|
||||||
- [ ] Pause, resume, and mute from the overlay
|
- [ ] Pause, resume, and mute from the overlay
|
||||||
|
|
@ -194,7 +221,6 @@ cargo test --workspace
|
||||||
- [ ] `.textream` files
|
- [ ] `.textream` files
|
||||||
- [ ] Global shortcuts
|
- [ ] Global shortcuts
|
||||||
- [ ] Update check
|
- [ ] Update check
|
||||||
- [ ] More speech languages in the model registry
|
|
||||||
|
|
||||||
### Deliberately out of scope
|
### Deliberately out of scope
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -28,22 +28,96 @@ pub struct SpeechModel {
|
||||||
|
|
||||||
/// Models offered in the UI.
|
/// Models offered in the UI.
|
||||||
///
|
///
|
||||||
/// Only the quantised variants are listed. The float encoder for this model is
|
/// The macOS app asks the operating system which languages it can transcribe
|
||||||
/// 85 MB against 41 MB for int8, and on a task where a fuzzy matcher absorbs
|
/// and offers those. Windows has no equivalent worth using — its own recogniser
|
||||||
/// recognition error, the extra 44 MB buys nothing a presenter would notice.
|
/// covers only English, French, German, Japanese, Mandarin and Spanish — so
|
||||||
pub const MODELS: &[SpeechModel] = &[SpeechModel {
|
/// this registry plays that role, and a language is available exactly when
|
||||||
id: "en-20m",
|
/// somebody has published a streaming model for it.
|
||||||
label: "English (small)",
|
///
|
||||||
language: "en",
|
/// Quantised weights where the publisher provides them. A float encoder is
|
||||||
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17",
|
/// typically twice the size, and on a task where a fuzzy matcher absorbs
|
||||||
files: ModelFiles {
|
/// recognition error it buys nothing a presenter would notice.
|
||||||
encoder: "encoder-epoch-99-avg-1.int8.onnx",
|
pub const MODELS: &[SpeechModel] = &[
|
||||||
decoder: "decoder-epoch-99-avg-1.int8.onnx",
|
SpeechModel {
|
||||||
joiner: "joiner-epoch-99-avg-1.int8.onnx",
|
id: "en-20m",
|
||||||
tokens: "tokens.txt",
|
label: "English (small)",
|
||||||
|
language: "en",
|
||||||
|
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17",
|
||||||
|
files: ModelFiles {
|
||||||
|
encoder: "encoder-epoch-99-avg-1.int8.onnx",
|
||||||
|
decoder: "decoder-epoch-99-avg-1.int8.onnx",
|
||||||
|
joiner: "joiner-epoch-99-avg-1.int8.onnx",
|
||||||
|
tokens: "tokens.txt",
|
||||||
|
},
|
||||||
|
download_bytes: 43_600_000,
|
||||||
},
|
},
|
||||||
download_bytes: 43_600_000,
|
SpeechModel {
|
||||||
}];
|
id: "en",
|
||||||
|
label: "English",
|
||||||
|
language: "en",
|
||||||
|
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-en-kroko-2025-08-06",
|
||||||
|
files: KROKO_FILES,
|
||||||
|
download_bytes: 71_300_000,
|
||||||
|
},
|
||||||
|
SpeechModel {
|
||||||
|
id: "de",
|
||||||
|
label: "German",
|
||||||
|
language: "de",
|
||||||
|
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-de-kroko-2025-08-06",
|
||||||
|
files: KROKO_FILES,
|
||||||
|
download_bytes: 71_300_000,
|
||||||
|
},
|
||||||
|
SpeechModel {
|
||||||
|
id: "fr",
|
||||||
|
label: "French",
|
||||||
|
language: "fr",
|
||||||
|
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-fr-kroko-2025-08-06",
|
||||||
|
files: KROKO_FILES,
|
||||||
|
download_bytes: 71_300_000,
|
||||||
|
},
|
||||||
|
SpeechModel {
|
||||||
|
id: "es",
|
||||||
|
label: "Spanish",
|
||||||
|
language: "es",
|
||||||
|
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-es-kroko-2025-08-06",
|
||||||
|
files: KROKO_FILES,
|
||||||
|
download_bytes: 156_200_000,
|
||||||
|
},
|
||||||
|
SpeechModel {
|
||||||
|
id: "zh",
|
||||||
|
label: "Chinese (Mandarin)",
|
||||||
|
language: "zh",
|
||||||
|
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-multi-zh-hans-int8-2023-12-13",
|
||||||
|
files: ModelFiles {
|
||||||
|
encoder: "encoder-epoch-20-avg-1-chunk-16-left-128.int8.onnx",
|
||||||
|
decoder: "decoder-epoch-20-avg-1-chunk-16-left-128.onnx",
|
||||||
|
joiner: "joiner-epoch-20-avg-1-chunk-16-left-128.int8.onnx",
|
||||||
|
tokens: "tokens.txt",
|
||||||
|
},
|
||||||
|
download_bytes: 76_500_000,
|
||||||
|
},
|
||||||
|
SpeechModel {
|
||||||
|
id: "multi",
|
||||||
|
label: "Arabic, Indonesian, Japanese, Russian, Thai, Vietnamese",
|
||||||
|
language: "multi",
|
||||||
|
repo: "csukuangfj/sherpa-onnx-streaming-zipformer-ar_en_id_ja_ru_th_vi_zh-2025-02-10",
|
||||||
|
files: ModelFiles {
|
||||||
|
encoder: "encoder-epoch-75-avg-11-chunk-16-left-128.int8.onnx",
|
||||||
|
decoder: "decoder-epoch-75-avg-11-chunk-16-left-128.onnx",
|
||||||
|
joiner: "joiner-epoch-75-avg-11-chunk-16-left-128.int8.onnx",
|
||||||
|
tokens: "tokens.txt",
|
||||||
|
},
|
||||||
|
download_bytes: 338_700_000,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
/// The Kroko releases all use the same plain file names.
|
||||||
|
const KROKO_FILES: ModelFiles = ModelFiles {
|
||||||
|
encoder: "encoder.onnx",
|
||||||
|
decoder: "decoder.onnx",
|
||||||
|
joiner: "joiner.onnx",
|
||||||
|
tokens: "tokens.txt",
|
||||||
|
};
|
||||||
|
|
||||||
/// The default model, used when the user has expressed no preference.
|
/// The default model, used when the user has expressed no preference.
|
||||||
pub fn default_model() -> &'static SpeechModel {
|
pub fn default_model() -> &'static SpeechModel {
|
||||||
|
|
@ -220,4 +294,40 @@ mod tests {
|
||||||
fn every_status_is_reported() {
|
fn every_status_is_reported() {
|
||||||
assert_eq!(statuses(Path::new("C:/nowhere")).len(), MODELS.len());
|
assert_eq!(statuses(Path::new("C:/nowhere")).len(), MODELS.len());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn every_model_is_completely_described() {
|
||||||
|
for model in MODELS {
|
||||||
|
assert!(!model.label.is_empty(), "{} has no label", model.id);
|
||||||
|
assert!(!model.language.is_empty(), "{} has no language", model.id);
|
||||||
|
assert!(
|
||||||
|
model.repo.contains('/'),
|
||||||
|
"{} needs an owner/name repository",
|
||||||
|
model.id
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
model.download_bytes > 1_000_000,
|
||||||
|
"{} reports an implausible size",
|
||||||
|
model.id
|
||||||
|
);
|
||||||
|
for name in [
|
||||||
|
model.files.encoder,
|
||||||
|
model.files.decoder,
|
||||||
|
model.files.joiner,
|
||||||
|
model.files.tokens,
|
||||||
|
] {
|
||||||
|
assert!(!name.is_empty(), "{} is missing a file name", model.id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn models_land_in_separate_directories() {
|
||||||
|
let root = Path::new("C:/appdata/textream");
|
||||||
|
let mut directories: Vec<PathBuf> = MODELS.iter().map(|m| directory(root, m)).collect();
|
||||||
|
directories.sort();
|
||||||
|
let count = directories.len();
|
||||||
|
directories.dedup();
|
||||||
|
assert_eq!(directories.len(), count, "two models share a directory");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -187,6 +187,19 @@ Today we are shipping something I have wanted for a long time.
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function removeModel() {
|
||||||
|
if (!selectedModel || downloading || running) return;
|
||||||
|
try {
|
||||||
|
const updated = await api.removeSpeechModel(selectedModel.id);
|
||||||
|
models = models.map((m) => (m.id === updated.id ? updated : m));
|
||||||
|
// Follow mode needs a model; without one it would start and never move.
|
||||||
|
if (settings.mode === "wordTracking") await pushMode("classic");
|
||||||
|
say(`${updated.label} removed.`);
|
||||||
|
} catch (error) {
|
||||||
|
say(`Could not remove the model: ${error}`, "warn");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function toggleRun() {
|
async function toggleRun() {
|
||||||
if (running) await stop();
|
if (running) await stop();
|
||||||
else await start();
|
else await start();
|
||||||
|
|
@ -477,30 +490,57 @@ Today we are shipping something I have wanted for a long time.
|
||||||
|
|
||||||
<section>
|
<section>
|
||||||
<h2>Speech</h2>
|
<h2>Speech</h2>
|
||||||
{#if selectedModel}
|
{#if models.length > 0}
|
||||||
<div class="model">
|
<label class="field">
|
||||||
<div>
|
<span class="label">Language</span>
|
||||||
<strong>{selectedModel.label}</strong>
|
<select
|
||||||
<span class="tag"
|
bind:value={settings.modelId}
|
||||||
>{selectedModel.installed
|
disabled={downloading}
|
||||||
? "installed"
|
onchange={persist}
|
||||||
: megabytes(selectedModel.downloadBytes)}</span
|
>
|
||||||
>
|
{#each models as model (model.id)}
|
||||||
|
<option value={model.id}>
|
||||||
|
{model.label}
|
||||||
|
{model.installed ? "· ready" : `· ${megabytes(model.downloadBytes)}`}
|
||||||
|
</option>
|
||||||
|
{/each}
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
{#if selectedModel}
|
||||||
|
<div class="model">
|
||||||
|
<div>
|
||||||
|
<strong>{selectedModel.label}</strong>
|
||||||
|
<span class="tag"
|
||||||
|
>{selectedModel.installed
|
||||||
|
? "installed"
|
||||||
|
: megabytes(selectedModel.downloadBytes)}</span
|
||||||
|
>
|
||||||
|
</div>
|
||||||
|
{#if selectedModel.installed}
|
||||||
|
<button
|
||||||
|
class="action"
|
||||||
|
disabled={downloading || running}
|
||||||
|
title={running
|
||||||
|
? "Stop the prompter before removing a model"
|
||||||
|
: "Delete the downloaded files"}
|
||||||
|
onclick={removeModel}>Remove</button
|
||||||
|
>
|
||||||
|
{:else}
|
||||||
|
<button
|
||||||
|
class="action"
|
||||||
|
disabled={downloading}
|
||||||
|
onclick={downloadModel}
|
||||||
|
>
|
||||||
|
{downloading ? `${downloadPercent.toFixed(0)}%` : "Download"}
|
||||||
|
</button>
|
||||||
|
{/if}
|
||||||
</div>
|
</div>
|
||||||
{#if !selectedModel.installed}
|
{#if downloading}
|
||||||
<button
|
<div class="progress">
|
||||||
class="action"
|
<span style="width:{downloadPercent}%"></span>
|
||||||
disabled={downloading}
|
</div>
|
||||||
onclick={downloadModel}
|
|
||||||
>
|
|
||||||
{downloading ? `${downloadPercent.toFixed(0)}%` : "Download"}
|
|
||||||
</button>
|
|
||||||
{/if}
|
{/if}
|
||||||
</div>
|
|
||||||
{#if downloading}
|
|
||||||
<div class="progress">
|
|
||||||
<span style="width:{downloadPercent}%"></span>
|
|
||||||
</div>
|
|
||||||
{/if}
|
{/if}
|
||||||
{:else}
|
{:else}
|
||||||
<p class="note">No speech models are registered.</p>
|
<p class="note">No speech models are registered.</p>
|
||||||
|
|
@ -509,6 +549,10 @@ Today we are shipping something I have wanted for a long time.
|
||||||
Recognition runs on this machine. Nothing is uploaded, and there is no
|
Recognition runs on this machine. Nothing is uploaded, and there is no
|
||||||
account.
|
account.
|
||||||
</p>
|
</p>
|
||||||
|
<p class="note">
|
||||||
|
Turkish is not here yet — no streaming model has been published for it
|
||||||
|
in a format this engine can load.
|
||||||
|
</p>
|
||||||
</section>
|
</section>
|
||||||
</div>
|
</div>
|
||||||
</aside>
|
</aside>
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue