An open test · 16 phrases
The dictation test for code
Sixteen phrases that dictation reliably gets wrong on technical text —
~/.zshrc, git commit --no-verify,
useCallback, src/lib/router.ts:123. Read them
aloud into whatever dictation tool you already use, compare what lands
against what should have landed, and score it yourself.
Bring your own tool. This page measures nothing on your behalf and sends nothing anywhere — the scoring happens in your browser, and the only judge is your own screen.
-
Step one
Open the editor or terminal you actually work in. Put the cursor in a scratch file.
-
Step two
Dictate each phrase once, at your normal speaking pace. Don’t slow down for the machine — that is not how you would really use it.
-
Step three
Mark it correct only if every character matches. A path that is right except for one space is a path that does not resolve.
File paths
A path is a single token built out of separators. Every separator is a place the transcription can put a space, and a space is the difference between a path and a sentence.
- Say aloud
“tilde slash dot zshrc”
Should appear
~/.zshrc
“zshrc” is out of vocabulary and a leading dot before a word is orthographically rare, so decoders reach for “Z shirt see” or “dot ZSH RC”.
- Say aloud
“cd dot dot”
Should appear
cd ..
A two-letter command followed by a bare double dot. Common outputs are “CD..”, “seedy dot dot”, and the spelled-out words.
- Say aloud
“src slash lib slash router dot ts colon one twenty three”
Should appear
src/lib/router.ts:123
Three separators in one token, then a line-number colon that most engines treat as sentence punctuation and drop entirely.
- Say aloud
“dot slash node underscore modules slash dot bin slash vitest”
Should appear
./node_modules/.bin/vitest
An underscore has no spoken form — the engine has to infer the convention — and a leading “./” is almost always swallowed.
Commands and flags
Flags are the case where a near-miss is worse than a blank. A path that comes out wrong fails loudly; a flag that comes out wrong runs something you did not ask for.
- Say aloud
“git rebase dash i capital head tilde three”
Should appear
git rebase -i HEAD~3
“dash i” lands as “-I” or “dash eye” about as often as “-i”. And you have to say “capital” out loud, because nothing in the sound of a word carries its case — git will not accept a lowercase head.
- Say aloud
“git commit dash dash no dash verify”
Should appear
git commit --no-verify
The classic. Two hyphens get typographically “corrected” into a single em dash — which the shell rejects outright — and the third one, inside “no-verify”, tends to come back as a space.
- Say aloud
“chmod plus x scripts slash deploy dot sh”
Should appear
chmod +x scripts/deploy.sh
“chmod” is out of vocabulary and “plus x” is far more likely to arrive as two English words than as a mode flag.
- Say aloud
“grep dash r n router src slash”
Should appear
grep -rn router src/
Two short flags spoken as separate letters have to fuse into one token, and the trailing slash has to survive at the end of a line where every prior instinct says put a full stop.
Code identifiers
Identifiers are words your language model knows, joined by a convention it has no acoustic reason to apply. Nothing in the sound of “use callback” asks for a capital C.
- Say aloud
“use callback”
Should appear
useCallback
camelCase means closing a space and capitalising a letter on the strength of context alone. The audio contains no signal for either.
- Say aloud
“kubectl get pods dash n prod”
Should appear
kubectl get pods -n prod
People pronounce “kubectl” at least four different ways, so there is no stable spelling prior to fall back on.
- Say aloud
“max underscore retry underscore count”
Should appear
max_retry_count
You have to say the separator out loud, and then the engine has to write the symbol instead of the word — while also not putting spaces around it.
- Say aloud
“async slash await”
Should appear
async/await
A slash between two keywords with no spaces around it. Most engines add the spaces, or write the word “slash”.
Your own words
The first three tiers test accuracy. This one tests a design choice — whether your tool treats how you said it as noise to be tidied. When the text is going to an agent, the hedges and the corrections are the instruction. Score the words, not the typography: a straight apostrophe instead of a curly one, or a comma where we printed a dash, is not a miss.
- Say aloud
“don’t refactor the whole file, just fix the one function”
Should appear, word for word
don’t refactor the whole file, just fix the one function
Mark it wrong if “don’t” became “do not”, or if “just” — which is doing the scoping work in that sentence — was dropped as filler.
- Say aloud
“I think this is probably a race condition, but I’m not certain”
Should appear, word for word
I think this is probably a race condition, but I’m not certain
Mark it wrong if the hedges were removed. “I think” and “probably” are how you tell an agent the confidence level of what follows.
- Say aloud
“use map here — actually no, use flatMap”
Should appear, word for word
use map here — actually no, use flatMap
Mark it wrong if the correction was silently resolved to “use flatMap”. Tidier, and it deletes the fact that you considered and rejected map.
- Say aloud
“this is really slow, like unusably slow”
Should appear, word for word
this is really slow, like unusably slow
Mark it wrong if it came back as “This is slow.” The intensifier was the whole message; the sentence without it is a different report.
Your result
Self-reported, and that’s fine.
0 of 16 phrases marked correct.
- File paths
- 0/4
- Commands and flags
- 0/4
- Code identifiers
- 0/4
- Your own words
- 0/4
Nothing here is transmitted. The result is stored in your own browser so a refresh doesn’t lose it, and the copy button puts a plain-text summary on your clipboard so you can paste it wherever you were going to argue about it.
Why does dictation get file paths and shell commands wrong?
Because the thing that makes transcription good at English is the thing
that makes it bad at code. An engine resolving ambiguous audio leans on a
language model trained overwhelmingly on ordinary prose, and reaches for
the sequence that looks most like a sentence. A file path is the exact
opposite of a sentence: one token assembled out of separators that have
no spoken form, holding words that are out of vocabulary.
tilde slash dot zshrc has one correct rendering and dozens
of plausible English ones, and nothing in the audio tells them apart.
Which is why the tiers on this page are grouped the way they are. They are not a survey of everything speech recognition finds hard — they are the three technical registers where the failure is both common and consequential, plus a fourth that is not about accuracy at all.
What this test does not measure
- It is not a benchmark. Sixteen phrases scored by the person who spoke them is not word error rate. One accent, one microphone, one room, one run. Treat the result as a demonstration you performed rather than a measurement you took.
- It is deliberately unfair to prose tools. Every item is chosen because it is hard. A tool that scores badly here may be excellent at the email you were going to write next, which is what most dictation is actually for.
- Tier four is a design choice, not an error. A tool that expands your contractions and drops your hedges is doing what it was built to do. Whether that is a failure depends entirely on whether the text was going to a person or to an agent.
- Halopen does not ace it either. We publish the corpus knowing our own app misses items on it. An instrument that only flattered the people who built it would not be worth handing to anyone.
Where the corpus comes from
The three technical tiers mirror the two registers Halopen’s own transcription router treats as special cases — shell commands and file paths in Terminal, iTerm2 and Warp; code identifiers and file paths in VS Code and Cursor. Those categories were not chosen for this page. They were already in the app, because they are where the reports came from.
It is worth being precise about what kind of corpus this is, because there is a real tradition it does not belong to. Speech systems have been measured since 1969 against the Harvard Sentences — 72 lists of 10 sentences published in IEEE Std 297-1969, Recommended Practice for Speech Quality Measurements — and their defining property is that they are phonetically balanced: each phoneme appears about as often as it does in English, so a score generalises.
This corpus is the deliberate opposite. It is unbalanced on purpose, selected for difficulty rather than representativeness, which is exactly why a result here tells you something about one narrow register and nothing about speech recognition in general. It is a set of hard cases, not a measurement.
Use it, fork it, argue with it. If you think an item is unfair or you have one that belongs here, write to [email protected] — a test corpus gets better by being disputed.
If tiers one to three went badly
That category is the one Halopen was built around.
Native macOS dictation that writes what you said. It tells the decoder to expect shell commands and file paths in Terminal, iTerm2 and Warp, and code identifiers and file paths in VS Code and Cursor — and it drops to verbatim in all eight code and terminal apps, because a shell command is not a sentence to be improved.
8,000 words a month, free forever. No card. macOS 14.0+.