Start with the decision you need to make. A benchmark spreadsheet helps only when the comparisons can support that choice. Before comparing browser speech, a small server and a managed service, decide what “good enough” means for the people using it, what evidence would change your mind and how every paid resource will be stopped.
Keep only routes that meet the agreed gates. This is a proposed experiment; no benchmark results are reported here.
Name the decision before naming a model
Ask a specific question: can a small CPU worker transcribe technical answers accurately enough, can a tested browser voice read questions clearly, or does a managed service justify its cost for a particular device group? Keep speech recognition (STT) and speech synthesis (TTS) as separate decisions. A provider that works for one may not suit the other.
Write down the users, language and accents you need to test, expected answer lengths, device types, peak simultaneous sessions, privacy destinations, latency target and spending limit. Mark every number as either a requirement, a proposed threshold or a measured result. Do not let an illustrative price or a provider's broad claim become a benchmark finding.
Compare the same work
Prepare consented recordings from a range of speakers, including short and longer technical answers. Keep some speakers and clips out of configuration tuning; score those only after the comparison settings are frozen. Record reference transcripts and agree how to treat numbers, fillers, abbreviations and technical terms before calculating errors.
For speech recognition, report word error rate and errors on meaning-critical terms. For generated speech, have a person listen to the same questions in a randomized order and rate intelligibility, pronunciation, pacing and overall usability. Browser event logs or successful API responses cannot tell you whether someone could hear or understand the speech.
A tiny Linux instance may be inexpensive per hour but slow for one answer, short on memory or limited by burstable CPU credits. A queue can spread work across workers; it cannot make a single job's inference time disappear. Test direct warm processing and queued processing separately, and include startup, idle time and worker readiness in the cost picture.
Measure the pause a person experiences
Record the time from the end of a person's answer to the first audible part of the next question. Break that pause into capture finalisation, upload, queue wait, recognition, application response, synthesis and playback start. Report the complete distribution and failures, not only the fastest successful request. When a browser event is used as a proxy for audible output, label it as a proxy.
Test the actual concurrency you plan to claim. Staggered sessions represent ordinary conversation; open-loop arrivals reveal queue growth even when a slow response would otherwise delay the next request; synchronized bursts expose sudden backlog. A test using simulated browser timing is a server-load simulation, not evidence that dozens of real phones support a speech API.
Include failure, privacy and recovery
A fair comparison includes microphone denial, a missing language pack or voice, a dropped network, duplicate queue delivery, a worker restart, a delayed result and a user cancelling playback. Decide which data may go to a browser vendor, your own server or a managed service. Unknown processing location should be treated as unknown, not assumed to be on-device.
If a transcript must be retried, keep the original answer intact, cap the number of attempts and avoid committing two versions of the same turn. For generated speech, prevent old audio from starting after a person has cancelled or a replacement is already playing. If safe recovery is not available, show the text and offer a manual path.
Quote the run and plan its teardown
Count the gateway, workers, storage, queues, logging, public network access, managed audio requests, model downloads, idle periods and cleanup time. Keep raw unit counts with the dated price source. Show marginal request cost separately from the fully allocated cost of the always-ready system; a low per-request fee does not pay for an idle server automatically.
Put a spending limit, maximum runtime, allowed instance types, data-retention period and expiry time in the run configuration. Stop new work at the limit, terminate only resources created for that run, then audit for leftover addresses, disks, snapshots, queues, logs and scheduled cleanup jobs. Preserve the cost ledger outside the resources being destroyed. A laptop closing must not be the only thing that stops a paid experiment.
Record what failed as carefully as what passed
A small model that misses technical words, a browser voice that is unclear on a tested phone, or a worker that meets throughput but misses conversational latency is still a valuable finding. State the exact device, model, workload and load tested. Separate measured results from projections and say which unanswered question would justify the next experiment.
This guide summarises a proposed experiment plan. Its model matrix, sample counts, latency targets, cost ceiling and pass thresholds are choices for that experiment, not universal standards or AWS service guarantees. Verify current instance and service availability, provider terms, rates and regional behavior before spending. For a business process that includes AI or speech, see VikaSquare's workflow automation scope; a real project still needs its own requirements, privacy review, technical tests and budget.