What happened this week
Five distinct SLM papers converged on the same broad question the field has been asking since W22: how do you get a modern text-first LLM to produce speech without giving up its language-model capabilities? Different labs attack from different angles — frozen-backbone S2S conversion, dynamic frame rate control, instruction-following without instruction tuning, evaluation that respects speech-native prosody — and one production-infrastructure paper turns the corresponding inference stack from a research prototype into something you can actually deploy.
Foundational — extending S2T LLMs into S2S without the tax
PRIME-Speech (Hu, Lu, Fan) is the paper to read this week. The premise: strong S2T LLMs already provide robust speech perception and text reasoning, but adding S2S output is hard because fine-tuning the backbone degrades S2T performance while attaching a downstream talker reintroduces a serial TTS bottleneck. PRIME-Speech is a frozen-backbone S2S conversion framework that trains only speech-generation modules — a causal audio post-decoder synchronises with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from a text output that has to be re-synthesised. The result is an S2S surface that preserves the S2T LLM's original capabilities intact.
FlexiSLM (Li, Wang, Tian) opens the other axis. Existing SLMs represent speech at a fixed frame rate (25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no inference-time trade-off between quality and speed. FlexiSLM applies recent dynamic-frame-rate audio tokenisation to SLMs, enabling controllable inference-time frame rates — a very low average frame rate for latency-sensitive settings, higher rates for quality-sensitive ones, all in the same model. A concrete lever for the same inference-time control problem last week's LiveKit Turn Detector attacked from the other side.
Foundational — evaluation goes speech-native
Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems (Hallur, Thebaud, Tinchev et al.) closes a real gap: current S2S evaluation lacks interpretable speech-native measures for conversational prosody and rhythm. Because F₀, speaking rate, articulation rate, and pausing shift with model-predicted speaker traits and interaction state, pooled human statistics can be poorly calibrated for evaluating a particular output. Using 4000+ hours from the Seamless Interaction dataset, the paper constructs matched reference regimes for F₀ mean, F₀ expressivity, speech rate, articulation rate, pause ratio, and mean pause duration — a concrete alternative to the text-mediated evals FD-Bench and EVA-Bench have been using.
Foundational — instruction following without instruction tuning
SpeechCombine (Du, Zhang, Qian) makes an unusual claim: instruction tuning for SLMs may not be needed at all. Existing SLM training largely replicates the text-LLM paradigm by synthesising large-scale speech pre-training and instruction-tuning datasets — but speech sequences are significantly longer than text, so this strategy is hard to scale. SpeechCombine proposes an instruction-tuning-free approach that unlocks compositional speech-text abilities through a training strategy that decouples the modality and the instruction axes. If this holds up, it changes the cost structure of building SLMs materially.
Foundational — the production inference layer
An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation (Wang, Tian, Arora) is the practical bookend. High-throughput LLM inference engines lack native support for multimodal generation — decoupled AR+NAR audio-token generation and delay-pattern interleaving conflict with vLLM's single-stream loop. The paper extends autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling on GPU. Whoever ends up running open SLMs at production throughput will use something that looks like this.
Product — Pipecat 1.5.0 lands TTFA metrics
Pipecat 1.5.0 adds Time to First Audible (TTFA) metrics to TTS services, reported alongside the existing TTFB (Time to First Byte). TTFA measures the time to the first audible sample — TTFB plus the leading silence many providers pad onto the start of a response — so comparing the two shows how much perceived latency is padding versus real service response time. Audible onset is detected from short-time RMS energy that rejects noise-floor blips and brief transients. Also lands Together AI STT/TTS services, an NVIDIA TTS per-sentence synthesis mode with zero-shot voice-cloning prompts, an on-heartbeat-timeout event handler for PipelineWorker, Gemini Developer API backend for GeminiTTSService, and the AssemblyAI U3 Pro mode parameter for latency/accuracy tradeoffs.
What is not here
livekit-agents 1.6.3 (Jun 22) and 1.6.4 (Jun 24) shipped inside W27 (window Jun 22-28), not W28. W27 was skipped by the scheduled task and remains a candidate for a separate backfill. No verified in-window dataset drop; Cartesia, Hume, Deepgram Voice Agent, and ElevenLabs Agents shipped no in-window technical changelog items.
Corrections to hello@fullduplex.ai.