Fish Audio S2 Pro / Drama 3
This first-party Soniqo page documents Fish Audio S2 Pro from the local speech-swift / speech-core implementation. Hugging Face bundles are linked below after the integration notes.
Landing cards and docs menus now point here first; source model and bundle links remain available from this page.
At a Glance
| Model | Fish Audio S2 Pro |
|---|---|
| Role | Experimental multilingual TTS with raw-reference cloning and style markers |
| Backend | MLX fp16 |
| Output | 44.1 kHz mono PCM |
| Languages | Multilingual |
| License | Research / non-commercial bundle; obtain Fish Audio license for commercial exposure |
| Status | Programmatic runtime; CLI integration is still pending |
| Source | Fish Audio S2 Pro |
| Swift product | FishAudioTTS |
| CLI / runtime | Programmatic runtime today; planned speech speak --engine fish-audio |
Use
The snippet below mirrors the current speech-swift API or command exposed by the repo.
import FishAudioTTS
let model = try await FishAudioTTSModel.fromPretrained()
let pcm = try await model.generate(
text: "आज मैं बहुत खुश हूँ। [excited]",
referenceAudioURL: URL(fileURLWithPath: "reference.wav"),
referenceText: "नमस्ते, यह संदर्भ आवाज है।"
)Model Links
Implementation Notes
- Download uses the newer explicit byte-weighted file manifest, so progress reflects real transferred bytes across the multi-GB shards.
- Control markers include [pause], [emphasis], [laughing], [excited], [angry], [whisper], [screaming], [shouting], [surprised], and [sad].
- The runtime generates 10 DAC codebook rows, then decodes generated or reference-conditioned codebooks through FishAudioCodec.
- Current tests cover bundle loading, codebook generation, codec encode/decode, Hindi cloning, and ASR round-trip gates.
Fish Audio Drama 3
Experimental native MLX support for Drama 3 is available through FishAudioDrama3Model. A 35B hybrid-attention MoE backbone, temporal audio tower and Fish DAC codec produce 44.1 kHz mono audio. The 4-bit checkpoint passed waveform and independent ASR checks for text-only generation, reference cloning and CLI output on Apple M5 Pro. 8-bit and BF16 remain unvalidated.
The default balanced mode trims idle MLX buffers above 512 MiB at stage boundaries. The shared model-lifetime cache cap also applies, preserving lower caller budgets; neither control limits total process memory. --fish-drama3-memory-mode unrestricted or memoryOptions: .unrestricted loads weights eagerly and skips stage trimming, while retaining the lifetime cap.
Text-only generation defers 794.4 MiB of reference encoder weights. The first reference encoding loads and retains them, so keep codec.safetensors available until then. Precomputed reference codes do not need the encoder.
Provide complete input text. Voice cloning requires a reference clip and its exact transcript. Speaker tags 0–15 and inline emotions are supported. PCM streaming is available through generateStreaming, generateStream and --stream. The default chunk contains 3 codec frames, about 139 ms of audio. Automatic long-text chunking and streamed text input remain unavailable. Reaching the frame limit throws an error.
speech speak "Hello from Drama 3." --engine fish-drama3 --fish-drama3-bundle-dir /path/to/bundle -o hello.wav
speech speak "Hello from Drama 3." --engine fish-drama3 --stream --fish-drama3-chunk-frames 3 -o streamed.wav
speech speak "Hello in the reference voice." --engine fish-drama3 --voice-sample reference.wav --fish-drama3-reference-text "The exact reference transcript." --seed 42 -o cloned.wavimport FishAudioTTS
let model = try await FishAudioDrama3Model.fromBundle(bundleURL, memoryOptions: .balanced)
let pcm = try await model.generate(text: "Hello from Drama 3.", options: .init(seed: 42))
try await model.generateStreaming(text: "Hello from Drama 3.", options: .init(seed: 42, streamingChunkFrames: 3)) { chunk in
print(chunk.frameIndex, chunk.samples.count, chunk.isFinal)
}| Option | Default |
|---|---|
--fish-drama3-model-id | mlx-community/fish-audio-drama-3-4bit |
--fish-drama3-bundle-dir | — |
--fish-drama3-reference-text | — |
--fish-drama3-max-frames | 2048 |
--fish-drama3-temperature | 1 |
--fish-drama3-top-k | 30 |
--fish-drama3-top-p | 0.9 |
--voice-sample | — |
--seed | 0 |
--stream | false |
--fish-drama3-chunk-frames | 3 |
--fish-drama3-memory-mode | balanced |
Built with Fish Audio. Bundles retain the Community License and NOTICE.
4-bit benchmark
| Mode | First PCM, ms | Warm synthesis, s | Audio, s | RTF | Peak RSS, GiB | Peak physical footprint, GiB |
|---|---|---|---|---|---|---|
| Text-only / Buffered | — | 2.306 | 3.483 | 0.662 | 14.457 | 24.220 |
| Reference cloning / Buffered | — | 2.681 | 3.204 | 0.837 | 15.064 | 24.789 |
| Text-only / Streaming | 275.3 | 2.529 | 3.483 | 0.726 | 15.571 | 22.321 |
| Reference cloning / Streaming | 730.2 | 2.860 | 3.204 | 0.893 | 14.676 | 23.885 |
Apple M5 Pro, 48 GiB, release build, 2026-10-17. Median of three warm calls, measuring complete waveform generation. Model loading and WAV file I/O are excluded; cloning includes reference encoding on every call. RTF = synthesis time / audio duration. Independent ASR: 16/16 clips, zero word errors. Use physical footprint for unified-memory sizing. First PCM is the first nonempty streaming callback; buffered rows have no chunk callback. The default chunk contains 139.3 ms of audio. Audible playback onset was not measured.