Contents

M6 Recording: voice actors in the booth

A showrunner can cast real actors. They read their lines in Studio's booth, and their takes play in place of the synthetic (Kokoro) voices: the line is as long as the recording, the mouth moves to it (Rhubarb on the recording), and the cut follows. Recording is local and free, like voicing; nothing leaves the computer (or the person's own service workspace on the web).

Read with docs/design/m4-director.md §7 (VoiceService, the take store, cache keys), docs/design/m3-showrunner.md (the session protocol) and docs/design/m7-redirection.md (the timeline).

What a line plays

A line's take was already one of two things (m4 §7): the recording its take names in the EpisodeIR, used as is, else the voice store's synthetic take for its cache key. Nothing in the compiler changes: choosing an actor's recording pins it in take ({path, hash, mediaType: "audio/flac"}), and compiling probes it, runs lip sync on it and times the line by its length. Choosing the synthetic voice again pins the kept synthetic take the line had (the demo's, in audio/takes/ with a words sidecar naming the line and its words), or removes take so the voice store's plays (voiced by episode.voice when it has none yet).

Choosing is a change set on the episode, the person's own, applied as made (author user, tool choose_take, "Use Ada Lovell's recording of sc01/l01"): Ctrl+Z undoes it, and the timeline compiles again on its change.event.

Where recordings live

A recording is source material: nothing can make it again. So it is kept with the show, in audio/takes/ (where reelwright voice --keep copies approved synthetic takes), never in the gitignored .reelwright/ store, in the same layout as the voice store:

• <sha256>.flac: 48 kHz mono 24-bit FLAC named by the hash of its bytes, encoded as tools/tts-kokoro/takes.py encodes the synthetic voices (bit-exact flags, block size 4096, no metadata);
• <sha256>.provenance.json: PROVENANCE.md's fields (kind: dialogue, the line, generator reelwright booth, model microphone, voice: who read it, the show's license, the sha256), provider {id: "recorded"} (the id show.json's providers.tts already reserves), and recording: actor, recordedAt, version, edit (how this version was made from the recording), input (the format and levels the booth sent: rate, channels, length, sample peak, the loudest 50 ms, clipped) and the ffmpeg version.

A recording has no words sidecar: its word timings come from Rhubarb's speech spans, as for any take without TTS words (core/src/compile/words.ts).

Versions. Trimming or gaining makes a new version, always from the recording itself (recording.source), never from another version, so a trim can be given back and every version is reproducible from the recording, its edit settings and the same ffmpeg. The highest version of a recording is the one shown (or the one the line plays, if it plays an older one); making a version that already exists makes it the newest again; giving every edit back shows the recording itself. Old versions stay on disk, so undoing a choice never points at a missing file. Deleting a recording removes every version of it no episode plays.

What the booth sends, and what is refused

The client sends a 16-bit PCM WAV (mono or stereo, 44.1 or 48 kHz). takes.add checks it (core/src/voices/recorded.ts checkWav) and refuses with why:

Clipping (three full-scale samples in a row) is kept as input.clipped and shown, not refused. The take is resampled to 48 kHz (aresample) and downmixed to mono.

Trim, gain, clean

takes.edit makes a version from the recording with these, in order:

1. Trim: trimStartSec and trimEndSec off the recording's head and tail, to the sample (atrim=start_sample:end_sample); at least 0.2 s stays.
2. Clean (clean): light noise reduction, afftdn=nr=12:nf=<floor>, where the floor is the take's own: the 10th percentile of its 50 ms windows' RMS (the quiet between words), held within -80 and -25 dBFS so a take with no pause cuts no voice. A room at -40 dBFS comes down by about 10 dB; the voice by under 1.
3. Gain: gainDb (-24 to 24), or normalize: the dialogue level the synthetic voices are mastered to, -18 LUFS integrated unless that would push the sample peak over -1 dBFS (takes.py's rule), measured after the trim and the cleaning. The gain applied is kept in edit.gainDb. A normalised take stays normalised when trimmed again (its loudness moves with the trim); a gain given replaces normalising; normalising turned off goes back to 0 dB.

Any setting not given is the version's own. The new version takes the old one's place, as the line's take too when the line played the old one.

Protocol (schemas/v1/session.schema.json)

Every takes.* change answers with the line's takes as they are after it (TakesChangeResult: takes.list's fields, the take made and the change that chose it), so the booth never asks twice. The WAVs and peaks a list names are made once per take, in .reelwright/cache/booth/<sha256>.wav|json.

(implementation) Pieces rather than one message: a 60 s mono take is 5.8 MB of WAV, 7.7 MB of base64, over what Studio's socket sends at once. 510,000 bytes a piece is 680,000 characters, a JSON message under 700 KB.

The booth (engine/addons/reelwright/ui/booth/)

RwBooth opens in Studio's main area from a line selected on the timeline (the inspector's "Record a take", which counts in at once, or "Takes…"; Studio adds both with RwTimelineInspector.add_block_button) or an episode's "Record lines…" (its first line), and Back (Esc) returns there.

• Header. The episode, the character's name, "Line 1 of 3 · Scene 1", Previous and Next (Page Up, Page Down) through the character's lines.
• Session. Who is reading (every take records it) and the microphone (the input devices the audio driver lists, when there is more than one), above the lines: in the header they left the character's name a hundred pixels on a laptop.
• Lines. The character's lines (pick another character above them), each with its place, its words and a mic when an actor's recording plays.
• Prompter. The line said before it ("Barnaby Culpepper: If it's another asteroid…"), the speaker and parenthetical, the line in large type, chips for emotion, intensity and loudness, and the actor's note.
• Transport. Record (Space) counts in (three beats of 0.8 s, the count beside the button, the line still in sight), then records: the prompter's border turns red, the take's length runs, and Space stops it. Esc drops a take being counted in or recorded. A live input meter (-60 to 0 dBFS, green to -12, amber to -1, red above, the peak held) shows the microphone's level whenever the booth is open. A take under 0.3 s is not sent.
• Takes. Each with Play (alone) and Picture (the episode from a second before the line, with this take, in a viewer beside the prompter, stopping a second after it), its waveform (a recording's of the recording, trimmed parts shaded, a handle at each end to drag), "Plays" on the one the line plays, who read it, length and peak, what was done to it; a recording's -3 dB, +3 dB, Normalise and Clean, and Delete (asked first: it cannot be undone); "Use this take" on the others. "Use the synthetic voice" when a recording plays and the line has no synthetic take to choose.

The take is kept as 16-bit PCM as it comes (RwBoothRecording), at the microphone's rate (44.1 or 48 kHz as they are; any other resampled to 48 kHz), so stopping never stalls.

The microphone (RwBoothCapture)

On the desktop an AudioStreamMicrophone plays on a bus of the booth's own, RwBooth, muted, whose AudioEffectCapture (4 s buffer) keeps what it hears: the audio thread fills it, so a busy frame loses nothing, and effects run before a bus's mute, so nothing is heard. It needs audio/driver/enable_input (on in engine/project.godot); Studio mixes at 48 kHz (audio/driver/mix_rate), the takes' rate. It is open while the booth is on screen. In a browser the page records the microphone itself (below, "In a browser").

(implementation)

• The microphone player is played once: played twice, the first playback's stop stops the input under the second (found on the web, when the booth recorded through it there: every take came in silent).
• PulseAudio takes a new input device only while input runs, and starting again goes back to the default: the device chosen is given again once the microphone plays.
• AudioServer.get_input_frames (4.6) reads the driver's buffer directly, but in a browser it stopped after 6,656 frames; the capture bus did not.
• A microphone that gives exact zeros for 1.5 s (none there, or no permission) is said: "No sound comes from the microphone…".

macOS asks for the microphone with the export preset's usage description (privacy/microphone_usage_description) and needs codesign/entitlements/audio_input; both are set in engine/export_presets.cfg.

In a browser

The booth records the microphone in the page, not through Godot (RwBoothCapture.SCRIPT, put on the page once as window.rwMic): getUserMedia({audio: true}) when the booth first opens it, into an AudioContext of its own at 48 kHz (as Godot asks for its own), where an AudioWorklet is handed each quantum of the microphone (128 frames) on the browser's audio thread and posts it to the page as one mono Float32Array, the channels' mean. Each frame read takes what has come, as 32-bit floats in base64. A quantum with no input is posted as silence, so the samples keep the context's time. The worklet runs into a silent gain, so every browser's graph pulls it and nothing is heard. Its module is a file beside the page (engine/app/web/rw-mic-worklet.js, put there by tools/web/build-studio.sh), not a blob: the page's Content-Security-Policy runs only scripts of Studio's own origin (core/src/host/studio-csp.ts).

Why not Godot's microphone, as on the desktop. The web export is single-threaded. Its audio worklet plays what the page's main thread mixed, and asks for the next mix by a message the main thread answers between frames; a quantum it has nothing for is skipped, never mixed. The microphone's input keeps coming meanwhile, so a frame that holds the main thread longer than the mix's buffer (about 40 ms) drops that much of the microphone: the take comes out short, its sound chopped. Measured with a probe of the booth's path (the microphone on a muted bus, its capture read every frame; headless Chromium, its CPUs free, a 6 s take), the main thread held for a set time each frame:

The same held 100 ms with a WAV fed to Chromium's stand-in device (1.45 s of 6.06) and with an oscillator for the microphone (1.48 s of 6.12): the context and the input keep the clock, and only the mix falls behind. So it is no artefact of the stand-in device: a person whose page draws slowly loses sound, and Studio's web frames on a modest computer run well past 40 ms. Starved of CPU instead (Chromium on two CPUs of a busy machine, then with six busy loops on the same two; the page's own frames light), the context, the stand-in device's input and the page's worklet kept within 1% of the clock (5.99 to 6.07 s of about 6.05), and Godot's mix 51 to 65% of it: the stand-in device never lost the sound, the main thread did.

What is different there:

• the browser chooses the microphone (its site settings); Studio lists none to pick;
• it asks for {audio: true}, as Godot did, so the browser's echo cancellation, noise suppression and automatic gain are on (Chromium reported all three): the take is processed before the booth hears it, unlike on the desktop;
• the count-in is timed by frames: on a page that draws slowly it runs slow. The take's length is its samples', and the page's recorder keeps every one, so a slow page never shortens a take: tools/web/booth-take-check.ts records 6 s with the page's main thread held 100 ms of every 120 and holds the take's sound to its clock within 0.3 s;
• tested in Chromium only.

The timeline

A Dialogue block whose take is a recording has a small mic before its words (recorded in episode.timeline), and its tooltip says an actor's recording plays, else the show's synthetic voice. Select a line: the inspector has "Record a take" and "Takes…". After a take is chosen, the timeline compiles again on the change's change.event, and the block is as long as the take.

Tests

• core/test/takes.test.ts: a recording sent in pieces is kept by hash (ffprobe: FLAC, 48 kHz, mono, its samples) with its provenance, and chosen by an applied change; each bad WAV is refused with its code and nothing kept; trims and gains are measured with ffmpeg (the samples to the sample, the peak 6 dB up, -18 LUFS normalised, under -1 dBFS); a cleaned room is 4 dB quieter; compiling after choosing times the line's clip to the recording's samples and runs lip sync on its file, and recorded names it; choosing back, undo, delete and its refusal; playing a take against the picture without choosing it; and the whole upload over the daemon's socket, every message schema-checked.
• engine/tests/ui/rw_booth_test.gd against tests/fixtures/session/booth.json (recorded from the real studio) with RwFakeCapture: the prompter, the count-in dropping what it hears, the WAV sent (header, length, samples), takes.add's params, choose, trim, gain and delete sending what the core expects, a refusal said, Space and Esc, a microphone that cannot open; pieces that fit the socket, and resampling.
• rw_timeline_test.gd: a recorded line is marked after the change compiles again. rw_episodes_view_test.gd, rw_studio_test.gd: the booth opens from the inspector's buttons on a line and from an episode, and goes back.
• engine/tests/support/rw_booth_tour.gd: the booth against the real daemon, screenshots of each step; --real-mic records from the computer's own microphone. engine/tests/support/rw_mic_probe.gd listens through RwBoothCapture alone.

Recording (the prompter's border red, the meter live):

A take trimmed and normalised, played against the picture from just before its line:

The timeline after: l01's block carries the mic and is as long as the recording (the episode went from 0:28.08 to 0:26.71); selected, the inspector offers "Record a take" and "Takes…":