Sound and delivery: music, singing, directed voices, and renders to spec
The October 2026 product audit (docs/audits/product-audit-2026-10.md) found that a musical
could not be made (finding 5), that the voice could not be directed (finding 6), that previews
were master encodes (finding 10), and that renders were 16:9, H.264 and whole-number frame rates
only (finding 11). This is what was built for each, how it works, and how good it is.
Music over the episode
Music was scene-local. A track ended at its scene's last frame, and a clip that ran past it was
an error. duck and swell cues were accepted and then ignored ("not mixed yet").
Music is now laid over the whole episode (core/src/compile/music.ts, EpisodeIR 1.44):
- Across cuts. A start cue with
continuesplays on across every scene cut, under the scenes after it. It stops at a stop cue for its track in any scene, at its out point, or at the episode's end, credits included, fading over its last second. One play is one clip, or one clip per loop, so there is no seam at a cut. Withoutcontinues, music still ends with its scene. - In and out points.
inSecandoutSecchoose the stretch of the track to play, and a loop goes round between them. - Under dialogue. Every music start dips while anyone speaks, as a sidechain compressor
keyed from the dialogue bus would duck it. The dip is computed from the audio plan, so it is
already down 0.15 s before the first word. It comes back up over 0.6 s after the last word,
and holds down across a pause shorter than both together.
duckDbsets the depth: default -10 dB, and 0 never dips.- A sung line does not duck it: the song is sung over the music.
- Duck and swell. These cues move a playing track's level over their
fadeSec(default 1 s), from wherever it was.duckdefaults to -12 dB andswellto +4 dB.- A swell with gainDb 0 brings the track back.
- Levels in the mix. Levels are a clip's
gainEnvelope(Timeline 1.21): points in dB at sample offsets, linear between.- The mixer writes each envelope as a mono float WAV, one gain per sample, and multiplies it
in (
amultiply). - So the automation is exact to the sample, and the music stem still sums with the others to the pre-master.
- The mixer writes each envelope as a mono float WAV, one gain per sample, and multiplies it
in (
Measured in core/test/music.test.ts. The show is the demo with two copies of its scene and a
440 Hz tone as the music.
| What | Result |
|---|---|
| Music stem, level | -33.07 dBFS |
| Music stem, under dialogue | -43.07 dBFS, a dip of -10.00 dB |
| 10 ms windows ±0.2 s around the cut | within 0.09 dB of the curve |
Before this change, the track ended at the cut with a 10 ms fade.
Room tone
A set's ambience (set schema 1.1), or a scene's own (Scene.ambience, off: true for none),
lays a bed from the scene's first sample to its last.
- The sound. By default it is
sfx.room_tone. Every show has it: a built-in recipe of brown noise under 250 Hz and a little pink air between 1.5 and 6 kHz, peaking at -38 dBFS, about -50 dBFS RMS. A show with its ownsfx.room_tone(a recording of its set) plays that instead. - Looping. A bed shorter than its scene loops, each loop crossfaded into the next over 50 ms.
- Across cuts. Scenes back to back with the same bed share one, unbroken across the cut.
- Stem. Room tone mixes into the effects stem.
- Where it is set. A set's own, in the set explorer with nothing selected: Room tone (none,
Reelwright's own, or one of the show's sounds) and, once it has one, its level. A scene's,
from the timeline's Music and room tone…, or
set_ambience.
Measured: across a cut between two scenes on the same set, the effects stem holds the bed's own samples to within 3.4e-10.
Foley
A rendered episode had lines, music, room tone and the sound cues someone wrote. Nobody walked
audibly, sat down, or set a cup down. Foley is now made from the compiled Timeline
(core/src/compile/foley.ts), so no one places it by hand. It is Timeline 1.28's clip kind
foley, each clip naming the sound it plays (sound).
- Footsteps. Each walk, stride, sneak, shuffle, run and hop key steps where the engine's
pose solver plants a heel. The core works out the gait's cadence as
RwPoseSolver._walkdoes: the leg's length from the character's metrics, the gait's stride and stance, and the style's walk. A step plays on the first frame at or after its strike. A body of another plan steps by its footfall pattern (a quadruped's walk in four beats a cycle, its gallop in pairs). Slides and floats make no sound. - The floor. A set's
floor(set 1.4: wood, metal, carpet, stone, grass, sand, water). Without one, it is read from the set's words (a deck, a beach, a lawn, a pier), else stone outdoors and wood indoors. Each floor has four takes. A step's take is drawn from the episode's seed, the character and the frame, so the same frame gives the same sound and a walk does not repeat itself. A run is 3 dB up, a sneak 9 dB down. A body taller than 1.6 m steps harder and a smaller one lighter (6 dB per doubling, −9 to +3). - Cloth and body. A sit, landing at the end of its blend; a stand; a turn of 45° or more within 1.5 s by someone not walking; a gesture of intensity 0.5 or more.
- Props. Picked up, put down, handed over and used, by what the prop is made of. That is its
material(prop 1.1: wood, metal, glass, ceramic, plastic, paper, cloth), else its words or its recipe's finishes, else wood. A thrown prop lands. - Doors. Someone coming on or going off through an entrance with a door: it opens before them and closes after.
- Where it plays. Each sound comes from where it happens. It is panned by the shot on screen at that moment and quieter far from the camera (−20·log10(d / 3 m), at most −12 dB).
- In the mix. The core sums all foley into one stereo bus before ffmpeg (
withFoleyBusincore/src/audio/mixer.ts), so a scene of hundreds of steps is one input. The bus has its own stem,foley, instems_wav.- It dips
duckDb(default −8 dB) under spoken lines, with the music's attack and release. - It plays at
levelDb(default 0). show.jsonformat.audio.foley.offturns it off for the show; a scene'sfoley.off(EpisodeIR 1.51) turns it off for that scene.
- It dips
- The sounds. 53 SoundRecipes in
core/src/audio/foley.ts: 28 steps, 8 cloth movements, 14 handlings, a handover, and a door opening and closing.- They are synthesized into the show's cache when first played.
- Each carries its licence, CC0-1.0, and its generator, Reelwright's built-in. The release gains a few kilobytes of numbers and no recordings.
- A show's own sound of the same id (
audio/sfx/foley_step_wood_1.flac) plays instead, as it does for the room tone.
In Studio
The timeline has a Foley lane under Effects, one block per sound, named by what it is ("step wood 2", "cloth sit 1 · muted").
- The inspector plays it as another sound (the show's or the built-in library's), sets its gain, mutes it, or turns the scene's foley off.
- The block's menu has gains, As made (the person's touches taken off), and Mute or Play it again. Delete mutes it.
- How it is saved. Each is
edit_episodeset_foley {scene, event, mute | sound | gainDb | reset}on the event's id (otto.w1.s2,culpepper.sit1,pick.pr90). The touch lasts through recompiles for as long as the action that makes the sound does. - A missing sound. The timeline's Add (A, S) puts one in at the playhead. The built-in foley sounds are in its library after the show's own sounds.
Measured
node tools/foley-check.ts renders two scenes with the engine and measures their stems. One is
Low Tide Blues' "Last Call" (tools/lib/foley-watch-show.ts): a detective comes in through a
door, two people cross a wooden club, one sits and stands, and a mug is picked up, drunk from
and set down, with the lines voiced by Kokoro. The other is the Lighthouse. The engine's heel
strikes come from engine/tests/support/rw_foley_probe.gd, which steps each walk a twentieth
of a frame at a time.
Measured on 2026-10-08, at 640x360, on the render's own stems:
| What | Low Tide, "Last Call" | Lighthouse |
|---|---|---|
| Foley sounds | 32: 21 footsteps on wood, 2 door opens and 2 closes, a sit, a stand, 2 turns, the mug picked up, drunk from and put down | 4 on the metal deck: a turn, a stand, a gesture and a sit |
| Heel strikes (the engine's) | 21: Cora 7 and 8, Octavius 2 and 4 | none: nobody walks |
| Footsteps found in the stem by their own sound | 21 of 21, each 0.00 to 0.90 frames after its strike (mean 0.37) | |
| Onsets heard, steps with no other sound near | 11, from −0.03 to +0.81 frames of the strike | |
| Onsets in the stem not a strike or another foley sound | 0 of 28 | 0 of 5 |
| Sounds alone, outside speech: heard against their clip's gain | 12, within 0.72 dB (the 2 ms fade-in takes up to that off a sound loudest from its start) | 3, within 0.05 dB |
| Sounds alone, under speech: heard against gain − 8 dB | 4, within 0.72 dB | 1, within 0.01 dB |
| Under speech, below its loud 20 ms (90th percentile) | 14.2 to 18.8 dB | 26.2 dB |
A footstep is found by a matched filter on its heel's crack. The crack is noise drawn for that take, so the filter tells it from another take: Cora and Octavius cross the club a frame apart for four steps. The filter works in a band of 1 to 5 kHz, where the heel cracks but the thump does not ring and the other person's toe barely hisses. A step plays on the first frame at or after its strike, so each is between 0 and 1 frame after it, and on that frame the engine draws the heel planted.
The spectrograms (spectrogram-mix.png, spectrogram-foley.png, spectrogram-walk.png in each
scene's folder) show each step as a thump under 500 Hz with its heel's and toe's cracks above,
the strikes ticked along the top of the walk landing on each. Each rustle is a broad band of
noise. The mug sounds as three ceramic partials. Every foley sound sits under the lines and
dips while they are spoken.
Importing music
A music file comes into the show through the sound library's import (sounds.import, built
for the sound library page on the edit-everything branch).
- Formats. It takes WAV, FLAC, Ogg or MP3, up to 256 MiB through
takes.uploadwithpurpose: "sound". - What is kept. It is kept as
audio/music/<slug>.flac: 48 kHz, 24-bit, its own channels, its level untouched. - Using it. A music cue plays it as
music.<slug>. - In Studio. Music and room tone… (in the timeline's inspector with nothing selected or on a music block, and in the command palette) imports a file, through the browser's own picker on the web, and starts a track at the playhead: in and out points, played on across cuts or not, looped or not. It also sets the room tone of the scene under the playhead.
Directing the voice
The voice maker is Kokoro-82M. It takes words, a voice and a speed, and nothing else:
- no emotion or style prompt;
- no pitch;
- no per-word control.
So a line's direction is split (core/src/voices/direction.ts). The speed goes to Kokoro. The
rest is done to its take by prosody (core/src/audio/prosody.ts), which:
- tracks the pitch every 5 ms with YIN on the line resampled to 16 kHz;
- lays two-period grains, cut at the voice's own glottal pulses, back down at the spacing of the pitch wanted (TD-PSOLA). This moves the pitch and keeps the formants: the speaker stays the same person.
A warp maps each moment of the output to a moment of the input. With it, words stretch and pauses open independently of pitch.
What reaches the voice, each adding to the last:
| From | Becomes |
|---|---|
| emotion × intensity (default 0.6) | pitch, range, speed, end bend, tremble (EMOTIONS: angry is +1.5 st, range ×1.5, 10% faster, ending 2 st down, at full intensity) |
| volume | shout +3 st and livelier; whisper flatter |
| parenthetical and note | "angrily", "slowly", "flat", "questioning", "trembling", "sarcastic", "higher", "deeper"… |
| quoted words in the note | stress 'never' or just 'never' stresses it; pause after 'wait' and a beat before 'the tide' hold 0.45 s |
delivery.pitchSemitones, emphasis, pauses |
as given (the line inspector's Pitch, Stress and Holds) |
| the character's binding | pitchSemitones, and formant (timbre) |
delivery.variant |
another take (below) |
What each one does to the take:
- Stress. A stressed word is 2.5 semitones higher at its middle, 15% longer and 2.5 dB louder.
- Range. It scales the line's pitch movement about its own median.
- End bend. It ramps the last word up (a question) or down (finality).
- Timbre. The line is resampled by the formant ratio, then its pitch is put back.
The take keeps its loudness and its 60 ms edges, and its word timings are moved to match. The read goes into the take's cache key only when it changes the take, so a line read as it always was keeps its take.
Measured on a synthetic voice (glottal pulses through three formants, core/test/prosody.test.ts):
- Pitch. Frame by frame, a shifted line follows its own contour within 0.1 semitone.
- Formants. The spectral centroid moves less than 6%. Resampling moves it 26% for +4 st.
- Stress. A stressed word peaks at +2.5 st and is 15% longer.
- Pauses. A 0.4 s pause is silent.
- Range. Halving the range halves the spread of the pitch.
- Timbre. ±3 st of timbre moves the centroid by 2^(±3/12) at the same pitch.
The fake voice in core/test/voice-direction.test.ts tests the whole path. A +4 st line made
at 286 Hz measured 360.4 Hz (360.3 asked).
How it sounds on Kokoro:
- Up to ±4 semitones and range 0.6–1.5. Natural.
- Large shifts, or a long stretch. These buzz a little on the vowels: grains repeat.
- Stress. Clear but mechanical: a real actor stresses with timing and energy more than pitch.
- Emotion. It is a colour, not a performance. Kokoro reads "I hate you" the same way with
or without anger, and prosody bends that one reading. An expressive voice adapter behind the
same
TtsProviderinterface would do better.
Another take
takes.another voices the line's next delivery.variant and lists it in the booth with how it
reads ("+0.4 semitones, livelier, leaning toward bm_lewis").
- How it differs. A take nudges the reading:
- pitch ±0.7 st;
- range ±20%;
- pace ±6%;
- end ±1 st;
- and asks Kokoro for a blend leaning 12–25% toward a sibling voice (same accent and gender), which changes the delivery more than any of the nudges.
- Reproducible. It is seeded by the line's words and the take number, so take 3 is always the same take 3.
- Choosing. Choosing a take sets
delivery.variant(a change set, Ctrl+Z) and unpins any recording.
The voice picker
The character studio's Voice panel lists every voice by name, accent and gender
(voices.list). Its controls:
- Speed. The binding's own.
- Pitch.
pitchSemitones, every line up or down. - Timbre.
formant, -1 bigger and darker to 1 smaller and brighter: three semitones of formant at the same pitch.
Audition (voices.audition) voices the character's first line in the show (or a stock line)
as set, without changing anything. It is a take in the store, so hearing it again is free.
Use this voice saves the binding through revise_character.
Singing
What exists, locally and free
Checked on 2026-10-06/07 against each project's own pages; what we could not confirm is marked so. The test for "ship it": English, free, its weights and voice licensed to be redistributed in an installer and used commercially, small, and fast enough on a CPU for the web service.
| Option | English | Licence (code / voice) | Size | CPU | Ships? |
|---|---|---|---|---|---|
| DiffSinger (OpenVPI) via an ONNX runner | yes, per voicebank | Apache-2.0 / every English bank found forbids reupload or commercial use (Hoshino Hanami: no reupload; TIGER: CC BY-NC-ND; Peiton: paid); OpenVPI's NSF-HiFiGAN vocoder is CC BY-NC-SA | ~0.5 GB a bank + 57 MB vocoder | RTF 0.71 reported by a hand-written port; ONNX Runtime unverified | no: the person installs |
| NNSVS / ENUNU | no English model | MIT / pretrained models research-only | ~110 MB | unverified | no |
| UTAU Arpasing (Milk English) + a WORLD resampler | yes | redistribution with credit; commercial use needs the author | small | fast | no (commercial terms) |
| RVC v2 (convert the Kokoro reading) | yes | MIT, VCTK base / no permissively licensed singing voice exists | ~350 MB | RTF ~1.1 (speech) | no: no voice to ship |
| Seed-VC singing | yes | GPL-3.0, trained on CC BY-NC data | 821 MB | GPU | no |
| Vevo 1.5 / YingMusic-SVC / RIFT-SVC | yes | CC BY-NC (-ND, -SA) | GB | GPU | no |
| SoulX-Singer (2026) | yes | Apache-2.0; training data undisclosed | 2.8 GB | GPU-heavy; CPU unverified | optional install only |
| ACE-Step 1.5, YuE, DiffRhythm | yes | MIT / various | 10 GB | minutes a track | no melody control |
| VOICEVOX singing, Sinsy, NEUTRINO | no | embeddable / BSD + CC BY / no embedding | — | — | not English |
| WORLD vocoder, speech-to-singing (Saitou et al. 2007) | yes | modified-BSD (WORLD), MIT (pyworld) / the character's own Kokoro voice | ~1 MB | far faster than real time | yes |
No English singer that is free, redistributable, small and CPU-capable exists today. The one thing Reelwright can ship is singing made from the character's own speaking voice, done properly: Saitou et al.'s speech-to-singing (WASPAA 2007) on the WORLD vocoder. Their listening test (10 listeners, one Japanese phrase, hand-segmented) scored it 0.59 for naturalness against 0.64 for real singing and −1.3 for the speech. It keeps each character's timbre between speech and song, which no voicebank can. A real singer stays an add-on the person installs (a DiffSinger runner and their own voicebanks, SoulX-Singer, or RVC with voices they own); a shippable one would mean training our own on CC BY-SA singing (ACV-001) or a commissioned singer.
What is built
A ~ lyric is directed sung: {}. A sung line is one of three things:
- On an imported vocal (
sung.vocal: amusic.<slug>,fromSec,toSec).- The line's take is that stretch of the vocal, mono at 48 kHz.
- Its words are timed to the vocal's notes, by
alignLyric. Notes start where the pitch moves by more than 0.8 semitone or the voice stops. The lyric's syllables take the notes in order: notes merge where there are more of them, and the longest splits where there are fewer. - Rhubarb lip-syncs the take as it does any take, singing included.
- With
inMusic(the vocal is in a whole song already playing as music), the line is timed and lip-synced but not mixed twice. - Timed
withthe song's start cue, at an offset offromSecless the song's in point, the line is sung where the song sings it. - Measured: words land within 40 ms of their notes, and phrase bounds within 3%.
- Laying the lines on it (
edit_episodelay_lyrics; in Studio, Music and room tone…'s "Lay the sung lines on it"). A scene's sung lines take the vocal's stretches in order, between in and out points:phrasesfinds its phrases: singing between at least 0.35 s of quiet.- With more phrases than lines, neighbours are joined so each line's share of the singing is nearest its share of the syllables (least squares, by dynamic programming).
- With fewer, the longest phrase (0.6 s or more) is parted at its quietest 40 ms in its middle half, until there are enough.
- Each stretch reaches 0.1 s before and after its singing, never into a neighbour's.
- With
inMusic, each line is timed from the scene's start of that music, at its stretch's start less the music's in point. - It becomes one
set_sunga line, so it is proposed, reviewed and undone like them.
- Another take of such a line is refused (
take.vocal): its take is the recording. - Directed again, a lyric that still sings the same words keeps its vocal or melody.
- To a melody (
sung.melody: notesA3 C4 E4:2 rest:0.5, one a syllable, atbpm,transpose).- Kokoro speaks the lyric.
singPlanthen gives each syllable its note. A syllable is split from its word by the word's phonemes, and its vowel is the voiced run in it.- The vowel is held for the note's length; the consonants keep their own length, up to a share of the note.
- WORLD sings it at a singer's pitch (below, "Who sings"). TD-PSOLA, where WORLD cannot run, sings the note's pitch with a 60 ms glide into it and a 5.5 Hz, ±30 cent vibrato.
- Extra notes hold the last syllable across them, as a melisma. Extra syllables share the last note.
- Measured: the fake voice sang 220.0, 261.7 and 329.7 Hz for A3, C4 and E4, and the synthetic voice was within 2%.
- To a tune of its own (
sung: {}). One note a syllable, on a major pentatonic, around the note nearest where the voice speaks, at 96 bpm, the last note held two beats. It is seeded by the line, so it is the same tune every time.
Who sings: WORLD, with TD-PSOLA behind it
A line sung to a melody or a tune of its own is planned in the core and made by a renderer:
- The plan (
singPlan, prosody.ts): which moment of the spoken take plays when (each syllable's onset and coda as said, its vowel held for its note), the words as sung, the notes and the held vowels. - The pitch (
sungContour, sung-f0.ts), a singer's, after Saitou et al.: the melody in cents, each change overshot and settled by a second-order system (0.0348 rad/ms, damping 0.5422), the note before it first dipping away the same way run backwards (preparation: 0.0292 rad/ms, 0.6681), a 5.5 Hz vibrato of ±35 cents growing after 0.25 s on notes of 0.4 s or more with a ±20% loudness swing, and a fine flutter of a few cents, seeded by the take, 5 ms a frame. - WORLD (tools/sing-world/sing.py,
uv run, numpy and pyworld-prebuilt): Harvest, CheapTrick and D4C analyse the spoken take; its frames are read off in the plan's order and time, the voiced ones (and every held vowel) take the contour's pitch, held vowels get a singer's formant as the paper sets it (the spoken envelope's peak between 2.5 and 3.5 kHz lifted 12 dB under a raised cosine 2 kHz wide, the aperiodicity's dip there deepened as much) and the vibrato's loudness swing, and WORLD synthesizes it. The character's timbre moves (formant) by warping the envelope. - TD-PSOLA (singLine, prosody.ts) makes it when WORLD cannot run here: the voice service tries the singer once on a made-up note before the first sung line; with no uv, or no network for its first install, sung lines are TD-PSOLA's and the log says why. The singer's id is in a WORLD take's key, so the two never share a take.
How it sounds
Measured on the real Kokoro voice (af_bella), "Down at the Blue Grotto tonight" to G3 A3 B3 D4
B3 A3 G3:2 at 100 bpm, by node tools/sing-check.ts (tools/sing-check.py: 40 ms frames,
10 ms apart, voiced where the normalised autocorrelation passes 0.45; each note's pitch by
Harvest, the median over its held middle):
| spoken | TD-PSOLA | WORLD | |
|---|---|---|---|
| Each note's median pitch, cents from it | — | −9.0 to +9.0 | −1.0 to +4.6 |
| HNR, median | 11.2 dB | 21.9 dB | 22.7 dB |
| Jitter (10 ms frame to frame, on a note) | 1.41% | 0.23% | 0.25% |
| Shimmer (the same) | 9.6% | 2.1% | 2.9% |
| Spectral flux on a note (200 Hz to 5 kHz) | 7.3 dB | 6.3 dB | 6.1 dB |
| Singer's formant band (2.5–3.5 kHz against 0.3–2 kHz) | −21.4 dB | −23.4 dB | −15.6 dB |
| Spectral centroid, median | 325 Hz | 324 Hz | 394 Hz |
Both are in tune and on the beat; WORLD the closer. (The core's YIN tracker reads two of
WORLD's notes 24 and 73 cents off: the singer's formant rings strongly enough near 3 kHz to
pull it toward a partial. Harvest, and the spectrogram, put them in tune.) The spectrograms
show what the averages hide: TD-PSOLA's held vowels repeat grains of a short spoken vowel, so
their upper harmonics break into steps and shimmer, and its transitions are straight glides;
WORLD's harmonics run continuously through a held vowel, every note change overshoots and
settles as a voice does, and the singer's formant lights the band near 3 kHz. The numbers agree
in direction (higher HNR, lower flux, +7.8 dB in the singer's formant band) though HNR and flux
by a little. At the paper's 12 dB the formant is bright on Kokoro's female voices;
SINGER_FORMANT.db (world-sing.ts) is the place to soften it.
It is still speech made to sing, not a singer: the timbre is Kokoro's speaking voice, a held vowel is a spoken one stretched four to eight times and so steadier than a singer's, and the consonants are the spoken ones. It reads as a character singing; for a song that has to sound good, import a recorded vocal and lay the lines on it.
Delivery
Previews
A preview is now a proxy:
- Video. x264 CRF 28 at preset veryfast, capped at 3 Mb/s for a 1280×720 picture (scaled by area for other sizes), with twice that as the buffer.
- Sound. 128 kb/s AAC.
- Size. It is the show's own picture scaled by the largest simple fraction whose size is
even both ways and fits a 16:9 frame 720 lines high, or 540 (
format.previewHeight). So 1920×1080 previews at 1280×720 (960×540 at 540), 1080×1920 at 720×1280, and 1920×804 at 1280×536.
A full render is still the master: CRF 16, slow, no cap.
Aspect ratios and frame rates
Cameras are solved for the show's own aspect, and the engine draws any even size.
Aspect ratios. Settings offers 16:9, 9:16, 1:1, 4:3 and 2.39:1, with sizes for each.
Frame rates.
format.fpstakes 23.976, 29.97 and 59.94 by name. Everything that turns frames into time uses the exact rate (core/src/compile/rate.ts):- the Timeline's
format.fpsis 24000/1001 as a decimal; - ffmpeg is given the fraction;
- the encode is checked to be
24000/1001.
A 23.976 frame is exactly 2002 samples at 48 kHz, so 22 minutes of sound stay on their pictures to the sample.
- the Timeline's
Another rate on request.
--fpsoncompile,render(with--deliver),stillsandexport otioresolves the episode at any rate show.json offers, named (25, 29.97) or as a fraction (30000/1001). This serves Cutwright's bridge (R4): a 24 fps show cut into a 25 fps project, or a clip cut for 29.97.- EpisodeIR timing is in seconds, so every cue keeps its time to within a frame of the new rate. A line keeps its whole take, and its mouth moves with the sound at the new rate.
- Only a rate other than the show's goes into
inputsHash. - Its frames are drawn in their own folder (
…-25fps).
Movie Maker. Capture steps at the whole rate. The frames are drawn by number either way.
Deliverables
A full render makes the show's format.deliverables, or what --deliver, renders.start deliverables or Studio's Export… asks for (core/src/render/deliver.ts):
| Preset | What |
|---|---|
master_prores_422hq / master_prores_4444 |
QuickTime .mov: prores_ks profile 3 (yuv422p10le) or 4 (yuv444p10le; yuva444p10le with alpha), BT.709, 24-bit PCM sound |
png_frames |
the verified frames, hard-linked, numbered from the range's first |
exr_frames |
OpenEXR written by the core (exr.ts): the sRGB curve undone into linear half floats, B, G, R (and A first, with alpha), ZIP-compressed in blocks of 16 lines. ffmpeg 6.1's EXR encoder stores PNG values unchanged. The frames hold the 8-bit render's precision, linear |
stems_wav |
dialogue, music, effects and foley as 24-bit WAV at the mix's linear gain, so they sum to the mix but for the master limiter's peaks |
captions_srt, captions_vtt, otio |
as before, now listed with the render |
render.json and renders.list give each with its size, and Studio lists them under the render
with Open or Show in folder.
Alpha
reelwright render --alpha (and stills --alpha) draws RGBA frames, transparent where
nothing is drawn: a see-through stage (set.schema.json stage.seeThrough, SetInstance 1.3)
for characters over footage. It asks for output.alpha in the jobs, keeps its frames apart
(<size>-frame_server-alpha) and says "alpha": true in render.json. Movie Maker captures an
opaque window, so --alpha needs the frame server.
The PNGs hold straight alpha, the color of a soft edge at full strength and its coverage in alpha, as a PNG is read. Getting there:
- Godot blends into a transparent target by mix, so what a transparent viewport holds is premultiplied: a half-covered red pixel reads back as (128, 0, 0, 128), and a white glyph's soft edge as (46, 46, 46, 46). Saved as they were, every soft edge would come out up to twice as dark.
- Godot also converts a transparent 3D picture to sRGB without dividing its alpha out first,
so a translucent pixel in an 8-bit view has the wrong color under it (a 50% grey quad read
(92, 92, 92, 128), which divides out to 183, not 128). A transparent shot view therefore
draws in linear light (
use_hdr_2d), half floats, linear and premultiplied. - The post pass (
rw_post.gdshader) keeps premultiplied color (blend_premul_alpha) through sampling, transitions and bleed, divides alpha out in linear, encodes sRGB, and grades the straight color. Fades, the CRT's edge and a faded-out previous shot thin alpha instead of going to black. It writes premultiplied color again. - The frame server un-premultiplies once, as its last pass (
rw_straight_alpha.gdshader). - Additive effects (sparks, glows) say their light in alpha too: over nothing, a sprite covers as much as its brightest channel adds, so laid over black in linear light the frame adds what it adds over a set.
Laid over black in linear light, an alpha frame of the see-through swatch matches the same frame drawn opaque to within a level at its soft edges (median 1, p99 9 at the sparks) and exactly elsewhere. Over footage in an editor that blends in sRGB, as most do, soft edges come out a little darker than Godot's own linear antialiasing would make them; that is the PNG's convention, not a loss.
Each delivery keeps the alpha where the format can:
| Preset | With --alpha |
|---|---|
png_frames |
straight RGBA, as the engine wrote them |
exr_frames |
A, B, G, R: linear color premultiplied by alpha, as EXR stores it |
master_prores_4444 |
yuva444p10le (stored 12-bit): straight alpha, as QuickTime stores it |
master_prores_422hq and the MP4 |
no alpha plane: the picture laid over black (premultiply) |
The alpha golden (tools/render-alpha-golden.sh, engine/tests/goldens/alpha_640x360) is the
style swatch's scene on a see-through stage, compared in CI with every channel at tolerance 0.
Frame export
A render that delivers png_frames also writes <out>/frame-export.json, a FrameExport 1.0
(schemas/v1/frame-export.schema.json, core/src/render/frame-export.ts): what another program
needs to take the frames and stems in and trust them. Cutwright, which cuts Reelwright renders
into footage, asked for it (its R1).
| Field | What |
|---|---|
release |
the Reelwright that compiled and rendered it (name, version) |
engine |
the RenderResult's engine: Reelwright, Godot, rendering method and driver, adapter (the first chunk's) |
episode, timeline |
the Timeline's id, schemaVersion, inputsHash, frameCount and exact fps |
range, size, alpha |
the Timeline frames [start, end), the picture's size, and whether the PNGs are straight RGBA |
frames |
every frame of the range in order: its Timeline number, its file (frames/png/<slug>_000000.png for the first) and its SHA-256, the one the engine reported and the render checked |
audio |
with stems_wav: the sample rate, the range's first sample and length, the mix's gain (already applied), and each stem's role (dialogue, music, sfx, and foley from FrameExport 1.1, when the render has foley), file and SHA-256 |
Files are named relative to the document's folder. A render asked only for frames and stems
(--deliver with nothing but png_frames, exr_frames and stems_wav) makes no H.264: its
render.json has no video, and the CLI prints ✔ …/frame-export.json: 12 frames 1920x1080 with straight alpha, 4 stems, no H.264. Studio lists only renders with a picture to play.
What is not done
- Expressive voice. Emotion is prosody on one reading, not an actor's performance. An expressive TTS adapter (cloud or a local model with style control) is the next step.
- A real singer. Sung lines are the character's speech sung through WORLD; there is no singing-voice model, because none in English can be shipped (see "What exists"). Next: a DiffSinger runner (an ONNX Runtime port of OpenUtau's MIT pipeline) for voicebanks the person installs and accepts the terms of, without bundling OpenVPI's NC vocoder; and, to ship a singer, one trained on CC BY-SA singing (ACV-001) or a commissioned voice.
- Consonants in song. Saitou et al. lengthen consonants by their class (a fricative 1.58 times, a nasal 1.77); the plan keeps them as spoken, up to a share of the note.
- Duck and swell in the timeline. Duck and swell cues are made in the script, by chat or
by
edit_episode. The timeline has no Add for them (the timeline's Add is its own work). - A real singer. There is no singing-voice synthesis model. A DiffSinger adapter, with voicebanks the person installs, is the path.
- Music and room tone in the timeline.
- Duck and swell cues are made in the script, by chat or by
edit_episode. The timeline has no Add for them. - Room tone is set per scene from Music and room tone…, or per set or scene by
set_ambienceor the file. The set explorer has no field for a set's own.
- Duck and swell cues are made in the script, by chat or by
- Titles over footage. A title card is opaque, with or without alpha.
- Projected shadows on a see-through stage. A style whose key casts shadows (groundShadow projected) has no floor to cast them on there: they fall only on the elements it draws. Blob shadows draw.
- Glow over nothing. Bloom adds light around bright surfaces but not alpha, so its halo over a transparent part of the frame is lost.
- EXR depth. EXR frames are linear but 8-bit deep. A float render path in the engine (HDR 2D viewports) would make them more than a container change.