Contents

Previews drawn in the person's own browser

The web service draws every render in software on two CPUs: the cold open's 28-second preview took 17.5 minutes on four of the dev box's CPUs, and render workers only divide that by how many there are, at a price (render-capacity.md). Most people open web Studio on a machine with a GPU, and Studio already plays an episode on it, in WebGL 2. Render capacity's section 5 measured how close that picture is to the render and recommended this: previews drawn in the browser that asks for them, final renders where they are.

This is what was built, in order:

1. One picture. The browser's renderer (Godot's Compatibility, WebGL 2) lit surfaces under a coloured light greener than the render does. Fixed in the engine, so the web player, a browser-drawn preview and the render agree (below, "One picture").
2. Previews drawn in the browser. Web Studio's where-to-render dialog offers "On this computer, in your browser"; the page draws the preview's frames offscreen at its size, encodes them with WebCodecs and sends them up; the service mixes the sound and joins them as it joins any preview, and the preview lands where a service preview does.
3. What it takes, measured. Godot's Compatibility renderer on the dev box's GPU as the stand-in for a laptop's, and SwiftShader in Chromium as the floor; the dialog's time for "in your browser" comes from them.

1. One picture

Render capacity's comparison (render-capacity.md, 5.1) found one real difference between the web player and the render, and it was the renderer's, not the browser's: under a coloured light (the console's amber practical, the window's blue), surfaces came out greener in Compatibility.

Why. Compatibility draws a shadow-casting light, the key, in a pass of its own, added to the picture after the base pass. The cel shader works in linear light but the picture it writes is sRGB, so its key pass adds the difference the key makes, with_key - without, both turned to sRGB. It took "without" to be the surface's own light (v_own) alone. But the practicals' pools are drawn in the base pass, so a lit surface already had them: the key pass computed its sRGB difference from too dark a start, where the curve is steep, and added too much of the key there. The excess showed mostly in green: +12 levels at the console's top, against +3 in red and +5 in blue.

The fix (engine only; Forward+ untouched). The key pass now knows the pools: RwSet shares each lit practical's place, reach, light and falloff as the project's global shader parameters (rw_pool_*, at most eight, only under Compatibility), and the cel shader's key pass (rw_pools, rw_color.gdshaderinc) adds them to what the surface had, as Godot's own omni light falls off ((1 - (d/range)^4)^2 * d^-attenuation) with the cel wrap. One thing had to be measured: what Godot hands light() as a light's colour. It is the colour times the energy times pi (LIGHT_COLOR), and the pools must be the same.

Compatibility stills (llvmpipe, the web export's renderer on the desktop) against Forward+ (lavapipe, the render), frames 300 and 500 at 962x540, mean difference over the whole frame and in the two places it was worst, in levels of 255 (Compatibility minus Forward+, red, green, blue):

The green is gone; what is left is a level or two either way, mostly the outlines a pixel apart that render capacity already found. Forward+ is the same pixel for pixel: the goldens are unchanged (Proof).

The pools are global, so two sets drawn in the same frame of one page would share one set's practicals. Studio draws one at a time in practice (the player, or a preview being drawn); it is written down under "Not yet".

2. A preview drawn in the browser

The service's side (core/src/studio/browser-draws.ts, session protocol 1.10)

• Asking. renders.start with on: "browser" (a preview only; a render is refused in words) puts the preview in the show's queue like any other. The daemon remembers which socket asked: that Studio, and no other, draws it. On the service the host lets it through to the daemon (never to a runner), and the month's and the day's render minutes do not stop it: it uses none.
• Everything but the drawing. It is checked, voiced and compiled as any preview is, its sound mixed, and it never joins the service's line (render-line.ts): its one engine job goes to the asking Studio as the render.draw notification: the job's file (as episode.playback names one, read with the files it names), its frames, its size and frame rate.
• The stream back. The page sends H.264 (Annex B, the Baseline profile: no frame out of order) in pieces of at most 700,000 base64 characters (renders.upload, each at the offset of the bytes before it, saying the frames done so far), then renders.drawn with what drew it (renderer, driver, adapter). A piece from another socket, or out of order, is refused.
• Joined as a farm's part is. The daemon wraps the stream as the job's segment without encoding it again (wrapArgs: timestamps from the frame rate; the colour said as a render's, BT.709, limited range), writes the job's result (every frame, the engine that drew them, an info note that they came from a browser), and the pipeline joins it with the mix as it joins a render worker's segment (pipeline.ts segments). The preview is a version like any other: renders/<episode>/v<n>/, its render.json saying it was drawn with gl_compatibility.
• Progress is what the page says it has drawn (EngineRunner.drawnOf): the render's events carry on: "browser" and the frames as they come, so every tab on the show sees it.
• When it cannot. The page says why (renders.drawn with ok: false): the preview fails with its words. Its socket closes (the tab shut): it fails, "The browser tab drawing this preview closed before it was done". It sends nothing for the job's quiet time: it fails as a stalled engine does. Nothing is drawn on the service in its place: the person asks again, here or there.
• Billing. Drawing in the browser is no render minute (jobs.ts): only what the service does (checking, compiling, mixing, joining) is counted.

Studio's side

• The question. Render preview on the web asks where (RwRenderWhere, from render-capacity.md). A third place sits under the service and the person's own computers: On this computer, in your browser, with its time for this browser (section 3: the page asks WebGL what it draws with, and a software renderer gets the software time, "drawing without a graphics card"), and "keep it open until it's done. Uses none of your render minutes." When even its shortest time is longer than the service's longest, it adds "Here on Reelwright is faster." and the service stays picked: a browser with no graphics card takes hours where the service takes minutes. It can still be chosen. It is offered for a preview only, and only when the service says it takes a browser's frames (the hello's browser-renders).
• Can this browser? Before offering it, the dialog asks the page (RwBrowserDraw.refusal): is Studio drawing with WebGL 2, and does WebCodecs' VideoEncoder say it can encode H.264 Baseline at the preview's size (isConfigSupported)? When not, the option is shown turned off, with the reason in words ("This browser cannot encode video (it has no WebCodecs)."). Nothing is asked of the service until the person starts.
• Drawing (ui/render/rw_browser_draw.gd). On render.draw the page reads the job over the service (RwFiles, as the player reads its own) and the files it names into the page's memory, then draws it with exactly what the command line's render draws with: the job's documents, RwTimelinePlayer and RwFrameServer, an offscreen viewport at the job's size. The frame server's last pass turns each frame into YUV 4:2:0 on the GPU, as a render's picture is (BT.709, limited range, each chroma sample the mean of its 2x2 pixels: runtime/render/rw_i420.gdshader), packed as the bytes of an I420 frame into a target a quarter its width and one and a half times its height; read back, that is the frame's I420, half the bytes of its RGB, and it goes to the page's encoder as it is (ui/render/rw_browser_encoder.gd, a small script on the page). A frame whose width is not a multiple of four is read back as RGB and turned into YUV by the script, as before. Then it is encoded by VideoEncoder (avc1.42001f, Baseline level 3.1, to 720p; avc1.42002a above; 3 Mbit/s as the service's preview cap; a keyframe every two seconds; Annex B), the encoded bytes taken back as they come and sent up in pieces of 500,000 bytes. Studio waits while the encoder holds more than six frames, so a slow encoder never fills the tab's memory. At the end it flushes the encoder and sends renders.drawn with this page's engine, renderer and adapter.
• One at a time. A second preview asked of the same tab waits for the first. A preview cancelled or failed meanwhile is drawn no further, and the service hears nothing more of it.
• The tab. While it draws, the render's progress says "in this tab: keep it open", and the page asks before it closes, as it does for an unsaved script. If it closes anyway, the service fails the preview in words (above).

Sound: the service's mix, always

The lead's brief said: encode the sound in the browser with WebCodecs' AudioEncoder where it exists, else send the picture alone and let the service mux it with its mix. This sends the picture alone always, and the service muxes it with the mix it has already made, the picture copied, not encoded again. Why:

• The service makes the preview's mix whatever draws the picture: the voices, the sound effects and the music levelled to the show's loudness. It is there before the first frame is drawn. The page has no mix of its own to encode: it would have to fetch the service's, decode it and encode it again, which is more work, more bytes over the person's connection and a second lossy pass, for the same sound.
• One path, not two: AudioEncoder's codecs differ by browser (Firefox encodes Opus, not AAC; Safari's support is new), and each would need its own mux and its own test.
• Joining them is what every preview's last step already does: the picture copied, the mix encoded once as AAC. Seconds on the service.

If the lead still wants the browser's AudioEncoder (for a future where the page mixes on its own, offline), it fits beside this without changing the protocol: renders.upload would carry a second stream.

3. What it takes

Browsers here may only draw on SwiftShader (a GPU page fault took the desktop down twice), so the two ends are measured two ways, and both from the page's own loop: RwBrowserDraw times each part of every frame and says so on the page's console (RwBrowserDraw: {...}, its last_cost), and tools/browser-draw-rate.sh runs that same loop natively, with the encoder and the service stood in for, on whatever renderer it is given.

• No graphics card (the floor): the end-to-end check (tools/web/browser-render-check.ts): Chromium on SwiftShader, pinned to four of the box's CPUs, drawing the cut cold open's 155 frames at 960x540 for the service.
• A graphics card: Godot's Compatibility renderer on the box's RX 6700 XT, through Mesa's Zink on a private Xvfb (GALLIUM_DRIVER=zink MESA_VK_WSI_DEBUG=sw, and no LIBGL_ALWAYS_SOFTWARE, which makes Zink pick a CPU device), under the GPU lease. The adapter is checked before anything is timed: "zink Vulkan 1.4(AMD Radeon RX 6700 XT (RADV NAVI22))". The box's godot wrapper otherwise draws on llvmpipe whatever is asked.

Where a frame's time goes

Seconds a frame:

The three Chromium runs drew the same 155 frames, the box less busy each time.

What that says:

• In software, drawing is the time: 86% of a frame is SwiftShader rasterising the scene (the frame, and Studio's own window once more, since a forced draw draws every viewport). Nothing else comes close.
• With a graphics card, the GPU is not the limit. It draws a 720p frame in about 30 ms. The engine's own work a frame (setting the stage: 36 ms) is as much again, and it costs the same in the page's WebAssembly as natively (74 ms on four loaded CPUs in Chromium, 36 to 81 ms natively across runs): the page is not slower at it. In a browser, the page's own work then comes on top: turning the frame into YUV and base64 in JavaScript (132 ms at 960x540, scaling with the pixels) and sending it (34 ms). That was the largest part of a frame drawn on a graphics card; the YUV is now made on the GPU, before the read back, and the handing halved (68 to 33 ms in paired runs: "YUV on the GPU" under Proof).
• WebCodecs is not the limit either: Studio never waited for the encoder (0.4 ms a frame over the run, the flush included); Chromium's encoder keeps up on its own threads.
• One page frame, not two. The frame server waited for two of the tree's frames before it drew, so evaluate()'s deferred redraws and transforms reach the renderer (rw_frame_server.gd). Natively those are 2 to 5 ms; in a browser each is a display frame, at most 1/60 s with a graphics card, and in Chromium on SwiftShader a whole redraw of Studio's window (0.13 to 0.18 s each). Godot flushes both right after it emits process_frame, so a frame asked for straight after the last one (a render's frames, one after another) is evaluated before the flush and needs only the next of the tree's frames. The frame server now waits for one then, and for two otherwise: the first frame, and one asked for after its caller waited on something else, may be evaluated after the flush (from a script's start, or a page's callback), when one wait would draw it stale. It knows which by a flag set as its wait ends and cleared by a deferred call, which runs at that flush ("One page frame, not two" under Proof).

The first native run on the GPU took 0.177 s a frame at 720p (the box at load 40, its shader cache cold); later runs 0.077 and 0.093 s. The estimate takes the slower warm one.

The estimate

estimateBrowser (core/src/render/estimate.ts), the service's estimate's browser, from those parts:

• With a graphics card: 0.11 s a frame (the engine's 0.093 natively, plus the page frame it waits for at 60 a second), plus the page's 0.11 s a megapixel.
• Without one: 0.18 s a frame (setting the stage and Studio's window once, in software: the fastest run's, its second window's redraw taken out) plus 5.4 s a megapixel (rasterising), plus the page's 0.11 s a megapixel: the fastest run, the other two inside the range.
• Each a range: the dev box's time, to twice it, for a laptop's slower GPU and CPU. The box cannot measure one; twice is an allowance, said as one, not a measurement.
• Which one the person sees: the page asks WebGL what it draws with (WEBGL_debug_renderer_info). A software renderer (SwiftShader, llvmpipe, Microsoft Basic Render) gets the software time, with "drawing without a graphics card"; anything else, or a browser that will not say, the graphics card's.

For the demo's whole cold open (674 frames, its 1280x720 preview): about 3 to 6 minutes with a graphics card, against about 21 on the service's two CPUs and about a minute on the box's own GPU as a runner; and about 1 h to 2 h in software. The cut cold open the check draws (155 frames at 960x540) is said as about 9 to 17 minutes in software; the last run was done in 8 min 48 s. A browser with no graphics card is told so, and the service is the better choice for it: the dialog says both times side by side.

Proof

One picture. tools/web/player-frames-check.ts and player-frames-compare.py (as in render-capacity.md 5.1), the eight frames of the cold open, before (e1df0039) and after:

Frame 500, the worst before, went from 6.1 levels and 20.5% visibly different to 4.2 and 3.2%. What is left between the web player and the render is what was there between Compatibility on the desktop and in the browser all along: the outlines a pixel apart and the paper grain. Forward+ did not move: the Lighthouse stills (9), the style swatches (2), the alpha golden and the sitcom places (24) are every pixel the same (tools/check-all.sh --goldens' four steps).

YUV on the GPU. The frame server's I420 against its RGB turned into YUV by the same sums (tests/render/rw_frame_i420_test.gd on Forward+; tests/support/rw_i420_check.gd on the Compatibility renderer the browser draws with, on llvmpipe and on the RX 6700 XT through Zink): every byte of every plane within one level. The end-to-end check on the base and with the change, side by side under the same load: the page's handing a frame to the encoder went from 68 to 33 ms (its base64 halved and no YUV in JavaScript); frame 100 of each against the service's still, 4.493 and 4.493 levels, 30.4 dB both; the two previews' frame 100 against each other, 1.3 levels (40.7 dB), what H.264 makes of inputs a level apart.

One page frame, not two (browser-preview-speed). The frame server's change against the base it was made on, d2907e656:

• The same pictures. On lavapipe (Forward+), the 9 golden stills, the 2 style swatches, the alpha golden and the 24 sitcom places every pixel the same as their goldens; frames 24 to 143 of the cold open drawn one after another and the 9 golden frames, all 129 the same bytes on the base and with the change. In the browser (below), the base's preview against itself run to run and against the change's: 44.5 dB and 44.0 dB run to run, 44.1 to 44.6 dB base against change. WebCodecs on SwiftShader does not make the same bytes twice, so that is the floor; the change moves nothing above it.
• One page frame a frame. On the box's RX 6700 XT through Zink, natively, the cold open's frames 24 to 143 as the page draws them: the page frames waited for over the 120 frames went from 0.60 and 1.18 s to 0.32 and 0.29 s (each 2 to 5 ms natively; all of a frame 0.105 to 0.140 s either way, the box's load deciding). In Chromium on SwiftShader, the cut cold open end to end, base, change, base, change: 3.33, 4.07, 4.11 and 3.85 s a frame, the waits for the page 338, 318, 367 and 196 ms of them. The box's load (21 to 36) moves a frame more than the half second the change saves at most, so the runs say the waits halved, not the total.
• The guard, broken three ways (always one wait, never the window open, never closed), each failed rw_frame_server_test.gd's "a frame asked for straight after the last waits one tree frame".

A preview drawn in the browser, end to end (tools/web/browser-render-check.ts, on the web export of the final engine): the service with two CPUs on lavapipe, Chromium on SwiftShader on four, the cut cold open at 960x540.

• The dialog, as a browser with a graphics card sees it (its WebGL told it draws on one) and as this one does: the graphics card's time; and the software time, "drawing without a graphics card", with "Here on Reelwright is faster." and the service picked.

• Picked and started: the service asked that page alone to draw (render.draw), at 960x540; the page drew all 155 frames, sent them up in 9 pieces in order and said it was done: 616 s, 3.97 s a frame (3.21 s the run before, with the box less busy).
• The version: an MP4 of H.264 Constrained Baseline (the page's own, not encoded again), 155 frames, and the service's mix as AAC; render.json says gl_compatibility and render.drawn_in_browser; it is v1 of the cold open.
• The review opens v1 at frame 100 in a tab of its own, its sound joins, and Space plays it on to its end, the review still on screen.

• Frame 100 decoded from it against the service's own still of that frame (Forward+ on lavapipe): 4.5 levels a channel, 3.9% of pixels visibly different, 30.4 dB; against the other shot's frame 30, 42.8 levels.

Tests and mutations.

• Core: host-browser-draw.test.ts (drawn in the asking browser alone, the pieces in order, landed as a version with its mix, not billed; minutes used up do not stop it, a full render is refused; a closed tab fails it in words; the page's own failure said), render-estimate.test.ts, host-render-line.test.ts (the estimate's browser times, previews only; a browser preview never goes to a computer of the person's), renders.test.ts (a desktop Studio refuses it), session-fixtures.test.ts (render.draw, renders.upload and renders.drawn recorded, engine/tests/fixtures/session/browser-draw.json). The whole core suite in two shards.
• Engine: rw_browser_draw_test.gd (frames drawn at the job's size and handed over in order, the pieces at their offsets, renders.drawn with what drew them; a frame the encoder refuses said in words; a preview cancelled meanwhile drawn no further; a software renderer known by its name), rw_render_where_test.gd (the browser's time, its own for a browser without a graphics card, the service said to be faster then and kept picked, the option off with the reason, none for a full render or a service that cannot), rw_render_queue_test.gd (keep the tab open), rw_set_pools_test.gd (the pools the shader is given).
• Each guarantee broken once and its test seen to fail: 11 in core (the pieces' order, not billed, minutes not stopping it, a closed tab, preview only, the desktop refusing, never to a runner, the estimate for previews only, software's own range, a slower machine's high end, the stream at the job's frame rate) and 12 in the engine (a cancelled preview stops, the pieces' offsets, an encoder's refusal said, a software renderer known, the browser's answer, the option off when it cannot, software's own time, the service said to be faster, keep the tab open, hidden and dark lamps left out, no more than eight, the lamps' light times pi).

Not yet

• Final renders are never drawn in a browser: a master is the show's own look, frame for frame, and an hour in a tab is fragile.
• The pools are the project's global shader parameters: a page playing one episode while it draws another's preview would light both with whichever set was evaluated last. Per-instance uniforms would separate them, at the cost of setting them on every material of a stage.