Frames and interface composition

This document answers what a Frame is, how its controls bind to a running instance, which widget kinds exist, and what every host must do when it mounts one. A Frame is a declarative presentation of a definition in a running-instance context. It is the authored app interface, not a screenshot of the graph. The editor shows each Frame on the Patch as a card holding its components, which move and resize in place, where most Toys are designed, with a derived binding link from each control to the node parameter it drives (patch editor); runtime presentation reads an explicit widget list.

Status: the shared React renderer, every widget kind below, nested and inline Frames, themes/styles, responsive projection and the host interface are implemented in packages/frame-runtime and used by the browser editor, published playback and the native Player WebView. Node faces (face, paint, curve, switcher, readout) and live video Screens bind through optional host hooks; the browser implements them, and hosts without a video engine or authoring (the native Player WebView, exports) show honest unavailable states (video, MIDI, control and mixing). General event actions remain planned. Physical touch, screen-reader, embedded WebView and DAW qualification remain open.

One renderer, every host#

The renderer is a React package with no dependency on the graph editor or account shell. Browser pages, a JUCE WebView and later Electron embed that same implementation. Framework-neutral binding/interaction contracts let another shell mount it without a second renderer. Legacy Svelte/React faces supply behavior and fixtures to port deliberately; do not ship two independent implementations of the same production widget.

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FAFAF7', 'primaryTextColor': '#1C1C1B', 'primaryBorderColor': '#1C1C1B', 'lineColor': '#64645F', 'fontFamily': 'DM Sans, Inter, system-ui, sans-serif'}}}%%
flowchart LR
  subgraph TOY["Toy source"]
    DEF["Definition<br/>public parameters · nodes"]
    FRAME["Frame<br/>widgets · bounds · theme"]
    FRAME -- "binds parameter IDs of" --> DEF
  end
  subgraph HOST["Host (browser · Player · export)"]
    FH["FrameHost<br/>instance path · live values · gestures"]
    INST["Running instance<br/>root / stage-1 / voice-2"]
    FH <--> INST
  end
  RENDER["Shared renderer<br/>FrameView"]
  FRAME --> RENDER
  DEF --> RENDER
  RENDER <--> FH

  classDef source fill:#F0442E,stroke:#1C1C1B,color:#FAFAF7
  classDef shared fill:#DCEBDD,stroke:#1C1C1B,color:#1C1C1B
  classDef host fill:#DED8F0,stroke:#1C1C1B,color:#1C1C1B
  class DEF,FRAME source
  class RENDER shared
  class FH,INST host
  style TOY fill:#FAFAF7,stroke:#1C1C1B,color:#1C1C1B
  style HOST fill:#FAFAF7,stroke:#1C1C1B,color:#1C1C1B

Bindings and nesting#

Controls bind to stable public parameter IDs relative to the owning definition. At runtime the host supplies an instance path such as root / first-stage / voice-2, and a reused Frame resolves its relative bindings in that context. A child frame widget names a group node instance and a Frame belonging to that node's definition; an inline frame widget omits the node and references a Frame of the same definition (see performance and reuse). Reusing one definition twice yields two controls and independent state.

Frames have stable IDs, dimensions, explicit member/widget IDs, labels, bounds and an entry point. Loose interface components may also live in Definition.widgets at absolute Patch coordinates; explicit Frame.widgets membership selects the published interface. Graph-node canvas coordinates are editor metadata; moving an unrelated graph node cannot add it to a published Frame. Publishing includes the complete processing dependency closure even when most nodes are not visible. Frames use bounded reference geometry with optional portable anchors and limits; see responsive Frames.

Widget kinds#

The schema admits these widget kinds. Numeric controls and graphical roles bind public IDs; observations use the shared host interface.

Arbitrary HTML, CSS injection, JavaScript, remote iframes and executable widget bundles are not admitted by the draft schema. General event actions remain planned.

Shared node faces#

Node cards, inspectors and Frame face widgets use the same face kit. Pure scene models produce SVG geometry and accessible handles; the shared gesture grammar freezes axes during a drag, commits once per gesture and supports keyboard edits, reset and cancellation. FACE_BINDINGS defines each face's roles and compatible processor, with generated Rust admission matching TypeScript. Data faces additionally name a node; a host with authoring can persist a data edit through commitNodeData as one Undo step.

The Wavetable face includes waveform selection, + Add (a copy of the selected waveform), Remove (keeping one) and the shared pencil/line/smooth/preset editor. Editing selection is local to each surface and independent of playback Position. Changes carry the whole 1–8-frame data.table through the existing live preview and atomic authoring bridge; rejected edits restore the previous table, and Undo/Redo supersedes an unfinished stroke. Fixed-frame curve widgets use the same sample editor.

The portable theme tokens screen, screenTrace and screenGlow override the shared glass, trace and glow defaults. A single scheduler updates visible live layers outside React. Dense spectrum pixels use Canvas below the SVG; zoom and reduced motion limit refresh rates. The engine exposes one signal trace, so the active inspection or face claims it by priority. Polyphonic playheads read each engine voice’s stage, elapsed time and gain; Chorus dots read its actual modulation phase. These optional host observations never enter portable source or DAW state. Unsupported engines omit the moving marks.

The Bitcrusher (std.crush, face crush) draws a quiet 1 kHz sine and its quantized, held output over 4 ms, including Mix. Two right-hand rails edit Bits and Rate through the shared gesture lifecycle. It is a nominal-clock example without jitter or dither, not a live signal trace. Its portable roles are bits, rate_hz and mix; the signed grid and sample holds are checked against shared-engine PCM.

Interaction contract#

A control sends begin-gesture, parameter changes and end-gesture through the host interface. It displays authored default, live value, automation/modulation ownership and unavailable/read-only state distinctly. Repaint is suspended when hidden; input receipt and note release never depend on animation frames. Outbound observations are coalesced. DSP never waits for rendering.

Pointer cancellation, lost focus, editor close, disposal and disconnected sources release held notes and momentary gates according to source ownership. A widget never releases another input source's notes. Keyboard navigation, accessible names, visible focus, value readouts, reduced-motion behavior, touch sizing and resize behavior belong in widget acceptance fixtures.

A plugin embeds its UI resources for offline use. A hosted page gets only the Frame, the runtime adapter, the verified release and required assets. Host-level start, input-selection and error controls may surround it; creator branding and free-host branding are separate from the Frame's authored processing.

Non-audio applications#

Camera, display, media and sensing nodes run as host and video domains beside the audio engine and reach Frames through live Screens, paint surfaces and lane readouts; their requirements are derived, and control production stays outside audio rendering (signal types and domains, video, sensing). The browser runs them; the native Player WebView and exports show the faces as unavailable, and a native host does not yet report unavailable domains before activation. General widgets and service actions, remote-system control with remote mode and connection status, and IoT services remain planned.

Implementation#

• packages/frame-runtime — FrameView, ParameterControl, theme resolution, responsive projection, icons
• packages/runtime-protocol — the host interface Frames talk to
• apps/web — Frame Design editor and published playback host
• native/plugin-shell — embedded WebView host

Related#

• Appearance and Frame design, responsive Frames, interface components
• Toy source format and running-instance protocol
• Decision 0003: Frames and authoring