Authoring for people and agents

How do the graph editor, the Frame designer, the in-app prompt panel and later CLI/MCP adapters change a Toy without bypassing one another? Every author, human or model, goes throu

How do the graph editor, the Frame designer, the in-app prompt panel and later CLI/MCP adapters change a Toy without bypassing one another? Every author, human or model, goes through packages/authoring: the same node catalog, the same structured operations, the same revision checks. No model-specific JSON becomes the stored Toy representation.

Status: implemented locally — typed operations and history, manual candidate JSON, optional server-side Claude text generation, compilation, audition, acceptance and Undo. Richer domain-level explanations, durable server history, serialized inverse operations and an MCP adapter remain planned. Evidence: recorded live Claude proposal journey and implementation status.

Proposal lifecycle#

A user asks for a complete Toy, a node group, or a Frame that controls existing parameters. The agent inspects a bounded selection, the current revision, public interfaces, available nodes, target limits and relevant examples. It returns an ordinary candidate operation batch, a concise explanation, requested capabilities and validation diagnostics. Acceptance and publication stay separate actions.

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FAFAF7', 'primaryTextColor': '#1C1C1B', 'primaryBorderColor': '#1C1C1B', 'lineColor': '#64645F', 'actorBkg': '#DCEBDD', 'actorBorder': '#1C1C1B', 'signalColor': '#1C1C1B', 'noteBkgColor': '#DED8F0', 'noteBorderColor': '#1C1C1B', 'fontFamily': 'DM Sans, Inter, system-ui, sans-serif'}}}%%
sequenceDiagram
  participant User
  participant App as Editor / prompt panel
  participant Agent as Agent adapter (API)
  participant Authoring as packages/authoring
  participant Host as Runtime host

  User->>App: prompt or manual candidate JSON
  App->>Agent: bounded selection, revision, catalog, limits
  Agent-->>App: candidate operation batch + explanation + diagnostics
  App->>Authoring: validate against base revision
  Note over App: before/after source diff, separate from the model's explanation
  App->>Host: optional bounded audition in a separate generation
  User->>App: accept
  App->>Authoring: atomic revision with inverse operations
  Authoring-->>App: new revision, Undo entry
  App->>Host: prepare new runtime generation
  Note over Host: failure leaves the current sound running
  User->>App: publish (separate authorized action, release checks)

Rule	Consequence
A stale expected revision is a conflict	The candidate is rejected and must be regenerated against the latest
Acceptance writes one atomic revision with inverse operations	One Undo removes the whole proposal
Activation prepares a new runtime generation separately from acceptance	Accepted source and the running sound are distinct states
Failed compilation or cancelled inference never overwrites accepted source	The current sound is never interrupted by a bad proposal
A proposal can be schema-valid and musically bad	Technical checks and listening judgment stay separate
Imported notes, asset text and documentation are untrusted context	They cannot override application or user authorization

Data design for models#

The portable model is designed so an agent can edit it without ingesting the whole project.

Principle	In the Toy format
Descriptive stable IDs and keyed maps, not index references	`nodes`, `connections`, `parameters`, `frames` are keyed objects
Explicit endpoints	Connections carry `from` and `to` objects with node and port
Self-describing controls	Public parameters carry units, ranges, defaults, smoothing and named targets
Presentation binds to identity	Frames bind parameter IDs, never pixel positions or runtime-generated names
Small files, inspectable parts	A definition or selection can be discovered and inspected on its own
Catalog as documentation	Descriptions, typed ports, parameter domains, examples, capability needs, versions, target support
Diagnostics with addresses	Error codes and authored paths
Extension data preserved, not executed	Namespaced extension data survives round trips but confers no execution authority

Authoring API responsibilities#

Method group	Purpose
discover, inspect	Catalog and selection context for people and models
create candidate, validate, diff	Build and check a proposal without touching accepted source
prepare / render audition, compare	Hear or see a candidate in an isolated generation
accept, undo	Commit one atomic revision; reverse it
save, request publication	Persist a draft; start the separate release path

Operations carry actor and provenance, base revision, idempotency key, affected object IDs and the declared target profile. Store concise intent and observable artifacts; private model reasoning is not required. Revisions are server-ordered and sequential, with isolated candidate branches. Offline/local editing keeps a local history and reconciles explicitly; concurrent multi-user CRDT editing is deferred (the current collaboration subset is in community). The full envelope is in the operation contract.

Service boundaries#

The AI integration lives in the API/worker layer behind authoring permissions and per-job budgets. Provider credentials stay server-side. Users choose whether private source and assets may leave their device and which service receives them. Cancellation propagates through model and render jobs; budgets bound tokens, duration, asset uploads and retry count. Provider choice never affects local playback or recall. The app works without the agent service.

New Toy recipes#

The New Toy chooser's AI-first flow (v0.0.4 plan) uses the same proposal path. A recipe from @tunetoys/recipes fixes the form, the starter base, the effort level and a guidance paragraph; the form compiles to a small brief, and the browser applies the returned batch to its own copy of the shared base from @tunetoys/discovery before the person presses Create. The request is one cached system block (rules plus the compact catalog), one base-Toy block cached for official bases, and the brief, so a generation costs the brief plus output; one corrective turn reuses the prefix. Usage is metered per run and generation is bounded by per-address brakes, a rolling account allowance and a deployment ceiling (HTTP contract). The recipe forms and result step in the browser are planned.

Current review experience#

Before acceptance the browser shows actual source-property before/after values, separately from the model's explanation. The diff is bounded to 500 changes with explicit truncation and expandable full values. It is a structural source diff, not a semantic explanation.

High-level agent text, control and nested-Frame actions preserve unstated appearance, layer metadata and responsive constraints when updating an existing widget of the same kind. New widgets and kind changes use replacement semantics. The emitted canonical widget.put is still a complete replacement, so every write passes the same source validator and revision boundary.

Implementation#

packages/authoring — operations, candidates, history, agent.ts proposal helpers
services/api — agent.ts Claude Messages adapter and candidate routes (HTTP contract)
apps/web — PromptComposer.tsx, ProposalChanges.tsx
packages/node-catalog — the catalog both people and agents discover; compactCatalog() is its model-facing rendering
packages/recipes — New Toy recipes, briefs and the brief compiler