Releasing TuneToys app versions

The step-by-step process for promoting a tested immutable Linux container to the AWS pilot, how artifacts are identified, and what rollback can and cannot do. This releases the website/API only; creator Toy publications and Player/plugin builds are different release streams.

Status: Implemented as a manual process — no automatic production deployment, rolling deployment or zero-downtime path exists. Saving source, committing, pushing or running npm run build/check does not update tune.toys. The v0.0.3 pipeline has landed but has not run on GitHub; CI has no AWS identity, so images are published to ECR by the operator, and the host promotes them by digest with promote.sh once the owner installs it.

Rules that never change: keep a retained rollback artifact and confirm a fresh private S3 backup before each promotion; app deployment must never move a customer's pinned Toy version or replace audio in a running DAW. Current production identity and checks are in the AWS pilot record; host setup is in hosting.

Release flow#

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FAFAF7', 'primaryTextColor': '#1C1C1B', 'primaryBorderColor': '#1C1C1B', 'lineColor': '#64645F', 'fontFamily': 'DM Sans, Inter, system-ui, sans-serif'}}}%%
flowchart LR
  DEV["Reviewed source"] --> CHECK{"Checks pass?"}
  CHECK -->|Yes| IMAGE[["Frozen Linux image<br/>source + engine receipt"]]
  IMAGE --> TEST{"Exact artifact<br/>qualified?"}
  TEST -->|Yes| BACKUP[("Fresh S3<br/>recovery archive")]
  BACKUP --> LOCK["Deployment lock"]
  LOCK --> LIVE(["Restart Lightsail API"])
  LIVE --> HEALTH{"HTTPS and<br/>flow checks"}
  HEALTH -->|Pass| KEEP["Record release<br/>and back up"]
  HEALTH -->|"Fail and compatible"| ROLLBACK["Restore previous image"]

  classDef source fill:#F0442E,stroke:#1C1C1B,color:#FAFAF7
  classDef shared fill:#DCEBDD,stroke:#1C1C1B,color:#1C1C1B
  classDef host fill:#DED8F0,stroke:#1C1C1B,color:#1C1C1B
  class IMAGE source
  class CHECK,TEST,HEALTH shared
  class BACKUP,LOCK,LIVE,KEEP,ROLLBACK host

The 2 GB host is not a build machine: build on the development/build host and preserve the shared prepared engine bytes. Use a temporary test environment rather than an always-on staging bill. The current production target is Linux amd64, account 982471468586, Ohio us-east-2, instance tunetoys-pilot, static IP 3.20.14.74. Confirm these against AWS before acting; historical documentation is not target-discovery authority.

Version and artifact identity#

• Use a human app version/changelog for communication and immutable input/image/engine digests for deployment. About reads the version from apps/web/package.json.
• Choose a patch/minor/major app version before building; for example, npm version patch --workspace @tunetoys/web --no-git-tag-version --ignore-scripts updates the web version and lockfile without publishing or creating a Git tag. Review both files.
• API, Player and independent product versions have separate compatibility meanings. Never bump engine ABI, source schema or plugin identity as a cosmetic app-version change. A new app version does not rewrite immutable packages, media hashes, stable parameter IDs, plugin IDs, signed exports or saved DAW state; engine ABI/semantic and portable schema changes need their own compatibility/fixture review.
• The repository currently has development package versions; a package version or Git tag alone does not prove which bytes are running. Record the source commit when the checkout has been committed. For the current pilot, frozen source manifests also identify reviewed artifacts produced while other chats edit the checkout.
• A receipt contains inputSha256, engineSha256, imageId, platform and its frozen context. Never deploy a floating tag such as latest. A changed checkout prevents the build tool from advancing its local latest pointer; the retained older artifact is not automatically approved. Inspect it, compare its manifest with the desired changes and qualify that exact receipt, or rebuild from a stable checkout. Check current source/tests separately from the source actually inside a retained image.
• The sidebar release is an example of a narrow update built from the prior deployed snapshot while later editor/DSP work continued, preserving the deployed engine/API. Normal releases should use one isolated, reviewed checkout so the final artifact includes exactly the intended changes.

Publish to ECR#

This is the current publish step. It replaces the archive transfer when the host has promote.sh installed; the code-only promotion steps below remain the fallback. The automated path is the same tool: release-web.yml runs it with --ambient-credentials once the tunetoys-ci role exists, which the organization's service control policy currently prevents (see the workflows README).

1. Commit the reviewed change, then build and qualify from that clean checkout: npm run build:container -- --platform=linux/amd64 and node tools/test-container.mjs /absolute/path/build.json --smoke.
2. Preview: node tools/publish-image.mjs /absolute/path/build.json --dry-run. It prints every command and any refusal without calling AWS.
3. Publish: node tools/publish-image.mjs /absolute/path/build.json. The AWS profile is --profile, else AWS_PROFILE, else tunetoys; the repository and bucket are --repository/--bucket or TUNETOYS_ECR_REPOSITORY/TUNETOYS_AUDIT_BUCKET (values in the workflows README). It refuses a dirty checkout, container inputs that differ from the receipt's frozen snapshot, and a local image whose platform or labels differ from the receipt. It logs in to ECR, tags the receipt's image ID <repository>:<inputSha256> and <repository>:sha-<commit>, pushes, prints digest=sha256:…, and writes the receipt (without its local path, plus registry, commit and caller) once to s3://<audit bucket>/receipts/<inputSha256>.json. Tags are immutable: an existing tag or receipt is accepted only when it names the same image. The profile needs ECR push on tunetoys/api and s3:PutObject on receipts/; the host's tunetoys-host-delivery policy grants pull only.
4. On the host, after a fresh confirmed backup is available: sudo tunetoys-promote promote <imageId> --expect <running imageId> --digest <digest> --input <inputSha256> --engine <engineSha256> --kind image --run <digits> --commit <sha>. It pulls by digest, verifies identity and labels, refuses a migration downgrade, takes the lock, proves the backup, swaps image.env, restarts, checks health and rolls back an image-only failure. A schema release runs tunetoys-promote rehearse … --kind schema first, then promote … --kind schema --recovery <option>. Then complete steps 8 and 9 of the release flow.

The receipt's imageId is whatever the building Docker daemon reports, and promote.sh compares it with the pulled image's ID. A builder using the containerd image store reports the OCI index digest (the local builds on November 1, 2026 did), while a classic-store daemon reports the image config digest. Confirm that the builder and the host report the same ID for one image before the first promotion from ECR.

Code-only promotion#

Use the trusted administrator SSH identity and pinned known-hosts file. Keep private operator configuration outside Git. Review commands before running them; credentials are not command arguments or release payloads.

1. Save the candidate with docker image save using its receipt's exact image ID, compress it and compute SHA-256. Retain the exact receipt alongside it. No .env, local account database or private operator directory belongs in an image.
2. Transfer through authenticated SSH to a unique temporary staging directory. Verify the archive checksum before loading. Inspect the loaded image and require Linux amd64 plus matching toys.tune.input-sha256 and toys.tune.engine-sha256 labels. Loading an image is staging, not promotion.
3. Start a fresh backup with sudo systemctl start tunetoys-backup.service. Inspect the new /var/log/tunetoys/operations.log entry confirming its versioned S3 archive. A successful service exit is insufficient: a run skipped because the deployment lock is held deliberately emits no backup success. Investigate/retry before changing production.
4. Acquire an exclusive flock on /srv/tunetoys/operator/backup-work/deployment.lock. Backups and promotions share that inode; never remove/recreate it. Compare /etc/tunetoys/image.env and the running container against the expected previous image. Abort if another operator changed production.
5. Preserve the previous root-only image.env and public release metadata under /srv/tunetoys/operator/releases/<input-sha>/. Write only TUNETOYS_IMAGE=sha256:<verified image ID> to a mode 0600 temporary file in the same config directory and atomically rename it over image.env. Do not source Docker env files as shell code or print the private api.env.
6. Restart with sudo systemctl restart tunetoys-api. Check http://127.0.0.1:8787/v1/health and https://tune.toys/v1/health, public HTML/asset loading and changed UI behavior. Keep the lock until the promotion/compatible rollback decision finishes. Configuration stays on the encrypted mounted data disk; Caddy/domain/mail usually need no change.
7. If readiness or smoke checks fail and the old image is compatible with the unchanged database/config/data, atomically restore the preceding image.env and restart. Check readiness again. Do not delete the failed artifact or its receipt; retain it for diagnosis.
8. Release the lock, run/confirm another backup, and update the deployment record. Retain old lazy-loaded web assets across a future deployment strategy or tell editing users to refresh when ready; do not force a reload that discards their work.

Expected brief downtime comes from stopping/restarting the single API container. A later blue/green approach needs old asset retention, database compatibility and capacity beyond this small host.

Schema or configuration releases#

Current PostgreSQL startup requires an exact migration ledger and hashes. Applying a new migration makes an old image's startup verification fail, even when SQL is additive. An older image continuing to run is not proof that it can restart. Never promise automatic image-only rollback after a migration. See database delivery.

1. Review the forward migration, runtime table privileges, import/cutover impact and old/new data behavior. Do not edit applied migration files. Rehearse the candidate image with a restored, populated database and private media/native key.
2. Choose and test recovery explicitly: a rollback image that includes the new ledger but compatible old application behavior; fix forward; or restoring the pre-migration database under maintenance. A database restore can discard subsequent writes, so stop writes and define the recovery point first. A production restore is not a routine image flip.
3. Capture a fresh confirmed backup including roles, configuration/native key, TLS, exact image and S3 byte inventory. Hold the deployment lock through migration and promotion, using the separate protected migration environment/role. The application role must not migrate or gain owner privileges.
4. Run the candidate's explicit migration/bootstrap flow from database delivery, reconcile reviewed table grants, start the candidate and verify existing accounts, releases, library/Player grants and new schema paths. A missing or altered ledger must fail before serving requests.
5. Perform the agreed recovery if qualification fails. Otherwise release maintenance/lock, verify and create a fresh post-release backup. Record both migration history and running image.

The billing migrations are in source only until this process is performed. Do not deploy a newer billing image against the old pilot ledger, or run a migration as an incidental startup side effect. A future compatible-ledger policy would need version bounds and backward/forward tests; it is not current support.

For configuration changes, retain the prior protected values and stable native authorization key. Validate optional provider configuration before restart. Never copy the developer .env wholesale. Stripe connection/live activation is its own reviewed rollout, not a side effect of deploying the code; see Stripe setup.

Acceptance and monitoring#

Minimum checks: app HTML/assets, sign-in/session continuity, an existing owned draft, public playback, private cross-account denial, invited-upload policy, exact engine/media bytes, library sync and the changed feature. Use bounded original audio; do not require user devices or live purchases to pass automated fixtures. Native changes also require Player/plugin build and declared DAW qualification; web-only changes do not claim that qualification.

Record the app/source/image/engine identities, operator/time, changes, migration/configuration impact, checks, backup version, rollback choice and known limitations. Monitor health, memory, error rate, missing-backup alarms, email and relevant business-event failures after rollout. Budget alerts notify and do not stop resources.

A daily snapshot or uploaded archive is not a proved replacement-host restore. Keep the remaining hosted/browser/DAW recovery gates visible in the deployment record. Restore rehearsal and periodic dependency/OS updates are operational work separate from ordinary feature releases.

Later automation#

Automate this same sequence after manual releases are repeatable: CI checks, immutable reviewed artifacts/receipts, retained versioned S3 archives, target verification, fresh-backup proof, protected promotion credentials, lock/CAS, schema compatibility gates, public smoke checks and explicit recovery. Trigger production promotion deliberately; do not turn every push into a live deployment. Signing credentials and Stripe live keys never enter a browser build or untrusted pull-request runner.

The v0.0.3 continuous delivery plan specifies this automation, and its files have landed disabled: release workflows under .github/workflows/, infra/lightsail/promote.sh, infra/aws/delivery.json and the gate, smoke and publish tools. Nothing runs on push until the repository variable TUNETOYS_RELEASE_PIPELINE is enabled and the enablement checklist is complete. As it stands, an enabled push would check, build and qualify the image on CI and then stop: without the tunetoys-ci role it publishes nothing and reports "gate skipped: no AWS identity". The plan: the same sequence driven from a push to main, with image-only releases promoted automatically once supervised releases pass and schema releases always held for approval.

Related#

• Hosting runbook
• AWS pilot record
• Database runbook
• Release, state and compatibility contract for creator Toy releases