Infrastructure

Scaffolding decisions for hosting, domains, env, CI. Product code lands per PLAN.md; this file is the ops contract it lands into.

Domains

Three roles, each on a host you control. The names are yours; the roles are what the rest of this file assumes:

Keep the mail domain out of that list. A zone carrying MX records and a transactional sending subdomain should not also have an app routed at its apex — the two want different DNS and fail in different ways.

The regular deployment path has two targets. Per-branch preview Workers are not supported; targets here are hosts you own and name. The bounded mobile App Review exception is documented in app-review-instance.md.

Hosting

• Default target: Cloudflare Workers + D1 (one database per org), Queues for jobs, R2 for receipts + logical exports. Workers Paid plan required (D1 30-day Time Travel, higher limits).
• Second target: single container (Node 22 + SQLite file) — same codebase.
• The Worker entry is entries/worker: thin Hono, an instance-identity page, and /healthz reporting the environment and the commit actually running.
• workers_dev is false in every environment. The default *.workers.dev subdomain serves the same Worker with nothing in front of it — a side door around whatever gates the custom domain.
• D1/R2/Queues bindings land in wrangler.jsonc with the stories that use them. A binding nothing reads is a lie about what the Worker needs.

Backups (D3 / D18)

• Available now: hosted D1 Time Travel (30 days on the Paid plan, per-minute, on by default) and a stopped-container physical snapshot of db.sqlite plus attachments. Restore is a confirmed operator action, never automatic.
• Also available: ez backup / ez restore / ez export --verify for a container bundle, the Worker's nightly CSV bundle in R2 (backups/<date>/, recorded in backup_runs), the operational nightly SQL dump (.github/workflows/backup-sql.yml, #773), and ez backup --sync-to s3://... to copy a container bundle to a bucket you own (#38). The container and Worker bundles are different formats and the Worker bundle has no restore command; the container physical snapshot is neither. See RESTORE.md, including bucket versioning, Object Lock and put-only credential guidance.
• Not yet built (#38): the Worker's scheduled sync, off-site status in backup/status, destination preflight, and encrypting before objects leave.

Env & secrets

Registry lives in .dev.vars.example (copy to .dev.vars locally). Rules: secrets never in git; production secrets only via wrangler secret put or CI secrets; every new env var lands in .dev.vars.example in the same PR that reads it.

CI holds two repository-wide Cloudflare credentials. They are consumed only by deploy.yml and the manual provision-d1.yml / provision-queues.yml / provision-r2.yml workflows:

R2 bucket discovery and creation use Cloudflare's account REST API. In the API token editor, grant Account > Workers R2 Storage > Edit and scope it to the same account named by CLOUDFLARE_ACCOUNT_ID. A Worker R2 binding does not by itself grant the deployment token permission to list or create buckets. The workflow prints only this permission requirement on HTTP 403; it never prints the token, account ID, or Cloudflare response body.

Each GitHub environment (dev, prod) also holds its own API_CURSOR_SIGNING_KEY: canonical unpadded base64url for exactly 32 random bytes. deploy.yml installs it through wrangler secret put before every deploy, so tracked config never contains secret material. The two environments must not share a key.

Browser sign-in uses the provider-generic OpenID Connect core with google as the first configured provider. Google publishes discovery at https://accounts.google.com/.well-known/openid-configuration; that document currently identifies https://www.googleapis.com/oauth2/v3/certs as its JWKS. The runtime follows and validates discovery rather than hardcoding endpoint responses, requires PKCE S256, and pins ID tokens to RS256.

Create two Google OAuth Web clients, not one shared client, and register exactly one callback on each:

Store each client's OIDC_GOOGLE_CLIENT_ID and OIDC_GOOGLE_CLIENT_SECRET in its matching GitHub environment. deploy.yml converges the cursor key and Google pair in one Wrangler secret bulk request. If both Google values are absent, that request sends explicit null values to delete any stale Worker copies; if only one is present, validation fails before the remote mutation. Provider access tokens, ID tokens, refresh tokens, authorization codes, and raw browser state are never persisted. OIDC starts are limited to 20 per hashed Cloudflare client address in a rolling 10-minute window. Each start transactionally removes expired rows; consumed rows become eligible for cleanup no later than expiry.

Transactional email uses the SES v2 HTTPS API; SMTP is not part of the Worker contract. Each GitHub environment supplies AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as secrets, with AWS_SESSION_TOKEN when temporary credentials are used. It supplies SES_REGION, SES_FROM, and optionally SES_CONFIGURATION_SET as environment variables. deploy.yml validates configuration completeness before mutating Worker secrets, and the provider validates every value before runtime use. A partial or invalid contract fails closed; absent SES secrets are converged to explicit null deletions.

The IAM principal needs only the region/account resources used by the instance and these actions: ses:GetAccount, ses:ListEmailIdentities, ses:GetEmailIdentity, ses:GetSuppressedDestination, and ses:SendEmail. The configured sender identity must be verified with healthy DKIM in SES_REGION. When a configuration set is named, create it in that same region and attach event destinations for delivery, bounce, complaint, and rejection events. Provider errors never include AWS response bodies in the delivery log.

SES v2 SendEmail has no idempotency token. Ezacto signs a stable ezacto-email-<delivery-id> correlation value into an SES message tag and custom header, while the durable queue lease fences concurrent attempts and the receipt is persisted before acknowledgement. A provider success followed by a database failure may still be redelivered, so this is an auditable at-least-once boundary rather than a false exactly-once claim.

Live acceptance for issue #46 requires real sandbox configuration: provision the Queue binding tracked by issue #43, verify the sender and sandbox recipient, confirm account/sending/DKIM health, send through the deployed Queue consumer, and verify the administrator email log contains the real SES MessageId, AWS request ID, and latency. Then add a sandbox recipient to the account suppression list and prove no SendEmail request occurs while the exact terminal reason is logged. Synthetic fixtures do not satisfy that acceptance.

Each environment also holds one EZACTO_BOOTSTRAP_TOKEN, a canonical ezacto_ bearer generated independently for that instance. It is the first owner token, not a general deployment secret: bootstrap-instance.yml temporarily installs it on the Worker, persists only its selector and SHA-256 digest, proves the live API and compiled CLI can authenticate, and removes the Worker copy in an always() cleanup step. Store the original bearer in the operator's local CLI or credential manager when it is generated: GitHub retains it for automation but will not reveal it later. Workflow logs and artifacts never contain it.

Each environment also stores EZACTO_OWNER_PASSWORD for the explicit bootstrap browser owner workflow. It is not a deployment variable and is never installed as a Worker secret. The workflow reads it only into masked step environment, sends it in runner-temp request bodies, accepts the real browser session lifecycle, and removes those files. The value must contain 12–1024 Unicode code points and no more than 4096 UTF-8 bytes. Validation names a missing or invalid secret and restates this policy, but never reports the supplied value, its observed length, or derived data. Keep the same value in the owner's password manager; GitHub cannot reveal it after it is stored.

The Cloudflare token cannot touch any other zone, and cannot create zones. Rotate it by issuing a new token and replacing the repository secret; revoke the old one after the first green deploy.

D1 provisioning

Run the manual provision D1 workflow once before adding bindings. It converges the exact database each environment names: vars.DEV_D1_DATABASE_NAME (ezacto-dev) and vars.PROD_D1_DATABASE_NAME, the same variables render-wrangler-prod.mjs reads, so a rename lands in one place. An existing exact-name match is reused, no match is created, and duplicates fail closed. The workflow uploads a short-lived JSON artifact and job summary containing the non-secret database ID for each environment, and installs that environment's cursor-signing secret.

Commit the reported IDs under the matching env.dev and env.prod d1_databases entries in entries/worker/wrangler.jsonc. The normal deployment then applies the binding and its smoke gate proves the exact release is live.

Queue provisioning

The automatic dev deploy converges its exact resources before it mutates Worker secrets or bindings. Prod remains an explicit operator action: run the manual provision Queues workflow for prod before its tracked Queue binding is first deployed. The same workflow can explicitly converge dev when needed. Both paths read the names from entries/worker/wrangler.jsonc, reuse an exact-name match, create a missing resource, and fail closed on duplicates. They provision both the delivery Queue and its dead-letter Queue:

Cloudflare's Queue list/create API accepts the existing Workers Scripts Write permission, so this does not broaden the repository token beyond the deployment scope already documented above.

The same Worker is the EMAIL_QUEUE producer and push consumer. The consumer accepts one message per batch with one concurrent invocation while SES sandbox limits are in play. Cloudflare's bounded platform maximum of 100 Queue retries is the transport-failure budget, not a provider-call count: lease contention, D1 receipt persistence, and unexpected consumer failures can all consume Queue deliveries without consuming a provider attempt. Its five-second default retry delay matches the attempt-lease contention delay and prevents a hot retry loop. The durable email_log.attempt_count independently limits provider I/O to five claims and supplies the explicit per-message 60 / 300 / 900 / 3600 second provider backoff, which overrides the Queue default. Provider exhaustion is persisted and acknowledged. A message that still cannot complete after 100 transport retries reaches the dead-letter Queue instead of being discarded.

Every dev deploy idempotently converges both resources; a prod deploy only checks them and fails before changing secrets when the operator has not run provisioning. After wrangler deploy, both paths read Cloudflare's Queue API and verify the exact Worker producer, consumer, dead-letter target, batching, concurrency, and retry settings before the host smoke test can pass. APP_BASE_URL is a tracked per-environment Worker variable so a configured Queue and SES provider actually enable queued authentication mail.

Resource provisioning alone does not satisfy live mail acceptance. Each GitHub environment still needs its SES credentials and static variables described above; the sender and any sandbox recipient must be verified in AWS before a real password-reset delivery can be accepted.

Instance bootstrap

After the D1 binding is deployed, run the manual bootstrap instance workflow for that environment with the organization and owner identity. The database must already be migrated and its organization, user, email, and token tables must be empty. The workflow creates exactly one organization, active administrator owner, verified primary email, and all-scope owner API token in one D1 batch transaction.

The first statement claims a singleton audit record; the last statement asserts the complete seeded state. An identical retry is a no-op, while different input, pre-existing identity rows, or a partial write fails closed. Successful completion requires both live /api/v1/whoami and the compiled ez login/ez whoami path.

That first bootstrap intentionally creates no password. After setting the environment's EZACTO_OWNER_PASSWORD, run bootstrap browser owner. The temporary bootstrap authority can enroll only user 1 when its active, administrator-owner row, verified primary email, immutable bootstrap audit, and unrevoked owner token all agree. An exact password retry is a no-op; a different password or altered bootstrap state fails closed. Completion is the live password sign-in, secure HttpOnly session cookie, session-authenticated /api/v1/whoami, logout, and rejection of the revoked cookie. Google OIDC is an independent alternative once its client credentials are configured.

CI (.github/workflows/ci.yml)

verify = typecheck + lint + test + build across workspaces (--if-present, so the pipeline is green pre-code and picks packages up as they land), plus the gitleaks gate with its own self-test. Merge gates per PROCESS.md: lint, test, build, review — a red gate blocks merge.

CD (.github/workflows/deploy.yml)

Prod is never automatic. "It merged" and "we are ready to move a real book of work onto it" are different questions, and only one is answered by a push event.

After every verification job succeeds, PR CI uploads a verified-source receipt with the tested Git tree, PR head, repository and run ID. Deployment has a five-minute source gate: it resolves the merged PR for the release, requires its latest CI run to have succeeded, downloads that run's receipt, and compares its tree with the checked-out main release. It refuses missing/expired receipts, red or unfinished checks, branch dispatches, and changed source. If the base advanced, merge current main into the PR and verify that result before merging. Receipts are retained for 90 days; older releases require a fresh verified PR. The full suite runs once on the PR, rather than again after merge.

The deploy job builds and provisions as before, deploys with --var RELEASE:$GITHUB_SHA, and then polls the live host's /healthz until it reports that SHA. A wrangler exit code says the upload was accepted — not that the domain resolves, the certificate is up, or the route is bound. The target host is read out of wrangler.jsonc, never repeated in the workflow.

The public demo

The dev deployment doubles as the demo anyone can click through. It publishes its own sign-in credentials on its front page, so the guarantee that makes that tolerable is that nothing done to it survives the night.

Two switches, and both must agree: DEMO_MODE=true and ENVIRONMENT other than prod. DEMO_MODE is an operator's switch and ENVIRONMENT is the deployment's identity, so the flag copied into a production worker's vars — the way this goes wrong — publishes nothing and wipes nothing. Everything demo-shaped asks demoDeployment() in entries/worker/src/app.ts.

Because anyone can sign in, the demo reaches nothing outside its own database except the mail its operator allows (#931), whatever secrets the dev deployment carries:

• Mail goes out through the real transport, because the demo is where mail is tested before production — but only to recipients at the domains listed in DEMO_MAIL_ALLOWED_DOMAINS. The one producer every mailer is built on drops every other To and Cc address before the queue; a message left with nobody is not queued at all, and its log row ends failed with the reason demo_recipient_not_allowed (the Deliveries screen says it was not delivered, and the email health figures leave it out, since no provider saw it). The queue consumer applies the same list to anything already on the queue. The log keeps every address a message was written to; only what goes is narrowed.
• Providers — QuickBooks, BILL, Wise, Stripe — are composed with no credentials, so no mirror, payment link, payout or reconciliation can run.
• Routes that would connect or configure a provider, mint a payment link, quote or send a payout, change a sender identity, the invoice defaults (company information and the notes that carry bank details) or an invoice's payment options, or upload any file, answer 403 demo_disabled — "Not available on the public demo." So do the ones that would lock the next visitor out: a password reset, enrolling or removing a second factor, switching a sign-in method, adding, verifying or removing an SSO domain, deactivating anyone, changing anyone's permission profile or sign-in address, removing a user, and ending any session but your own (signing out still works). The list is entries/worker/src/demo-lockdown.ts, and the screens disable those controls with the same note.
• Storage takes no visitor's file: uploads meet a read-only store, and the demo is not exported nightly into a bucket the rebuild never empties. Invoice documents are still rendered and staged, because allowed mail carries them.

Which domains the demo may email

DEMO_MAIL_ALLOWED_DOMAINS is a comma-separated list of recipient domains, matched exactly and case-insensitively — example.com admits someone@example.com and not someone@mail.example.com:

DEMO_MAIL_ALLOWED_DOMAINS=example.com,example.org

It is deployment configuration, not code, so it never appears in this repository: the deploy workflow reads it from the GitHub environment variable of the same name (vars.DEMO_MAIL_ALLOWED_DOMAINS on the dev environment) and puts it on the Worker as a plain var. Nothing in the app can change it. Unset or empty on the demo, the demo emails nobody; on any deployment that is not the demo it is ignored.

Nor does anything in the app show it. Anyone can open the demo, and a list a visitor could read off a page would tell them where the demo can send mail, so no page, script or API response names a domain from it: the screens say only that the demo emails "approved addresses", whatever the list holds — even when it is empty.

The history is anchored to the day it is built, not to a fixed calendar, so the demo always shows a book of work running up to today.

Rebuilding it by hand

Nothing serves the reset over HTTP: an endpoint that empties a database is worth more to an attacker than everything else on the host put together. Trigger the 0 3 * * * schedule from the Cloudflare dashboard's cron trigger, or locally:

npx wrangler dev --env dev --test-scheduled
curl 'http://localhost:8787/__scheduled?cron=0+3+*+*+*'

The rebuild is idempotent and resumable: it reads what is left to bill off the uninvoiced rows themselves, so a tick that dies costs its own slice and the next one picks up where it stopped.