ezacto developers

Migration specification

Localized copy for the build. Source of truth: ezacto knowledge/docs/. Synced 2026-08-27 — if editing, edit the brain copy and re-sync.

title: ezacto-migrate — Harvest Extraction & Migration Specification type: spec status: draft version: 0.1.0 date: 2026-08-25 sources:

ezacto-migrate

The wedge (D7) and the acceptance test for the domain model (PLANNING-PLAN §5). Runs first against CONFLICT's own account — the OSS gate — then, unchanged, for anyone leaving Harvest.

Definition of done for a migration: the reconciliation report (§6) shows zero unexplained deltas, and every explained delta is one of the documented gaps in §7. "It imported without errors" is not done.

Product shape: a standalone CLI (published as ezacto-migrate, built from the oss repo's packages/migrate), zero dependency on the rest of ezacto at extract time. Extract works even if you never install ezacto — that is the point of the wedge: get your data out first, decide later.

ezacto-migrate auth        # PAT + account discovery
ezacto-migrate extract     # Harvest → snapshot dir (resumable)
ezacto-migrate verify      # snapshot internal consistency + reports checksums
ezacto-migrate load        # snapshot → ezacto org DB (D1 or SQLite file)
ezacto-migrate reconcile   # ezacto DB ↔ snapshot ↔ Harvest reports
ezacto-migrate sync        # incremental updated_since re-extract + upsert (parallel-run)

1. Auth & preflight

PAT only (no OAuth app needed for a one-shot tool). Headers on every call: Authorization: Bearer, Harvest-Account-Id, User-Agent: ezacto-migrate (email) — the missing User-Agent 400 is the first support ticket otherwise.

Preflight sequence:

  1. GET id.getharvest.com/api/v2/accounts — enumerate accounts, pick, resolve Harvest-Account-Id (never ask the user to find it in the UI).
  2. GET /v2/company — record wants_timestamp_timers (which time-entry shape to expect), clock (how to parse started_time/ended_time strings back to canonical HH:MM), the four *_feature flags (which trees will be absent), and display settings (imported into organization).
  3. GET /v2/users/me — confirm the token's user is an administrator; warn loudly if not (a member-scoped extract silently sees a fraction of the account).

2. Extract

2.1 Order (FK-safe, from research §15.3)

 1 company                         7 projects
 2 users → per-user billable_rates, 8 task_assignments      (account-wide sweep)
   cost_rates, teammates            9 user_assignments       (account-wide sweep,
 3 roles                              is_active=true AND =false)
 4 clients → contacts             10 estimates → messages
 5 tasks                          11 invoices → messages, payments
 6 expense_categories,            12 time_entries            (windowed from/to)
   invoice_item_categories,       13 expenses (windowed) → receipt binaries
   estimate_item_categories       14 reports checksums       (reports budget)

Account-wide assignment sweeps (8, 9), not per-project loops. Explicitly sweep is_active=false too — users/{id}/project_assignments only returns active.

2.2 Rate budget

General endpoints: 100 req / 15 s; reports: 100 req / 15 min. Steps 1–13 use only the general budget; step 14 is the sole reports consumer. Client behavior: token bucket at ~6 req/s sustained, honor Retry-After on 429, exponential backoff on 5xx. Cost model for a mid-sized account (tens of invoices, ~140 requests): the per-invoice message+payment loops dominate at 2 calls each — minutes, not hours. Print a call-count estimate before starting.

2.3 Snapshot format (the load-bearing artifact)

A directory, not a database — inspectable, diffable, committable if the user wants:

snapshot/
├── manifest.json          # account id, company name, started/finished_at,
│                          # tool version, per-resource counts + page counts,
│                          # updated_since watermark per resource
├── raw/<resource>.jsonl   # one Harvest object per line, verbatim, unmodified
├── receipts/<expense_id>.<ext> # Harvest vocabulary; loads as an expense attachment
└── checksums.json         # step-14 report aggregates (§6)

Raw means raw. No transformation at extract time — transform bugs must be fixable by re-running load without re-extracting (the extract is the expensive, rate-limited step; the transform is free). Pagination follows the response links verbatim (doc mandate), per_page=2000, cursor mode.

2.4 Resumability

After every page: append objects, fsync, update manifest.json progress. A crash or 429 storm resumes mid-resource from the last cursor. extract re-run on a complete snapshot becomes an incremental pass using per-resource updated_since watermarks. Deletes are invisible to updated_since (no webhooks, research §13) — sync therefore also does periodic full-ID sweeps per resource and marks vanished ids as deleted-upstream in the manifest.

3. Transform & load

Consumes raw/, produces an ezacto org DB. Pure function of the snapshot: same snapshot in, byte-identical DB out (ordering fixed, timestamps carried from source).

Key mappings (details in domain-model §7):

Harvest ezacto Rule
ids harvest_id columns Native ids assigned fresh; every table keeps the Harvest id, unique-indexed — this is what makes sync upserts and the shim's id echo possible.
decimal hours seconds ×3600, round half-even, record residue in load report if any.
entry rounded_hours rounded_seconds Convert the imported value to seconds and store it verbatim; never apply native rounding during load.
money amounts (totals, amount, payments) cents ×100 exact; fail loudly on >2 decimal places, never round silently.
per-unit rates (unit_price) cents ×100, round half-even, record residue in load report if any. A Harvest mileage category at 0.485/mile is the IRS half-cent rate, not bad data; the line amount stays exact, so a rounded rate never moves a total.
invoice tax/discount percentages *_rate_ppm Parse decimal percentage text exactly into parts per million of one (7.25% = 72_500); fail beyond four fractional percentage digits, never pass through REAL.
started_time "8:00am" HH:MM Parse per snapshot company.clock.
access_roles array profile + manager_grants Per domain-model §2.2.
is_billable+is_fixed_fee billing_method Truth table; conflicting combos (billable=false, fixed_fee=true) recorded as anomalies, imported as non_billable.
billable/cost rate rows append-only tables Verify Harvest's derived end_date chain matches ours (invariant 8); mismatch = anomaly, ours wins.
entry billable_rate/cost_rate snapshot columns Copied verbatim from Harvest, never re-resolved — Harvest's historical resolution is truth for imported rows.
invoice/estimate creator {id,name} nullable real creator-user FK + immutable source creator id/name provenance Resolve the FK when the imported user exists; never discard source attribution or create a bare/mandatory placeholder when it does not.
invoice/estimate message sender scalars immutable sent_by, sent_by_email, sent_from, sent_from_email snapshots Copy verbatim; a later nullable sender-identity relation is enrichment, not historical truth.
invoice derived totals + payment_options + updated_at native D21 totals + importer-owned source_* observation Copy source amount/due/tax/tax2/discount cents, option array, and canonical source timestamp verbatim, then compute native totals by D21 so any undocumented rounding/allocation delta remains explainable. Refresh the observation atomically only when source updated_at advances; equal/older sync input is a no-op. Never guess what Harvest ach means natively.
payment recorder/gateway/transaction manual/manual payment + immutable source provenance Resolve a nullable real recording-user FK when possible; retain recorder name/email and gateway id/name verbatim. Preserve the source transaction id without guessing a native provider from the gateway name.
payment response paid_at + paid_date canonical XOR + immutable source_paid_at/source_paid_date Copy both response fields. Prefer paid_at canonically when both are present; compare its UTC date to paid_date, recording an anomaly on disagreement without discarding either source value. The shim re-emits the preserved pair.
invoice state + timestamps same States imported as-is; state machine governs post-import mutations only.

3.1 D1 write constraints (from D3 — designed in, not discovered)

4. Receipts

Download every receipt.url binary; store under content hash; verify file_size matches. Failures are anomalies, not fatal (Harvest serves receipts through time-limited URLs — re-extract refreshes them). On load, a Harvest receipt becomes an expense-owned row in the shared attachment model; there is no separate native receipt table or expense.receipt_id. Hash representation, logical dedupe, and the exact field mapping are specified in Attachment storage contract.

5. The sync loop (parallel-run)

Market research: switchers parallel-run for up to a year. So sync is not a convenience — it is the adoption path. extract --since + upsert-by-harvest_id

6. Reconciliation (the actual gate)

Three-way check, emitted as reconciliation-report.md + machine-readable JSON:

A. Snapshot ↔ Harvest reports (step 14, reports budget):

B. ezacto DB ↔ snapshot: row counts per resource; Σ seconds and Σ cents per (user, project, month) — invariant 14; invoice totals/due per invoice to the cent; retainer dangling-id count (expected, §7).

C. ezacto DB internal: domain-model §8 invariants run as checks against imported data (notably 4, 5, 8, 11).

Every delta is classified: rounding (bounded, explained), gap (§7, enumerated), or UNEXPLAINED (fails the run). The report prints the three classes separately; the gate is zero UNEXPLAINED.

7. Known, documented gaps (from research — decided handling)

Gap Handling
Retainers: no API. Invoices reference dangling retainer.ids. Create money-denominated stub retainer rows from the distinct ids found on invoices; balances are unknowable via API. migrate finish-retainers prints a worksheet (client, linked invoices) for manual balance entry from the Harvest UI. Ledger opens with a cents-denominated manual adjustment entry.
Recurring invoices: no API. recurring_invoice_id dangles. Same: stub rows + worksheet from the UI's 3 visible definitions (subject template, cadence, amount are all on screen — HRVST20/15).
Estimates/approval/activity-log modules disabled on the source account Nothing to extract; extractor skips per company feature flags and says so. Invoices raised before the module was turned off can still carry an estimate.id (the source account had one); the link is dropped and recorded as an unresolved_estimate_reference anomaly rather than failing the import, since there is nothing to link to and nothing to recover.
The one invoice that actually carried an estimate reference. Invoice 9076155 (ezacto id 556, number 742) pointed at estimate 1035594, which no snapshot contains: estimate_feature was false on the source account, so estimates, estimate_messages and estimate_item_categories all swept 0 rows, 0 pages. Loaded with estimate_id NULL and recorded as an unresolved_estimate_reference anomaly, per the row above; the reconciler cites it as migration-spec-7-estimates-module-disabled, so the run still passed at 0 unexplained. Named here because the source account is closed and the reference can never be resolved: of 741 invoices in production, estimate_id is NULL on all 741, and a future reader must not take that to mean no invoice was ever raised from an estimate. One was.
Report-only fields (utilization) Derived, not stored — recomputed by ezacto; reconciled in A.
Forecast Out of scope (research §14).
statement_key / invoice and estimate client_key secrets Not imported. Regenerated — importing another system's public-URL bearer tokens imports its leak surface. Old Harvest links die at cutover; release note item.
Avatars Best-effort download; failures cosmetic.
Negative time entries. Harvest corrects an over-logged timesheet with a negative entry offsetting an earlier one (the reference account had a handful, one of them recent and unlocked). Imported. time_entries.seconds, seconds_without_timer and rounded_seconds were CHECK (… BETWEEN 0 AND …) and are now abs(…) <= limit, the same signed shape invoice_payments.amount_cents took. Skipping was never safe: Harvest nets these entries out and ezacto could not, so a month's total read above Harvest's by the whole of the skipped correction — an overpayment to a contractor on a live payroll run. Budgets and retainer balances keep their non-negative CHECK, because a budget really is a magnitude. Not a gap.
Non-positive invoice payments. Harvest records $0 payments settling $0 invoices, and negative payments settling credit notes (the reference account had several of the former and one of the latter). Imported. invoice_payments.amount_cents was CHECK (… BETWEEN 1 AND …) and is now abs(...) <= limit, the same signed shape invoices.amount_cents has always had; bank_deposits keeps the strict rule, because a bank credit really is positive. Skipping was never safe for this class: invoice state is derived from imported payments (packages/db/src/internal/invoice-import.ts), so an invoice left with no payments landed open, and every settled invoice in the reference account flipped from paid. Not a gap.
Two Harvest accounts for one person. Harvest holds sixty users for fifty-eight people: two were created twice, once under a work address and once under a personal one. The pairs are handed to the loader as an identity map and squashed: the duplicate's row is never inserted, its address joins the survivor as a secondary, and every reference to it is redirected. Rows the survivor can hold only one of — an assignment on a shared project, a rate starting on a shared date, a teammate link — merge into the one already there rather than aborting the load, each recorded as a duplicate_row_merged anomaly beside the duplicate_user_squashed one. Reconciliation cites those counts against its row counts, so sixty source users becoming fifty-eight is accounted for rather than read as loss. One collision is refused instead of merged: a timer running on each of the pair, since a person may hold only one and closing the other would need an ended_time Harvest never recorded. It is refused before the snapshot is admitted, so it costs no rows — stop one of the two timers in Harvest and extract again.
Per-unit rates finer than a cent. Harvest accepts unit_price at arbitrary precision — mileage categories carry the IRS half-cent rate (0.485/mile), and Harvest's own computed lines can carry repeating decimals (67.1428571). Rounded half-even into cents and recorded as a rate_residue anomaly carrying the source literal, the same handling decimal hours get. The line amount is still parsed exactly by the money rule, so a rounded rate can never move an invoice total — reconciliation proves this rather than assuming it.

8. Milestones

# Milestone Proves
M1 auth + extract complete against CONFLICT account, resumable, manifest counts match UI spot-checks API client + snapshot format
M2 verify + checksums reports budget handling, per-currency grouping
M3 load to local SQLite, invariants pass the domain model itself — this is the S3 gate
M4 reconcile zero UNEXPLAINED end-to-end losslessness
M5 load to hosted D1 via queue D1 constraint handling
M6 sync steady-state for 2+ weeks parallel-run viability
M7 retainer/recurring worksheets done, books balance the manual-gap path

M3 failing is a domain-model bug first — fix the model, then the loader.

The milestone numbers are not the running order

M6 is numbered before M7 and must not be run before it. Worksheet completions are bound to the snapshot and context digests they were entered against, so a later sync rewrites the manifest and voids every one of them — and the new snapshot cannot be loaded into the existing database, so the worksheets have to be redone from scratch in a virgin one. Reading this table as a sequence prescribes exactly the redo it is meant to prevent (issue 288).

The running order is:

sync → verify → load → reconcile → worksheets → ship

No sync between the worksheets and shipping. M6 proves parallel-run viability and belongs to the rehearsal; the final sync is the one that precedes the load you actually ship. docs/cutover-runbook.md states the same rule at the point of use.

If a late sync turns out to be unavoidable, keep the filled worksheet JSON — only the four header digests need regenerating, not the operator's transcribed data.