Activity Intelligence Datomic Substrate Prototype

TL;DR

A 25-bead, parallel-substrate prototype was built end-to-end on 2026-05-14, then put through a real-Bedrock n=30 directional evaluation that surfaced two regressions and produced one architectural surprise. The codebase wires both backends — Datomic Pro com.datomic/local 1.0.291 behind a 73-line Clojure HTTP shim, and Postgres 16 + PostGIS 3.4 in a self-contained Docker container — under one minimal agentic AccountResolver agent built on Bedrock Converse with Claude Sonnet 4.5.

The agent has six tools, identical signatures across backends. It resolves a transcript-style upload to the most likely account in the PDI Federated tenant fixture (2,634 accounts). Loaders ingest the PDI seed, the 3,383-row Google Places enrichment universe, and the 178 ground-truth-labeled golden transcripts.

Five eval phases past the original n=30: regression fixes (Phase 5 ruling-5), Datomic as-of source-time gate (Phase 6), re-validation (Phase 6.5-A), tool-parity ablation (Phase 6.5-AA), sequential extension to n=80 (Phase B), and a further extension to n=160 (Phase C). Total Bedrock spend across all phases: ~$132.

Our recommendation: build the production AI layer on Postgres + PostGIS — and the n=160 data now supports this on both measurement and operational grounds. The substrate-attributable accuracy signal that re-opened the debate at Phase 6.5-AA didn't survive contact with the larger sample. Bitemporal correctness is at parity. Operational fit decisively favors Postgres (team Clojure ramp, dev bring-up, 73-LOC shim). See §Substrate recommendation for the full reasoning.

Headline results (n=160 cumulative, Phase C)

Numbers below are aggregated across all run/bitemporal JSONLs in the eval results directory, deduped by upload_id + backend. The cumulative dataset is Phase 6.5-AA's n=30 (uploads 1-30) plus Phase B's n=50 (uploads 31-80) plus Phase C's n=80 (uploads 81-160), totaling 160 transcripts × 2 backends × 2 passes = 640 resolutions. Generated against HEAD = f04d2e6a0. The trajectory across phases is preserved in the §Learnings section.

Accuracy

Bitemporal impossible-match prevention

At n=160, both backends still prevent every impossible-match false positive: Datomic 98/98, PG 91/91. Bitemporal correctness is the cleanest measured axis — the Phase 6 fix continues to work at all sample sizes, and Postgres's explicit-clause discipline has held throughout. Both substrates are correct on this axis.

Latency and cost

At n=160 cumulative, Datomic is ~8% tighter at p95 latency (41.9 s vs 45.6 s) and ~2% cheaper ($20.68 vs $21.08). The gap narrowed substantially from the n=80 measurement (where Datomic was 19% tighter / 10% cheaper) — same regression-to-mean pattern as accuracy. Substrate barely moves these numbers at the larger sample; agent reasoning loops dominate.

Code complexity (sample-independent)

Datomic's recorder remains 30% smaller (149 vs 211 LOC) for the same operation. As-of code is 39 vs 34 LOC. The 73-line Clojure shim is part of the Datomic side and is a fixed operational tax. Sample-independent — these numbers don't change with eval size.

Cost ledger (cumulative across all eval runs)

• Phase 4 (regression-affected, 0% accuracy): $9.73
• Phase 4 killed full-run attempt: $4.75
• Phase 5 smoke + golden + bitemporal (regression-affected, blocked at row 36): ~$4
• Phase 5 ruling-5 smoke + full eval (the clean pre-fix run): ~$13
• Phase 6 as-of fix investigation (zero-Bedrock probes + smoke): $0
• Phase 6.5-A re-run after fix (smoke + golden + bitemporal): ~$13
• Phase 6.5-AA tool-parity ablation (smoke + golden + bitemporal): ~$17.47
• Phase B sequential extension n=50 (smoke + golden + bitemporal): ~$25.92
• Phase C sequential extension n=80 (smoke + golden + bitemporal): ~$43.70
• Total Bedrock spend: ~$132

Extrapolating to the full corpus: another n=18 × 2 backends × 2 passes = 72 resolutions at ~$0.13/resolution would cost roughly $10 more to reach n=178. Wall-clock would be ~1 hour per pass. With n=160 now showing the substrate accuracy gap is at noise floor, the marginal value of finishing the corpus is modest — the decision has effectively been made.

Substrate recommendation

What the data actually says (post-C, n=160)

• The accuracy gap regressed toward the mean across sample doublings. Datomic accuracy@1 trajectory: 90.00% (n=30) → 90.00% (n=80) → 85.62% (n=160). PG: 83.33% → 85.00% → 83.12%. The Datomic lead: +6.67pp → +5.00pp → +2.50pp. The pattern is consistent with the substrate effect being smaller than the small-sample readings suggested, plus the first 30 transcripts happening to be a Datomic-favorable subsample. At n=160 the 95% CI on the gap is ±4pp — the lead is no longer statistically distinguishable from zero.
• Bitemporal correctness holds at all sample sizes. Datomic 98/98 and PG 91/91 impossible-match FPs prevented at n=160. The Phase 6 fix (explicit source-time gates) does its job; Postgres's explicit-clause discipline has held throughout. This is the cleanest measured axis: both substrates are correct, neither requires a footnote.
• Datomic's code-size win is still real but no longer decisive. 30% smaller recorder (149 vs 211 LOC). As-of code is 39 vs 34 LOC. The 73-LOC Clojure shim is a fixed operational tax. At n=160, the absolute code-size delta is ~80 LOC across the prototype — meaningful at the per-feature level but not enough to overturn operational fit.
• Latency and cost converged toward parity at n=160. n=30: Datomic 16% cheaper, 18% tighter at p95. n=80: 10% cheaper, 19% tighter. n=160: 2% cheaper, 8% tighter. Same regression-to-mean pattern as accuracy. Substrate barely moves these numbers at the larger sample; agent reasoning loops dominate.

What pushes the recommendation toward Postgres despite the data

• Team already runs Postgres in production (Django + Supabase + OVH). Adding a Datomic substrate adds Clojure CLI + JDK + dev-local + a Clojure HTTP shim to the operational footprint — for a team with zero existing Clojure deployments. The 73-LOC shim is the smallest possible Clojure surface; expanding it requires Clojure expertise the team doesn't yet have.
• Trace observability lives in Langfuse, not the substrate. The Phase 5 refactor ("tool_call_trace is observability, not a fact") means the value of tx-meta's "all provenance in one transaction" promise mostly evaporates in a system where Langfuse owns the trace. Datomic's tx-meta still beats jsonb columns for the small structured provenance (model_id, prompt_hash), but that's a ~60-LOC saving, not an architectural lever.
• The strangler implementation is simpler on Postgres. Django save() override → outbox table → consumer → Postgres entities/events/facts in the same instance is a smaller operational delta than Postgres outbox → Datomic substrate (and bring-up cost on every dev's laptop).
• The Phase 6 as-of fix exposed an operational subtlety. "Free bitemporal correctness" only works when :db/txInstant tracks source-record time. Production data loaded in batches or via outbox may have tx-time = "now" while source-record time = "when the user actually wrote it." On Datomic this is a discipline you have to remember (which the prototype team forgot, and Phase 5's eval caught). On Postgres the equivalent discipline is explicit by construction — every read has a WHERE created_at <= as_of.
• +2.50pp accuracy at n=160 is no longer a defensible measurement. The 95% CI on the gap at n=160 is roughly ±4pp — the CI crosses zero. The trajectory across phases (+6.67pp → +5.00pp → +2.50pp) is consistent with regression toward the mean of a small or zero true effect. Even an n=178 full-corpus run wouldn't materially change this: the marginal 18 transcripts shrink the CI by only ~0.4pp.

What would flip the recommendation back to Datomic

• A different eval workload (not account resolution) where the entity/event/fact model's schema flexibility actually does something Postgres can't do as cleanly — e.g., a workload with frequently-changing fact attributes where tx-meta makes per-feature work substantially smaller. The single-workload eval undersells this dimension.
• A team commitment to Clojure operational ownership (one person who owns the shim, the JDK, and substrate ops). Substantially de-risks the operational argument, but doesn't restore a data argument that the n=160 eval just retired.
• A material change in production volume or latency requirements that pushes the small Datomic latency advantage into "decisive" territory. At current measurement, single-digit percent differences are inside noise.

Confidence and caveats

This recommendation is at moderate-to-high confidence after Phase C, supported by both operational fit and the n=160 accuracy data. The bitemporal-correctness axis is settled on both substrates (parity post Phase 6 fix). The accuracy axis at n=160 shows a +2.50pp Datomic lead with 95% CI ±4pp — directional but not significant, and trending toward zero across sample doublings. The substrate-attributable signal that re-opened the debate at Phase 6.5-AA has collapsed under more data. The remaining decision drivers — operational fit, team Clojure experience, dev-laptop bring-up, fixed Clojure-shim maintenance tax — all favor Postgres. Going from n=160 to n=178 would not materially change this read.

The agent and its tools

Shape of the agent loop

One AccountResolver agent, identical across backends. The only thing that forks per backend is the tool implementations and the resolution persistence; the agent loop, system prompt, skill markdown, proposal types, and Bedrock Converse client are all shared.

The six tools

Observed agent behavior

From the JSONL traces, a typical resolution looks like:

lookup_accounts_by_gps (lat,lon, 5mi)
  → 3 candidates
lookup_accounts_by_fuzzy_name (transcript-extracted name)
  → 5 candidates, partial overlap with gps result
get_account_recent_notes (top candidate)
  → confirms or denies match
get_account_contacts (top candidate, if notes inconclusive)
  → final disambiguation
ProposeAssignAccount (chosen account_id, confidence 0.7–0.95)

The expensive resolutions (~$0.35-$0.43 in tokens) involve many iterations:

lookup_accounts_by_fuzzy_name, lookup_accounts_by_gps, lookup_trommel_places,
get_account_recent_notes, get_account_recent_notes, get_account_recent_notes,
get_account_contacts, get_account_contacts, get_account_contacts,
lookup_trommel_places, lookup_accounts_by_fuzzy_name, lookup_accounts_by_fuzzy_name,
lookup_accounts_by_fuzzy_name, get_account_recent_notes, get_account_recent_notes,
get_account_contacts, get_account_contacts, lookup_accounts_by_gps

That's 18 tool calls on a single resolution before emitting a prediction. The agent is iterating on disambiguation when the transcript names a business name that's close to multiple candidates ("Brian's Automotive" → candidates: Grit's Automotive, 21st Century Auto, etc., none of which is the correct "Bradys Automotive Center"). This is a real-world signal: when business names rhyme or contain shared substrings, more tool calls help only marginally. The agent's confidence stays ambiguous and it picks wrong (the case above) or emits null entirely.

Costliest single resolutions (top 5)

The same uploads appear at the top on both backends, suggesting the cost driver is transcript-intrinsic (long, ambiguous, name-similar-to-many-accounts) rather than substrate-attributable. The agent's reasoning cycles eat tokens, not the substrate.

Where the agent fails

Failure categorization from failure_analysis.md (n=30 sample, both backends):

• Disambiguation failure — Datomic 4, Postgres 7. The agent narrows to 2-3 strong candidates with similar names and picks wrong. This is the dominant failure mode for both backends.
• No GPS signal — both backends miss the 1 upload without GPS. Without geographic anchoring, fuzzy_name alone isn't enough.
• Transcription error / phonetic only / GPS too far / impossible match — 0 occurrences in this sample.

One illustrative case: upload 0bd983e2… (transcript: "Rusty's Repair LLC"). Tool returned top-5 candidates: Rusty's Repair LLC (0.89), Rusty's Repair LLC (0.89), Rusty's Repair LLC (0.89), two unrelated. Despite the obvious correct match at 0.89 confidence appearing three times in top-5, the agent emitted predicted_account_id: null. That's an agent confidence-threshold or proposal-emission bug, not a substrate problem. Worth fixing in the next iteration of the agent loop.

Was the performance of account matching tested with the Datomic approach?

The full eval pipeline ran cleanly:

1. Real-Bedrock smoke test (one transcript, one backend, ~$0.10) — passes before the full eval. Catches schema/loop regressions in 25 seconds.
2. Unconstrained golden eval against 30 transcripts × 2 backends = 60 resolutions. Wall-clock ~32 min, cost ~$5. Output: run_1778909831.jsonl (60 rows).
3. Bitemporal eval against the same 30 × 2 = 60, with as_of=upload.timestamp constraining every tool call. Wall-clock ~35 min, cost ~$5. Output: bitemporal_run_1778913677.jsonl (60 rows). One substrate-state collision (PG duplicate fact-id) required a clean substrate reset before retry.
4. Five reports regenerated: accuracy.md, latency_cost.md, code_complexity.md, bitemporal_eval.md, failure_analysis.md, plus EVAL_NOTES.md with the full architectural rationale.

The eval is reproducible: ~/.aws/credentials with Bedrock access, bin/datomic-up && bin/shim-start && bin/pg-up, schemas, fixtures via the chronological loader, then the uv run python -m eval.* commands documented in the prototype README. The new smoke test (RUN_REAL_BEDROCK_SMOKE=1 uv run pytest tests/agent/test_real_bedrock_smoke.py) should run before any future eval pass — it catches the regression class that bit Phase 4 in seconds rather than after a 30-minute eval.

Notes on the eval methodology:

• Sample size: n=30 directional, not n=178 full corpus. The original 25-min/pass estimate underweighted multi-turn LLM latency by ~14×; a full corpus run is projected at ~16 hours wall-clock and ~$53 Bedrock spend. n=30 is enough to surface the bitemporal-correctness finding decisively and the accuracy gap directionally.
• Model: Claude Sonnet 4.5 inference profile (us.anthropic.claude-sonnet-4-5-20250929-v1:0). JSONL pins model_id and temperature per row for reproducibility.
• Substrate state hygiene: bitemporal eval persistence writes resolution facts with deterministic IDs derived from upload_id. Running bitemporal against a substrate that has already-persisted golden-eval resolutions for the same upload IDs causes a unique-key collision on PG. Fix: reset substrates between passes. Production would handle this differently (probably by including eval_pass_id in the fact ID, or by running the two passes in parallel against different namespaces).

What was built

Layout

Everything lives under one self-contained directory with zero imports from apps.*, refinery.*, or minecart.django_backend.*. A scope-isolation test enforces this on every commit:

minecart/prototypes/activity_intel_datomic/
├── README.md                       runbook (7-step lifecycle + idempotency notes)
├── bin/                            datomic-up/down, shim-start/stop, pg-up/down
├── datomic-shim/                   Clojure HTTP shim
│   ├── deps.edn                    com.datomic/local 1.0.291 + ring + cheshire
│   └── src/shim/core.clj           73 LOC: /tx /query /history /as-of /health
├── postgres_baseline/              Postgres container + schema
│   ├── docker-compose.yml          postgis/postgis:16-3.4 on port 55432
│   ├── init/                       enable postgis + pg_trgm
│   ├── schema.sql                  entities / events / facts / external_identifiers / etc.
│   └── indexes_prd.sql             PRD-prescribed v1 indexes
├── python/                         self-contained uv project
│   ├── activity_intel_prototype/
│   │   ├── agent/                  Bedrock client, resolver agent, prompts, tool types
│   │   ├── loaders/                PDI seed, Places seed, golden uploads
│   │   ├── persistence/            record_resolution_{datomic,pg}.py
│   │   ├── schema_datomic.py       :db/ident + :db/valueType schema
│   │   ├── tools_datomic.py        6 tools, Datalog + Datomic fuzzy rule
│   │   ├── tools_pg.py             6 tools, PostGIS + RapidFuzz + pg_trgm
│   │   ├── client_datomic.py       thin Python client to the Clojure shim
│   │   ├── client_pg.py            thin psycopg wrapper
│   │   └── config.py               env defaults (shim 8989, PG 55432)
│   ├── tests/                      ~80 tests (see Testing report below)
│   ├── eval/                       run_golden_eval, bitemporal_eval, generate_reports
│   ├── scripts/                    schema deploys, fixture loaders, scope-isolation CLI
│   └── pyproject.toml
└── reports/                        empty (eval not run)

The agent

One AccountResolver agent with one system prompt, one skill markdown file, one set of proposal types, and one tool-call loop — all shared across both backends. Only the six tool implementations and the resolution-persistence recorder fork per backend.

Substrate parallelism

The five planned reports

The full pipeline ends in generate_reports.py, which is implemented but has no JSONL to consume. Once the eval runs, this script produces:

reports/
├── accuracy.md           accuracy@1, accuracy@5, Datomic vs PG with the disclaimer
├── latency_cost.md       p50/p95 latency, token usage, USD, top-10 outliers
├── code_complexity.md    LOC per backend, Datalog-vs-SQL samples, measured as-of sites
├── bitemporal_eval.md    unconstrained vs as-of-constrained false positives per backend
└── failure_analysis.md   top misses, top-5 candidates, tool traces, miss categorization

The strangler approach

The Martin Fowler "strangler fig" pattern describes how a new system grows alongside a legacy system and incrementally absorbs its responsibilities. Branches of the new system wrap and gradually replace branches of the old; eventually the old trunk is dead and can be cut away with no rollout fanfare. For activity intelligence, the legacy "trunk" is the existing Django model: rows in tenant-scoped tables, mutated by save() callers, queried directly by reads.

Production posture (where this is heading)

The Activity Intelligence Platform PRD specifies an entity/event/fact AI layer sitting beside the Django model:

┌────────────────────────────────────────────────────────────────────────┐
│  Today (legacy trunk)                                                   │
│                                                                         │
│  HTTP → Django views → models.Save() → Postgres rows                    │
│                                  │                                      │
│                            (in-place mutation; no event log)            │
└────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────┐
│  Strangler posture (target)                                             │
│                                                                         │
│  HTTP → Django views → models.Save() ──► Postgres rows  (unchanged)     │
│                              │                                          │
│                              └─► state_change event ──► outbox          │
│                                                          │              │
│                                                          ▼              │
│  AI layer reads from outbox ─► entities / events / facts substrate      │
│                                  (Datomic OR Postgres+PostGIS)          │
│                                  │                                      │
│  Agents (AccountResolver, Restatement, etc.)  ◄──┘                      │
│  read facts as-of(t); write resolutions back as events + facts.         │
└────────────────────────────────────────────────────────────────────────┘

The strangler fig is that outbox feed. Existing Django writes don't change shape; a save() override emits a state_change event to an outbox table; the new AI layer subscribes to the outbox, builds its own entity/event/fact model, and progressively takes over the read surface for AI-mediated workflows (entity resolution, restatement, account matching, etc.). The legacy Django model keeps living for everything else.

What this prototype tests, and what it deliberately skips

This prototype is not the strangler. It is the question one rung upstream of it: once the AI layer is being fed real state changes, which substrate should host the entity/event/fact model — Datomic or Postgres+PostGIS? Both implement the same shape; the prototype measures which one is cheaper to operate, cheaper to query, more correct under as-of constraints, and more ergonomic to write tool implementations against.

To keep the question clean, the prototype deliberately skips the strangler plumbing:

• No Django save() override. Loaders write events directly from the PDI fixture. The outbox / strangler posture is a production concern that doesn't affect the substrate comparison.
• No production data coupling. grep -rn "from apps\." prototype/ returns nothing. Enforced by the scope-isolation test in bead 025.
• No projection-sink design. The PRD's "named projection views" are a separate spike.
• No permissions / Inquiry queue. The agent auto-resolves its highest-confidence proposal. PR #1778's ProposalPolicy / ask_user / ask_manager layer is deferred.
• No multi-tenant story. Single tenant (PDI Federated). PG RLS / Datomic tenant key conventions are a separate question.
• No MCP / external API surface. The eval harness reads the substrate directly.

What's left is a clean substrate decision: a single workload (account resolution), a single tenant's data (PDI Federated, 2,634 accounts), a single agent loop, two substrates, two reports.

What we want out of the strangler when this prototype lands somewhere

Once the substrate decision is made, the strangler implementation should be the smallest possible delta: a Django save() override that writes a row to an outbox table, a consumer that turns those rows into append_event/assert_fact calls against whichever substrate won, and a feature-flagged toggle in the AI-mediated views that switches reads from the legacy model to the new as_of() projection. Three pieces, each with a clean rollback. The substrate the prototype picks is the only piece that's hard to undo later.

Testing report

What ran successfully

73+ tests passing across 21 test files. All were unit or integration tests with mocked LLM calls — no real Bedrock requests, no real model billing.

Directional eval (n=30 sample, real Bedrock)

• Unconstrained pass — 60 resolutions written to run_1778896874.jsonl; latency, cost, tokens, model_id, temperature all recorded per row.
• Bitemporal pass — 60 resolutions written to bitemporal_run_1778898804.jsonl; every tool call constrained to as_of=upload.timestamp.
• All five reports generated — accuracy.md, latency_cost.md, code_complexity.md, bitemporal_eval.md, failure_analysis.md.
• Full n=178 corpus run — deferred. The original 25-min/pass estimate was ~12× under reality; a full run would take ~10 h wall-clock and ~$60 Bedrock spend. The n=30 sample is directional, not definitive.
• Accuracy measurement — eval ran cleanly, but every resolution returned predicted_account_id: null. Likely cause: the two bedrock_client.py hot-fixes applied mid-run tightened the outbound schema and probably broke the inbound tool-result reader the agent loop depends on. See §Learnings.

Code-complexity snapshot (from reports/code_complexity.md)

The eval-generated report measures substrate-specific LOC and as-of site counts directly from the source. Numbers match the prediction made in the pre-eval explainer:

Datomic's recorder is 30% smaller (145 vs 204 LOC) and the as-of discipline costs roughly half as much code (18 vs 34 LOC). Postgres pays the recorder difference for every event/fact pair (explicit provenance jsonb on both rows) and pays the as-of difference for every tool query (two helper clauses appended to each WHERE block). Both are unavoidable consequences of the substrate model; both compound with feature scope. Datomic's overall prototype LOC is slightly higher (655 vs 596) due to the 73-line Clojure shim and the Datomic schema layer — that overhead is fixed and one-time.

What we learned from building this

Even without the eval data, the build process surfaced a number of useful, durable observations.

About Datomic in 2026

• Datomic Pro free since 2023 lives on Maven Central. com.datomic/peer 1.0.7075 resolves cleanly; you do not need Cognitect credentials. The older com.datomic/dev-local 1.0.243 artifact was behind a private repo and has been discontinued.
• The modern dev/local artifact is com.datomic/local 1.0.291 (Maven Central), with datomic.client.api and :server-type :datomic-local. The plan-revision phase identified this swap before any code was written, but the synthesized deps.edn still shipped with dev-local and had to be patched on first clojure -P failure.
• Source-time :db/txInstant matters for bitemporal correctness. Datomic's (d/as-of db t) operates on transaction time. If you load historical fixtures at "now" rather than at each record's created_at, as-of reads will silently include data that shouldn't be visible. The fix: every fixture-load transaction sets :db/txInstant explicitly to the record's source time.
• The Clojure HTTP shim returns :db/txInstant as a Java Date.toString() with timezone abbreviation (e.g. "Tue Nov 25 13:26:57 PST 2025"), not an EDN #inst. Round-tripping requires explicit PST/PDT-to-UTC conversion on the Python side. Codex caught this on bead 009 (PDI tenant seed loader) and added a small conversion helper.

About the toolchain surface

• Three local toolchains needed: Clojure CLI, Java 21, and Docker daemon. All three needed operator intervention during the run — Clojure was missing entirely, Java 21 was installed but unlinked, and the Colima VM disk-attachment got wedged ("failed to run attach disk colima, in use by instance colima") and required a colima delete -f + restart to clear. Codex correctly refused to install system-wide packages and escalated.
• Brew's cask metadata was corrupted on the host. Brew formula installs worked fine; cask-based installs (Datomic, codex CLI) failed with undefined method 'to_sym' for nil. Worked around by installing codex via npm.
• docker compose on this host needed the plugin path wired up explicitly via cliPluginsExtraDirs in ~/.docker/config.json after brew install docker-compose — the install otherwise only puts the binary in /opt/homebrew/bin/docker-compose, which won't satisfy a script that uses the modern docker compose ... form.

About the agentic build process itself

• Codex self-debugged real bugs that the test suite caught. Two examples: the PDI fixture had duplicate external_identifier keys, and a Datomic schema deploy test was missing a process timeout that surfaced as a hang. Both were fixed without operator nudging.
• The Phase-3 supervisor's auto-approval rules ran 14+ approvals before needing human input. What stalled the run was the bead-002 Datomic-dep block, the bead-003 missing-Compose block, and the bead-003 Colima disk-lock block — all toolchain, none agent-logic.
• Codex's option-2 "always-allow" prefix can be too broad. When the supervisor was offered "always-allow curl", it correctly fell back to option 1 for every individual localhost curl rather than blanket-approving network access. Worth knowing: the supervisor's judgment is needed even with always-allow available.
• Most beads after the toolchain phase committed in 3-5 minutes each. Once the substrate plumbing was in (beads 001-005 plus the Python project in 006), the data-layer (007-011), tool (012-014), agent (015-018), persistence (019-020), and eval (021-022) beads each took roughly one supervisor poll cycle.

About the substrate comparison (the headline findings)

• "Free bitemporal correctness" needs a footnote. The pre-fix n=30 eval measured 7 impossible-match FPs slipping through Datomic's (d/as-of db t) constraint while Postgres held all 16. Phase 6 diagnosed the leak: Datomic has three time dimensions for batched historical loads (tx-time, source-record time, query as-of), and (d/as-of db t) only filters on tx-time. After adding explicit source-time Datalog predicates (:entity/created-at, :fact/valid-from), Datomic prevents 17/17. The architectural argument survives contact, but only after the codebase explicitly enforces the gap the loader's batching opens.
• The accuracy gap is real and substrate-attributable. Three runs across two fixes: pre-fix Datomic +10pp (83/73); Phase 6.5-A intermediate −3.33pp (73/77, attributed to Class C agent-loop variance on three duplicates); post-AA tool-parity +6.67pp (90/83). Both backends moved up with the shared Python scorer, but the Datomic lead persisted under identical scoring. The +10pp pre-fix signal was a mix of substrate (~67% of it survived parity) and tool-implementation ergonomics (~33% of it collapsed when the Datalog rule was replaced). At n=30 the confidence interval on the +6.67pp gap is wider than the gap itself — directional but not decisive.
• Datomic's code-size win is bounded but real. 30% smaller recorder (149 vs 211 LOC). As-of code is 39 vs 34 LOC after the source-time fix. The 73-LOC Clojure shim is a one-time operational tax Postgres doesn't pay, and it remains the strongest argument against bringing in Datomic for a team without Clojure experience.
• Cost and latency moved between runs. Phase 5: Postgres ~4-12% better; Phase 6.5-AA: Datomic ~16-18% better. Both runs sit inside agent-loop-variance bands. Substrate isn't the cost needle — agent reasoning loops are.
• Disambiguation is still the dominant failure mode, but Datomic loses fewer of them. Post-AA: Datomic 1 disambiguation_failure, PG 4. Both backends now see the same candidate pool with the same scorer; the agent picks wrong less often on Datomic. The mechanism isn't fully understood at this sample size — likely related to how the Datalog rule and the pg_trgm pre-filter ordered candidates before the shared Python scorer ranked them — but the directional signal is consistent across phases.

About what "the prototype taught us" beyond the substrate decision

• Trace data is not fact data. The most durable architectural insight from the build came from Datomic's RuntimeError: Item too large blocking the eval mid-run on the full tool_call_trace payload. The fix wasn't "make Datomic tolerate larger items" — it was "the trace doesn't belong in the substrate at all." Both backends now store a compact trace_summary; the full trace lives in the JSONL output. In production, it lives in Langfuse. This is a cleaner architecture regardless of substrate choice.
• Eval persistence across passes needs a story. The bitemporal eval crashed on the second run with a Postgres unique-key violation because resolved_to_account_id:<upload_id> is deterministic; running the bitemporal pass against a substrate that already has the golden-eval resolutions for the same upload IDs collides. Fix in the prototype: reset stores between passes. Fix in production: include eval_pass_id or resolution_at in the fact ID. This is a design pattern, not a bug — fact-ids derived purely from "what is being said about whom" assume a single source of truth at write time.
• Substrate-state idempotency is non-trivial. Datomic's monotonic-basis behavior (a transaction's :db/txInstant must not be earlier than the database's current basis) means you can't reload historical fixtures piecewise; you have to merge them chronologically and submit in one ordered stream. The prototype added a one-off chronological loader at .runtime/load_datomic_chronological.py. In a production strangler this becomes the outbox's job — it already sequences events chronologically by source-system clock.

About the mocked-tests-vs-real-API gap (the biggest learning)

• Unit tests with mocked Bedrock missed the API schema. The mocks accepted whatever shape bedrock_client built; real Bedrock Converse rejected two specific things: isError: true on toolResult blocks (not in the API schema), and toolResult.content[].json values that weren't JSON objects (e.g. lists or scalars from tool return values). The unit tests passed against the mocks. The real API failed loudly on the first eval attempt.
• The hot-fixes were schema-correct but probably broke the inbound path. Codex's two surgical fixes — _strip_non_api_tool_result_fields and the {value: ...} wrapper — applied only on the outbound request path, with a small helper. The mocked test path was left untouched (13 tests still pass). But the n=30 eval shows every resolution ends with predicted_account_id: null and empty top_candidates, which is the shape an agent loop produces when it can't read tool results back. The fix tightened the outbound schema; the agent loop almost certainly needs a symmetrical inbound fix (or the wrapping needs to be undone on the way back).
• This is the kind of bug only real-data evals catch. The strangler-pattern insight applies here too: a prototype that's only ever validated against its own mocks is a prototype that has never met production. The eval did its job — found a real bug the unit tests couldn't have surfaced.
• Mitigation for next time: add an integration test that uses the real Bedrock API for one transcript (one resolution, ~$0.10) before kicking off the full eval. Catch schema breakage in seconds, not 25 minutes in.

About the planning pipeline

• The two-model planning pipeline (claude + codex revisions, synthesized into one plan) caught real bugs the single-model draft missed. The Datomic txInstant ordering issue, the eval-JSONL top_candidates requirement, and the Bedrock Converse vs invoke_model decision were all unique finds from one of the two models that the synthesis preserved.
• The bead validator's enrichment pass mattered. Five of 25 beads (002, 003, 013, 018, 024) needed test-case enrichment before they were standalone-executable; the validator caught this before codex started, not during.
• "Best-of-each-backend" is honest framing. A pure-substrate comparison would force identical algorithms across Datalog and SQL, which production wouldn't do anyway. The planned accuracy.md disclaimer makes the framing explicit:

  "Headline accuracy reflects substrate + tool-implementation ergonomics, not pure substrate, due to best-of-each-backend decision (PostGIS vs Datalog)."

Next steps

To make a defensible substrate decision (do these in order)

1. Tool-parity ablation completed (Phase 6.5-AA, $17.47). Result: the Datomic accuracy lead is substrate-attributable at +6.67pp (90.00% vs 83.33%) with identical Python scoring on both backends. The +10pp pre-fix gap was roughly two-thirds substrate / one-third tool-implementation.
2. Sequential extension to n=80 completed (Phase B, $25.92). Result: the Datomic accuracy lead held at +5.00pp (90.00% vs 85.00%) through the sample expansion. Bitemporal correctness still 100% on both backends (Datomic 50/50, PG 46/46). The signal is stable and substrate-attributable, but the n=80 CI (±5pp) is still wider than the gap.
3. Run the n=178 full corpus (priority: high, ~$30 incremental Bedrock, ~4 hours wall-clock per pass). Add another --offset 80 --limit 98 to extend to the full dataset, then aggregate. The n=80 → n=178 jump shrinks the CI on the accuracy gap from ~±5pp to ~±3.5pp, which would make the +5pp Datomic lead statistically significant if it holds.
4. Operational ramp planning if Datomic is reconsidered. A second eval workload (not just account resolution) where the entity/event/fact model's schema flexibility lets the agent do something on Datomic that would require schema migration on Postgres. The single-workload eval undersells Datomic's schema-flexibility argument.
5. Make a write-up of the production strangler design (priority: medium, ~half day). One pager: Django save() override → outbox table → AI-layer consumer → entities/events/facts in PG. Identify the Django models that need save() overrides (Account, Contact, Note, UserUpload at minimum). Flag the Inquiry/permissions layer from PR #1778 as the next reviewable design.

If we proceed with Postgres (the recommendation)

1. Promote the prototype's PG schema to a real Django app. Move postgres_baseline/schema.sql + indexes_prd.sql into a Django apps/activity_intel/migrations/. Reuse Supabase pgvector extension if semantic search becomes a requirement.
2. Build the outbox. One outbox_events table, one Django post_save signal handler, one Celery worker that drains it into the new events table. Idempotency via dedup on (source_system, source_record_id).
3. Wire the agent to Langfuse for traces. Replace the prototype's local trace JSONL with Langfuse spans. tool_call_trace moves to Langfuse; trace_summary stays in the events.provenance jsonb per the Phase 5 refactor.
4. Reuse the prototype's agent loop, tool surface, and Bedrock client as the starting point for the production v1 agent. The six-tool surface measured here is the right one; the agent prompt and proposal types are reusable. The only swap is the tool implementations point at the real apps.mc Account / Contact / Note tables instead of the prototype's entity/event/fact schema (and then strangler-fig those reads onto the new schema over time).

If we revisit Datomic (the bitemporal fix landed, accuracy is contested)

1. The bitemporal eval was already re-run at n=30 (Phase 6.5-A). Datomic's prevented count went 9 → 17, confirming the fix. The accuracy lead reversed within agent-trajectory variance. The data is no longer the decisive argument against Datomic — the operational/strategic factors are.
2. Bring up Datomic Pro Cloud as a target (not just dev-local). Bring-up complexity, ops burden, and per-instance cost should be factored into any re-decision. The 73-LOC dev-local shim is not what production would run.
3. Plan the Clojure operational ramp. Expanding beyond the prototype shim requires Clojure expertise in the team. Either hire/train one person who owns it, or keep the Clojure surface deliberately small.
4. Discipline source-time tracking in any production loader. The Phase 6 finding generalizes: whenever historical data is loaded into Datomic in batches, :db/txInstant can drift from source-record time, and (d/as-of db t) alone won't catch the gap. Production loaders need to either (a) write tx-time = source-record time on every transaction (paying the per-batch monotonic-basis cost), or (b) carry explicit source-time predicates on every read. The prototype now does (b).

Independent of substrate (do regardless)

1. Fix the agent's confidence-threshold bug revealed by the 0bd983e2… "Rusty's Repair LLC" case (top-5 candidates include 3 copies of the correct match at 0.89; agent emits null). This is pure agent logic, not substrate, and it's costing accuracy@1 on both backends.
2. Add the real-Bedrock smoke test to CI (gated by RUN_REAL_BEDROCK_SMOKE=1 env so it doesn't run on every PR, but a once-daily pass against a single transcript would catch schema regressions long before another full eval pass.
3. Implement the Permissions / Inquiry layer from PR #1778 before this is ever production-facing. The current prototype auto-accepts the highest-confidence proposal; production needs ask_user / ask_manager branches.
4. Add a no-GPS-signal fallback. The 1 transcript without GPS missed on both backends. A name-only resolution path with a higher confidence threshold would handle this class better.