Growth path

Four stages, each with the trigger that unlocks it. Stage 1 is the only one to build now.

Growth path

Build stage 1. Do not build stage 2 until its trigger fires. Each trigger is an observation, not a hunch.

  STAGE 1 — NOW                                        under $0.01/mo
  ┌──────────────────────────────────────────────────────────────┐
  │ Private S3 bucket, versioning, SSE, one IAM user.            │
  │ Markdown + YAML frontmatter. Scheduled aws s3 sync into a    │
  │ local mirror. Agents use native Read / Grep / Glob.          │
  │ Nothing custom.                                              │
  └──────────────────────────────┬───────────────────────────────┘
         TRIGGER: a query returns more hits than an agent can
         usefully scan, or the file count passes ~2,000, or the
         listing alone starts eating context.
                                 ▼
  STAGE 2 — INDEX                                                $0
  ┌──────────────────────────────────────────────────────────────┐
  │ A build script walks the local files and writes              │
  │ transcripts.db — SQLite, FTS5, BM25 ranking, frontmatter as  │
  │ columns. Derived and disposable; rebuilt from S3 in seconds. │
  │ Files stay the source of truth. Agent gets one more tool.    │
  └──────────────────────────────┬───────────────────────────────┘
         TRIGGER: three logged recall misses that survived
         multi-query grep attempts.
                                 ▼
  STAGE 3 — SEMANTIC                                    pennies/mo
  ┌──────────────────────────────────────────────────────────────┐
  │ Chunk + embed. Vectors in sqlite-vec beside the FTS5 index,  │
  │ or S3 Vectors if they must be shared. Hybrid BM25 + vector,  │
  │ both feeding one agent tool.                                 │
  └──────────────────────────────┬───────────────────────────────┘
         TRIGGER: a NON-agent consumer — a client-facing app, a
         dashboard, concurrent humans needing sub-second queries.
                                 ▼
  STAGE 4 — SERVER                                       $0–$25/mo
  ┌──────────────────────────────────────────────────────────────┐
  │ Postgres. Loaded from files. Files still source of truth.    │
  └──────────────────────────────────────────────────────────────┘

Reading the triggers

Stage 2 is a ranking problem, not a speed problem. Grep over this corpus is instant and stays instant. What breaks is that grep returns hits in file order with no relevance signal, so 200 hits give an agent no basis to pick. FTS5’s BM25 fixes that for nothing.

Stage 3 is a recall failure, not a size. Someone asks about a moment using words the transcript does not contain. Grep cannot find it. But a competent agent generates several query variants before giving up, and that recovers most vocabulary misses. So the trigger is repeated misses that survive multiple queries. Log them. Three real ones justify embeddings. Embedding cost is a rounding error and must not drive the decision — the index infrastructure is the cost.

Stage 4 is never an agent’s request. An agent asking for Postgres is an agent that would rather have files.

Do not build

Iceberg or S3 Tables — cost inversion on small objects, and they buy ACID, schema evolution and time travel that append-only blobs never use. OpenSearch, any generation. Kendra. DynamoDB — no full-text search and you would build search on top anyway. A managed knowledge base. A chunking pipeline. A vector index before a recall miss is observed.

What grep genuinely cannot do

1. Vocabulary mismatch. The word is not in the document.
2. Ranking under many hits. No relevance order.
3. Synthesis across more text than fits in context.
4. Predicates on facts that live only in the prose — “meetings where the deal size exceeded a threshold” is not greppable.

Point 4 argues for extraction, not storage: an LLM pass writing structured fields into frontmatter. Do the extraction into the files. The database stays optional after that.