Skip to content
Franklin Martinez
All work

docmatch

Invoice reconciliation with vision models, deterministic gates and a benchmark in CI

Abstract. docmatch reads invoices with vision language models, checks what they read with deterministic code, matches each invoice against its purchase order and receiving record, and sends anything that doesn't add up to a person. Every stage has its number in a benchmark that names the commit behind it. In progress.

Extract

Gate

  1. received
  2. extracted
  3. validated
  4. resolved
  5. matched
  6. approved | needs_review

Next: resolve each line to the catalog, then match the invoice against its purchase order and receiving record.

Fig. 2. A synthetic invoice from docmatch's CI corpus, with its labeled fields drawn where they sit on the page, then the gate's two rules. The gate needs no model and no label. A reading that fails is kept and never retried, because retrying would shop for a passing answer.

Why it exists

Accounts-payable teams still receive most invoices as PDFs, scans and phone photos. Turning one into a ledger entry is three problems: reading it, checking it against what was ordered and received, and deciding whether a person needs to look. Most public projects stop at the first with a single model call. docmatch treats reading as a replaceable backend and puts the engineering into the other two.

Models read, code decides

The pipeline is an explicit state machine in Postgres: received, extracted, validated, resolved, matched, then approved or needs review. Models only structure text. The arithmetic and every decision are deterministic code.

The validation gate (Fig. 2) holds only rules the labels themselves pass: totals agree and dates are sane. I tried line-total reconciliation, tax arithmetic and identifier checksums, and left them out, because they fail too often on the labeled data or apply to almost no documents. A value the model could not read never fails a rule, and a reading that fails is kept, never retried, because retrying would shop for a passing answer.

Phases with numbers, not dates

Each phase ends with a number in the README, and the next phase does not start until that number exists.

Extraction. Three backends compared on a pinned subset of 100 documents (Fig. 5). The cheapest model also scored the highest field F1.

Matching. A generator injects labeled discrepancies into purchase orders and receiving records. On the injected set, precision and recall are near perfect for every discrepancy type. End to end, from the image, precision drops by half, and the README says exactly why: an unpaired line misread by extraction becomes both a false extra line and a false missing line.

Entity resolution. Trigram, vector, hybrid, and hybrid with a cross- encoder reranker, over a vendor catalog. The rule for keeping the reranker was written before the number. It scored below the plain hybrid, so it was dropped.

Pipeline. The full loop from upload to approval over the HTTP API, with the cheapest backend.

Guarded in CI

A regression gate compares field and line-item F1 for every backend against the merge base, and fails a pull request that lowers either.

What I cut, on purpose

Celery and Redis (a Postgres-backed worker is enough on one node), judge-model eval frameworks (extraction has ground truth), Langfuse (self-hosting it brings back Redis, and the cloud would put document contents on a third party), and graph orchestration, settled by an ADR in favor of plain code.

Next

Packaging: a reproducible entry point so anyone can regenerate a benchmark row from a clean checkout, and a failure analysis of the top error classes. Then a Dominican adapter: e-CF XML ingestion, RNC and NCF checks, and the 606 purchase report.

Backend Field F1 Cost per doc Commit
gemini-3.1-flash-lite pages at 1600 px 0.615 $0.00203 28d0738
gpt-5.6-luna pages at 1600 px 0.588 $0.00344 42e69fa
Azure prebuilt-invoice API 2024-11-30 0.556 $0.01120 d3bb01e
Fig. 5. Three extraction backends on a fixed subset of 100 pinned documents. Each row links the commit that produced it.

Talk about this project