The audit

In August 2026, the shipped database went through a week-long audit campaign — a sequence of increasingly deep passes over data that was already live, asking one question at every level: does each piece of data actually say what it claims to say? Is this "chain" a chain? Is this pin the restaurant the Reddit quotes describe? Is this mention attributed to the right same-named business? The campaign ran on batches of LLM agents (Opus for judgment, with a second frontier model as an adversarial verifier), and every change it produced lives in a checked-in, rerunnable "apply list" — nothing was hand-edited into the database.

The end state: all 5,224 restaurants and all 466,858 mentions were audited; roughly 97% of mention groups verified clean; and every correction that shipped was independently confirmed by two models before it touched data.

A batch review classified every restaurant's chain status (household-name chains are hidden; NYC-identity chains and little-known "outpost" chains stay visible), then a Google Places sweep enumerated every location of all 1,270 visible chains — producing the multi-pin maps on chain pages (~4,000 locations).

• The zero-match investigation: 121 "chains" whose location search confirmed nothing turned out to be four different stories — closed businesses (Bar Boulud), name variants the matcher was too strict about (Anita Gelato appears on Google as "Anita La Mamma del Gelato"), records that were never NYC chains at all, and businesses Google simply doesn't list well.
• The outside-NYC flag: the sweep exposed a whole class of records for famous elsewhere restaurants — Lou Malnati's, Dishoom, Franklin's, Noma — that exist only because Redditors mention them in comparisons ("is anywhere here as good as…?"). Rather than deleting real mentions, these carry an outside_nyc flag: hidden from browsing and search, detail pages kept. Metro-adjacent places (Blue Hill at Stone Barns, NJ commuter towns) deliberately stay visible.
• Edge case that shaped the tooling: "Smor" and "Smør" were the same Danish bakery split into two records — the slashed ø survives Unicode NFKD normalization, so every earlier dedup pass had missed it. Name folding now handles ø/đ/ł/æ/œ explicitly.

The database's name column holds the community's shorthand ("Katz's") and doubles as the stable join key for every apply list — so it never changes. Public-facing names live in a separate display_name ("Katz's Delicatessen"), and search matches both. All 1,609 records whose Google name differed from their community name were individually verified by agents before adoption; 957 display names shipped.

The valuable byproduct: for two dozen records the Google name described a different business entirely — the vegan spot Kernel was linked to an IT consulting firm, Perry St to a bare street geocode, Mimi's Pizza to a frozen-yogurt shop. Name-only matching in the original enrichment was the root cause. Each was re-linked (or marked closed / deliberately pinless) with fresh evidence, where health-department permit data repeatedly proved more reliable than either Google or the restaurants' own stale websites — one "still open" website carried a 2009 copyright while the liquor license showed the space operating under a new name.

Machine-flagged candidates (shared place IDs, name collisions after aggressive folding) produced 56 pairs, each adjudicated by an agent with full mention context. 17 merged (TAO Downtown into Tao, 10000 Coffee into Ten Thousand Coffee, the Piccola Cucina venues into their brand record); 35 were confirmed distinct — every deliberately disambiguated pair like the two Tom's Restaurants held. The decisive evidence, over and over, was the raw mention text: "Sammy's Halal" and "Sammy's Noodle Shop" look identical as display names but never as raw mentions.

The deepest pass asked the linking question about literally every mention: is this post or comment attributed to the correct restaurant? Reviewing 466,858 mentions one-by-one would have been wasteful — ~75% are bare name-drops whose only evidence is the name string itself — so the unit of judgment was the (restaurant, raw-name) group: all 10,500 distinct pairings of a restaurant and a mention spelling, each judged once with its complete quote evidence inline. Judge the rule and you've judged every mention that rode in on it; but because agents read every quote verbatim, single strays still surfaced (one Chicago hot-dog mention inside Pat's; one mention of a Bogotá tasting room inside a Williamsburg pizzeria's group).

• Risk-tiered scheduling: a deterministic scorer ranked every restaurant's a-priori mismatch risk — shared or generic names, few mentions, crowded Google results (using the match-margin data the original pipeline had stored), discounted for fame and distinctive names ("Sunken Harbor Club" is nearly impossible to mismatch; "Jonny's" is an accident waiting to happen). High-risk records were judged first.
• Results: ~97% of groups confirmed consistent. 340 findings: mentions on the wrong same-named restaurant (Joe's Shanghai quotes sitting on Joe's Pizza), records whose pin was a different business, one record that wasn't a restaurant at all ("N1" — the menu number of Xi'an Famous Foods' cumin lamb noodles), and ~200 groups too ambiguous to rule on without more context.

No finding was applied on one model's word. A second frontier model re-reviewed all 340 findings adversarially — instructed to refute, not to defer — and rejected 45 of them. The rejections clustered into one systematic blind spot worth naming: renames and second locations. The first model repeatedly read "different name at this address" as "different business" when it was actually the same operator — a renamed flagship, a brand's second location, or in one case a chain (Zaxby's) that had genuinely opened its first NYC location after the "we need one here" threads were written. Only findings both models agreed on were applied.

The verification ran blind — no knowledge of earlier passes — which made agreement meaningful in both directions: it independently re-discovered every known mixed record, confirmed 17 of 20 earlier link repairs as now-consistent, and caught a live bug in our own tooling (an apply script matching by name had stamped same-named sibling records with each other's identities: the Murray's Cheese record briefly displayed "Murray's Bagels"). Every apply script now matches on id and name together.

The ~200 ambiguous groups (8,478 mentions on generic names like Tony's, Rudy's, Milano) got the full treatment: every mention individually re-read with its complete Reddit thread context — post title, selftext, the comment's full text, its parent comment. That context is what converts an unroutable "jon's is great" into a routable one when the thread is titled "Best pizza in Bay Ridge?".

• 85% stayed put — the honest outcome for genuinely generic mentions, under the rule that an unroutable mention stays where the linker put it (churn is worse than ambiguity).
• 766 moved and 275 detached (out-of-scope businesses: someone's hometown spot in a visitor thread), each individually confirmed by the second model before applying — 1,041 of 1,053 proposals survived verification.
• Durability: these per-mention decisions are keyed by Reddit post/comment IDs rather than database row IDs, so they survive full rebuilds and will transfer directly into the next extraction run.

A follow-up sweep asked a simpler question: why do 37% of mentions have no quote? The answer split into two causes — one deliberate, one a bug.

• Deliberate: quotes are rating receipts. The extraction prompt said it outright — "quote is null exactly when rating is null." A quote's job is to be the evidence behind a rating, the commenter's own words backing the score. Mentions that name-drop a restaurant without judging it (bare lists, logistics, "I'm going there tomorrow") were never supposed to have quotes, and after re-examining the question we kept that rule: this site exists to answer "is this place good?", and a quote that carries no judgment would dilute what a quote means. ~150k unrated mentions stay quoteless on purpose.
• The bug: typography vs. a byte-exact validator. For rated mentions, the pipeline verified each quote with a strict substring check against the source text. But models normalize typography when quoting — paragraph breaks become blank lines, curly apostrophes become straight — so ~20,000 genuine quotes failed byte-comparison and were silently nulled. The validator confused "not byte-identical" with "not verbatim."

The repair came in three parts. First, a free recovery: the original model outputs still existed on disk, so re-validating them with tolerant matching (forgiving only whitespace runs, quote/dash/ellipsis variants — never wording) recovered 9,841 quotes at zero model cost. Crucially, the tolerant match is used only to locate the passage; what gets stored is the source's own exact characters, so recovered quotes are strictly verbatim — the commenter's bytes, not the model's rendition. Second, a re-extraction pass over the remaining ~11,000 rated mentions whose original quotes were genuinely paraphrased: fresh extraction under an exact-copy prompt, every output machine-verified as a byte-exact source substring (8,253 passed; anything paraphrased was dropped again, correctly). Third, the upstream fix: the pipeline's validator now does the same locate-tolerantly-store-exactly dance, so the drop can't recur on the next extraction run.

End state: 99.1% of rated mentions carry a verbatim quote (up from 93.4%), and the remainder are genuinely unquotable — ratings inferred from bare ranked lists, where there are no words to quote.

A fair post-mortem question: was all this necessary because the original linking step wasn't QA'd enough? Mostly no. Breaking the campaign's findings down by root cause:

• Name-only matching in the Google enrichment (most wrong links). The original match step searched a name and took the best hit without checking it against what the mentions said. Kernel→IT-firm, Mimi's→froyo, Murray's→Bagels all came from there. The linking triage QA'd which restaurants exist; nobody QA'd whether the Google join agreed with the mention evidence — that's a different step, and it got less scrutiny than triage did.
• World drift (~a third of findings). Closures, renames, relocations, and even openings (Zaxby's!) after the data was captured. No amount of QA at build time prevents this class — it's freshness, not correctness, and it will keep accruing forever.
• Inherent ambiguity resolved with too little context. The mixed records (SEA, Blue Ribbon, Cha Chan Tang) and generic-name misroutes were cases where a name string genuinely underdetermines the referent. The original audit round did catch a lot of these (the 48,000-judgment ambiguous-name sweep documented under Restaurant linking) — this campaign's pass just had two advantages: full-quote context by default, and adversarial two-model verification. Same context rule learned back then, applied harder.
• Our own tooling (small but real): the name-keyed apply script, the ø-normalization gap.

So the honest framing: linking was QA'd well; the joins downstream of linking (the Google match) and the passage of time were the gaps. The lesson isn't "QA more" generically — it's every join needs its own QA, with at least the context of the step it checks.

• Precision over recall, at every level. A wrong "correction" costs more than a miss. Uncertain verdicts default to no action; disagreement between the two models defaults to no action; unroutable mentions stay put.
• Reports, then vetted lists, then applies. No step mutates the database directly. Agents produce findings; findings compile into checked-in decision files; small idempotent scripts apply them. Any decision can be reversed by editing the file and rerunning.
• Evidence hierarchy: raw mention text and thread context first, health-department permits and liquor licenses for existence questions, Google listings after that, and restaurants' own websites last (they go stale the moment a place closes).
• The context rule, reaffirmed: a checker must see at least as much context as the process it checks. Group-level judgment got full quote lists; mention-level judgment got full threads; and the known failure mode — confidently "fixing" things that were right — showed up exactly where context was thinnest (renames, second locations) until the second model's pass caught it.
• Never link a successor. When a restaurant dies and something else takes over its address (or its Google listing), the record is marked closed with no pin — the new tenant is not the restaurant the mentions describe.
• Deliberately pinless is a valid state. Some operating businesses (market stands, unlisted counters) have no usable Google listing; linking a lookalike just to have a pin is worse than no pin.

By the numbers: ~500 agent runs across the campaign, ~45M tokens of model judgment; 17 duplicate records merged, 17 missing records created, 957 display names set, ~60 place links repaired, ~50 closures confirmed, ~1,350 mentions re-attributed or detached, ~18,100 verbatim quotes recovered or re-extracted — and the rest of the database independently verified as already correct.