Audit, part 2

With the linking and double checking done, the data was finally refined enough for me to import the restaurants into my web app.

However, I quickly realized I was not done, not by a long shot. My previous audit step DID improve the quality of the output, but I could tell there were still errors while exploring the web app and viewing the data in the new interface.

I wanted to make the results as error proof as possible, so I conducted several more passes with LLMs to handle specific edge cases. However, this time I experimented with a new approach — using my Claude Max subscription, instead of OpenAI batch scripts. I ended up preferring this new approach, given it was substantially cheaper and more in line with my current workspace.

My general strategy with Claude Code was to write a dynamic workflow (link here) that spawned Opus subagents, which would use internet access + context from Reddit to clarify any edge cases, and I would have Fable review the ones flagged in this process. I discuss this more in the What I learned section.

I was expecting my original pipeline to catch many of these errors, so my second audit run was ad-hoc, and I didn't have a concrete plan. With that said, here are some of the edge cases I had to address:

• Chains. There were several chain restaurants represented such as McDonald's, Burger King, etc. I didn't want big chains to be represented, since no one cares about reviews for a chain they can find anywhere, but I DID care about local NYC chains, or small chains from other areas that had an outpost in NYC.
• Incorrect display names. A lot of restaurant names were inferred from Reddit comments instead of the Google results. While normalizing this, the agents uncovered some restaurants which were matched to completely different businesses with similar names.
• Duplicates. Some restaurants had duplicates because they slipped past my original fuzzy matching algorithm. For example, 10000 Coffee and Ten Thousand Coffee.

I also did a full review pass on pairs of names + restaurant matches in my database, using the multiple spawned agent approach. Each agent worked through mentions in batches, armed with the parent post or comment, full mention text, Google search tools, and information about the matched restaurant. Each mention was scored according to confidence, and low confidence mentions were rechecked by a more powerful model (Fable).

Inconsistent matches were found on ~3% of these 10k matches. Most of these inconsistencies:

• Matched to an address that was a different business (the restaurant had moved)
• Had ambiguous names like "Rudy's", "Joe's", etc. that our previous audit attempt had failed to identify for whatever reason

The emergence of new errors showed that there were still flaws in my original audit, including:

• Google matching for the restaurants looked primarily at the business name, failing to filter out non-restaurant businesses.
• Closures, renames, and reopenings were not properly accounted for.
• Ambiguity should have required further verification. Restaurant mentions with ambiguous names should have used full quote context by default and relied on an adversarial model checking the work of the first model.
• I should have done more thorough review during the restaurant linking step, to ensure correct matches. My LLM review during that step only looked at a portion of the mentions for each restaurant, when I should have used it to review all of them.

This second audit process was expensive in terms of time and API calls, but I think it was very necessary to make my labels as accurate as possible. And it showed the need for improving my original post-extraction audit when I do future runs.