What I learned

This was a lot of info, so here's a bird's eye view of how this extraction went and my plans for the future.

Overall, I was impressed by the accuracy of these smaller LLMs. This was not the first time I tried one of these Reddit analyses, but I was discouraged in earlier attempts because the cost effective models were not great at following multi-step extraction prompts. GPT 5.6 Luna was the first model I tried that managed to strike the perfect balance between cost and accuracy.

However, there were many points in the process that were inefficient and made the project challenging.

Firstly, I regret using the batch API for my initial extraction. This may have technically saved money compared to the normal API, but I think using the API was altogether the wrong move. Instead, I should have done what I did for the audit round, which was use my Claude Code Max subscription. Claude Code subscriptions are highly subsidized: my audit passes burned through roughly 45 million tokens of Opus usage, which would have been a few hundred dollars at API list prices ($5 per million input tokens, $25 per million output), all covered by the $100 a month Max plan. Tools like dynamic workflows make it easy to set up multiple worker agents running at once via headless Claude Code. Not only is this cheaper, but these agents have access to custom APIs and tools I provide for them, including Google search, which was absolutely vital in verifying the accuracy of the restaurant matches.

The main issue with my extraction, however, was how scattered it was. Each step of the pipeline (and the many experiments and audit rounds I ran afterwards) could take up to a few hours to complete, and required a substantial amount of double checking and handling of edge cases. This made the project painful to work on at some points, such as when I was staring down the barrel of reviewing thousands of matched restaurants and making sure there weren't any incorrectly matched. In addition, I was jumping between working on the web app implementation and running the LLM audits. This was a confusing process that will be hard to reproduce in future runs.

Instead, I should have spent more time planning my pipeline, and put more upfront time into handling edge cases and verifying output data. That way, I could have run the entire pipeline once and have been confident in the output of my results, instead of having to do so much on the fly. But more than this, I should have figured out how to lean on AI to handle more of those nasty edge cases and additional QA steps. It would have been easier and more pleasant to use automated reviews of the output, using general edge cases that I had picked out from sample runs. It is simply not feasible to review thousands of restaurant matches, or hundreds of thousands of mentions matched to restaurants.

I plan to rerun this pipeline a year from my previous extraction (which was around August 3rd), using a more end-to-end automated approach to analyze the data, with as little human involvement as possible.