Optimization results

With the gold set and testing infrastructure from the previous section in place, I retested the three steps of my LLM extraction pipeline with various optimizations, and measured the cost and accuracy at each step. In each table below, the highlighted rowfinal marks the option I ended up using in the final pipeline. To see the finalized savings, visualized, skip to the conclusion section.

How to read these tables · P (precision): of what the model flagged, how much was correct · R (recall): of what it should have found, how much it found · Exact / ±0.5 / MAE: rating agreement with the human label — exact match, within half a point, and average error · Canon: share of extracted dishes whose normalized name matches the human label · Cost: relative to the most expensive option (= 1.00) · Bold: best value in its column

One of the most obvious places to start is the model choice. I decided to just use OpenAI's models, because I didn't want the complexity of dealing with multiple providers.

The models I used from OpenAI were Luna, 5.4 Mini, and 5.4 Nano. These models all accept an optional "reasoning" parameter, but to keep things simple, I only provided this as "low" for Nano, and also tested Nano without reasoning. In terms of pricing, all of these models were in the same ballpark: between 10 cents - 37 cents per 1 million input tokens. Very fortunately for me, I started conducting these tests right when OpenAI cut their rates for Luna by 80%, which put the model at the same price as GPT 5.4 Nano. Being a smarter model at the same price as Nano made it the clear winner for most

Due to the very competitive pricing, as well as Luna's ability to correctly reason through multi-step instructions, especially on long comment chains, I picked Luna for all 3 steps. This move alone dropped by spending on tokens to 2.5 times cheaper than what it was when I was using GPT 5.4 mini. It could have been even cheaper, but I used the default reasoning level for Luna, which has output tokens associated with it that raise the cost (60 cents per million tokens)

I noticed that Nano repeatedly struggled with long comment chains, having a significantly lower recall rate than gpt 5.4 mini or 5.6 Luna.

Model	Reasoning	Cost	Chains		Posts
Model	Reasoning	Cost	P	R	P	R
gpt-5.4-mini	—	1.00	89.9	88.1	95.4	97.3
gpt-5.6-lunafinal	default	0.40	90.8	90.0	99.6	83.7
gpt-5.4-nano	—	0.26	90.8	78.5	98.5	94.2
gpt-5.4-nano	low	0.29	89.7	75.7	98.5	87.7

Takeaway: gpt-5.6-luna matches gpt-5.4-mini's accuracy at 40% of the cost. Nano is cheaper still, but gives up ~10 points of chain recall — long comment chains are where the cheap models crack.

Mean of 3 sync replicates against the frozen gold set (188 chains, 146 posts, 724 gold mentions). Costs come from actual token usage, including reasoning tokens, at standard API rates.

Conclusion

To measure the real end-to-end impact of all of this, I re-ran the full pipeline on the same 2-week archive. I did this once with the pre-optimization configuration, and once with the optimized one. The pre-optimization run uses gpt-5.4-mini with no input packing, no reasoning, and no minimum-mention floor.

Before and after

Step	Requests	Input tokens	Total tokens	Cost
1 — post mentions	453453	170,477170,930	179,960189,976	$0.09$0.03
1c — chain mentions	5,4672,666	3,263,8651,840,825	3,467,2572,280,454	$1.68$0.45
2 — post ratings	225169	245,707189,794	261,077214,277	$0.13$0.03
2c — comment ratings	4,3151,401	4,801,9541,748,461	5,132,6922,200,986	$2.54$0.45
3 — post dishes	9081	115,748104,402	137,943147,288	$0.09$0.04
3c — comment dishes	3,0331,877	3,493,8082,223,022	3,699,2582,545,398	$1.77$0.42
TOTAL	13,5836,647	12,091,5596,277,434	12,878,1877,578,379	$6.30$1.41

Conclusion: the optimized run cost ≈ $1.41, down from ≈ $6.30 for the pre-optimization run, or about 4.5x cheaper. Part of that is per-token pricing (gpt-5.4-mini charges $0.375 per 1M input tokens and $2.25 per 1M output, versus $0.10 / $0.60 for luna), and the rest is a smaller workload. The mention floor and input packing cut requests roughly in half and input tokens by 40%.

What each optimization saved

The below chart visualizes how much in API costs each optimization saved.

Model choicegpt-5.4-mini → gpt-5.6-luna−$4.06 · ≈83% of savings

3.75× cheaper per token on both input and output — worth more than the other two optimizations combined, even though luna spends extra output tokens on reasoning.