Optimizations

I took the pipeline and data labelling flow I outlined in the previous step, and ran it on a subset of posts and comments from arctic shift (over a 2 week interval). After iteratively refining prompts to handle edge cases, I eventually got to the point where I was happy with the quality.

But something didn't sit well with me. The output passed a vibe check... but with how much certainty could I actually say it was correct, considering I didn't have any tests?

In addition, I had to consider another metric I didn't even discuss yet - cost. Running my current pipeline on just 2 weeks of data cost around 5 dollars. Running it on 2 years would cost 50x this, or 250 dollars. That's quite a lot of money for a random side project, which might not even end up being that useful. I wanted to cut down on overall API usage, which would save me a bit of money and make the project more educational for me.

Below are a series of optimizations I made to my pipeline, model choice, and prompt structure. I estimate it saved me roughly $xyz (whatever percent), and improved precision by abc%.

There were 2 classes of optimizations I made. Pipeline optimizations, where I would selectively ignore data or shuffle around parts of my pipeline to reduce inputs and save money, and Prompt optimizations, where I would rely on intelligent prompting strategies, to cut down on the total number of tokens per prompt.

Pipeline flow optimizations

One cost saving measure came from reexamining the project requirements. In short - do I NEED to analyze every comment or post, and do I NEED to include every restaurant? After thinking about it for a while, I realized that my current flow might actually be detrimental to the end user experience. This is because I end up having every restaurant that is mentioned anywhere on the subreddit represented in my final app.

Looking at my sample 2 week reddit archive, I noticed how there was a long tail. There were a good portion of restaurants mentioned many times, but the vast majority only had a few mentions. I suspected a similar distribution would also exist in a multi year dataset.

Most users won't find all of these restaurants relevant. If we care about the best restaurants or those with the best version of a certain food, who wants to read about an obscure restaurant with only a few comments?

With that in mind, I had a crucial insight. Currently in my pipeline, I was linking the restaurants (the step where I match all of the restaurant mentions to a singular database entity) AFTER the LLM extraction step. Instead, I could do this step immediately after step 1 of the LLM pipeline, figure out which restaurants only had a few mentions, and then exclude them going forward. This way, rarely-mentioned restaurants would NOT go through the restaurant rating or cuisine steps of the LLM pipeline, reducing the token count and API cost.

Not only was this a great cost saving measure, it would lead to less data overall in my end application, meaning less room for bugs and bad comments. And I could spend more time reviewing the smaller dataset.

In addition, I employed certain strategies to ignore low quality comments.

• Comments with "deleted" or "removed" as the sole text were excluded from the input
• Comments below a depth of 3 were excluded from the input. Models (especially smaller ones) degrade greatly with a higher input amount, and I wanted to keep my output as accurate as possible. It would also save money on token costs, and most importantly, I think it would improve user experience. It can be disorienting navigating to a restaurant review that is multiple comments deep on reddit, and parse what the context of the comment is supposed to be.

Prompt Optimizations

The pipeline flow optimizations were great in increasing the quality of the data, but a crucial question remained: How do I optimize each individual step of my pipeline, and make the corresponding tradeoffs between accuracy and cost? I would have to make sure any changes didn't drastically impact the accuracy of the LLM labelling.

I needed a more formal, rigorous way of testing my prompts. After some research, I came up with the idea of having a gold set. Basically, I would create a relatively small set of human reviewed data (200-300 posts and comments), which I would use to measure the accuracy of the LLM labelling.

I wanted to make sure this gold set had a diverse set of input data, specifically geared towards input that the LLM struggled with. I backtracked through my prompts and noted edge cases as well as common "archetypes" of comments and posts in the training data. I tried to get about a dozen or more examples from each one of these archetypes, to ensure that we could see if any particular case caused failures from the LLM. Click on some of the archetypes to get an idea of what they are testing.

With the archetypes in place, it was a straightforward (but annoying) process of labelling the gold data. I undertook this progressively: For each LLM pipeline step I would first label the gold set for that step manually, and then come up with a series of variations to increase accuracy and reduce cost. For each variant, I would then run it 3 times with Open AI's API, and then take take the average score when compared to my gold set.

By "variant" I mean different strategy of prompt optimization. These normally had to do with prompt or reasoning optimization, packing multiple inputs into 1 request, or prompt rephrasing. I explain this more in the next section on the results.

A key piece of insight, was just how big the system prompts were. These were the precise instructions I fed to the LLM to get it to correctly classify our posts and comments. The percentage of tokens dedicated to system prompts was over 70%! If I was using a normal, sequential set of APIs, I could probably use prompt caching to dramatically reduce my bill. Unfortunately, batch processing does not allow that. Nonetheless, If i could find some way to decrease the average system prompt length, OR combine multiple prompts into only 1, I would save money.

I also realized how little was actually spent on posts as opposed to comments. For each step, the amount of tokens spent on posts was <10% the tokens spent on comment chains. This makes sense, since 1 post can have dozens of comments.

The results of my tests are explored in the next section: Optimization results.