Understanding AI Infrastructure Part 4: The Cost of Correctness (Tech Strategy)

In Part 3, I started to look at the costs of building and operating GenAI-based products and services. And these costs keep changing. But we are starting to get a better feel for ongoing unit economics.

And this is what determines profitability. And competitive dynamics.

My approach to assessing GenAI operating costs is the below checklist.

What is the cost of compute, including energy and cooling? Initial cost vs. ongoing?
1. The core compute is going to drive a lot of the costs. Especially if you are using an AI cloud service provider. If you have a downloaded open-source model, then it’s mostly the initial cost.
2. These compute costs dependent on the requirements of the AI workloads:
  1. The compute requirements
  2. The timing requirements
  3. The memory requirements
3. This is most of what I focused on in Part 3.
What is the cost of creating and maintaining the desired correctness over time?
1. How accurate and correct you need the foundation model to be is a big deal. And this factor determines how much of the operations are done by software versus humans. It can make the costs of GenAI products look a lot more like services than traditional software.
2. My main questions are:
  1. How much does correctness matter? What level of correctness does the product need to compete? What is the cost of inaccuracy?
  2. How does correctness change over time? Is it stable and flat lining, advancing, naturally changing or deteriorating?
  3. How much of a long tail is there in the domain? Is the long tail a liability or a benefit?
  4. How much of the process is iterative? How much are humans in the loop?
  5. What is the initial cost of training and getting to the desired level of correctness?
  6. What is the ongoing cost of inference at the desired level of correctness? How much are humans in the loop?
  7. What is the cost and frequency of fine tuning and/or retraining?
What are the cost implications of increasing scale?
1. What are the scale advantages vs. disadvantages?
2. Does the Jevon paradox apply?

Ok. Let’s jump into #2 on the checklist.

#2 What is the Cost of Creating and Maintaining the Desired Correctness over Time?

Everyone talks about intelligence and smarts. A smart car. A smart refrigerator. A smart chatbot.
Fine. But that’s not the key question in practice.

The is key question is “how correct does it have to be?”

How accurately does an AI transcriber need to be to summarize the conversation on a Zoom call?
How well does an autonomous vehicle have to be to drive your car? 99%? Just better than a human
How accurate does an AI research tool have to be in math? Can it be less than 100%?
How accurate does an AI doctor have to be in making a diagnosis? 95%? Better than a US doctor? Better than a Rwandan doctor?

Correctness is the metric for the performance of a lot of GenAI apps. It’s a key decision.

Because it turns out this can change operating costs dramatically.

It may cost a certain amount to build and operate a chatbot that is 80% correct in its replies about a medical diagnosis. But getting to 90% may cost dramatically more. And 95% may be way, way more expensive. There can often be an escalating cost of progress.

So, my first question on the list is:

2.1: How much does correctness matter? What level of correctness does the product need to compete? What is the cost of inaccuracy?

And in creative fields (like making cartoons or graphics) it turns out correctness is not that important. The AI tool just needs to create lots of variations. But there isn’t necessarily a right or wrong answer. A generated image of a human with 7 fingers would be inaccurate. But overall, creative fields are much easier.

But correctness in AV is very important. And the cost of inaccuracy can be very high.

The first step is to determine the desired level of correctness.

My second question is:

2.2: How does correctness change over time? Is it flatlining, advancing, changing or deteriorating?

Certain types of knowledge plateaus. It flatlines. Once your app is good at checking spelling and grammar, it is pretty much done. The level of correctness in this particular knowledge is fixed.

But science, medicine and other types of AI knowledge are advancing. To be correct, the model has to keep advancing as well. It doesn’t really plateau. Autonomous driving is like this today. Being accurate in driving on highways is not the same as being accurate in driving in downtown Dhaka.

And other types of knowledge are always changing. How do you have correctness and accuracy in making money in the stock market? Or in getting good returns on digital marketing activities? The “correct” answers in these activities can be different year after year.

Finally, there is also the question of deterioration. A high performing model can start to have lower accuracy over time. Often for unclear reasons. This is sometimes called “data drift”.

The effort required to maintain correctness over time can dramatically change the cost structure.

My third question is:

2.3: How much of a long tail is there in the domain? Is the long tail a liability or a benefit?

Think about a chatbot on an ecommerce site that is good at answer common questions like “where is my package?” “How do I get a refund”.

Having high accuracy answers to such questions is definitely doable.

What happens when you expand the conversation to “what type of apparel do you recommend for me?”. “What style and size of jacket should I wear on my trip to France in December?”.

As you increase the scope of potential interactions, you move into the long tail of topics and use cases. And the long tail can be a major problem in terms of achieving and maintaining “correctness”.

As mentioned, as you go from 90 to 95% correctness in a specific domain, you can explode the difficulty and costs of being correct. Well, when you go from common topics to the long tail, you can also explode the difficulty and cost.

That is a very big difference from traditional software.

Keep in mind, Google Search is a machine learning product that about the long tail. Lots of sites can answer basic common questions. But only google search can match unique strange questions with the right niche websites. It’s this matching in the long tail that gives search its network effect. This is also the same power in TikTok. Its network effect is from matching niche tastes with niche content. The long tail also helps in marketplace platforms.

But in GenAI, the long tail can be a huge problem. It can create big difficulties in performance. And there can be escalating costs. Limiting the domain of GenAI products is a critical decision.

My fourth question is:

2.4: How much of the process is iterative? How much are humans in the loop?

To create art, you need to do lots of iterations. The AI creates some images. Then a human assesses them and provides feedback and other direction. The AI generates a second set. A human decides on more additions or refinements. And so on.

The creative process is iterative, with lots of back-and-forth between humans and AI. And the process is similar for writing code and other complicated endeavors.

So, if we’re thinking about the costs of production, we need to think about how iterative it is. And about the costs of humans in the loop. In GenAI workloads, we might need humans for:

Moderation
Feedback and direction
Final judgement

All of this impacts the costs of production.

We can also think about iterations with customers. An AI companion has lots of back and forth to get the right answer for a customer. To find the right item to purchase.

Getting to the desired level of “correctness” can be a multi-stage process. And those steps may or may not include humans.

***

Ok, that brings us to my final 3 questions, which are all about the costs.

What is the initial cost of training and getting to the desired level of correctness?
What is the ongoing cost of inference at the desired level of correctness? How much are humans in the loop?
What is the cost and frequency of fine tuning and/or retraining?

The initial training cost is pretty hard to predict right now. What is possible keeps changing. And what it costs to achieve it keeps dropping (mostly).

But the ongoing inference cost is where things are looking problematic.

It appears most GenAI apps need humans in the loop to maintain correctness. They need to be providing feedback. They need to correct the responses.

Martin Casado at a16z has written great stuff about how the economics of most GenAI look like software plus services. Not traditional software like SaaS. Instead of gross margins of 70%, he says the gross margins are closer to those of service companies (i.e., 20%).

And all this follows from how much of a correctness problem you have. That’s why this is my #2 question.

And that is just for ongoing inference operations.

It gets even worse when you frequently have to do retraining. As mentioned, training can be 10x more expensive than inference. If you are frequently having to retrain your model to keep the correctness at the required level, that can be really expensive.

***

Ok. Those are most of my questions for #2.

In the last part, I’ll talk about scale. And whether this is a strength or a problem for these models and their costs.

Cheers, Jeff

—-

From the Concept Library, concepts for this article are:

Generative AI and Agents
GenAI Costs: Cost of Correctness

From the Company Library, companies for this article are:

——-

I am a founder, consultant and keynote speaker specialized in digital and AI agentic strategy.

I am the founder of Agentic Org Now, a firm that helps leadership teams launch AI and agentic initiatives fast.

I write (a lot) about digital and digital AI strategy (3 best selling books, +2.9M followers on LinkedIn). You can get a free book at the email sign-up below.

My Moats and Marathons book series is a framework for building and measuring competitive advantages in digital and AI businesses.

This content (articles, podcasts, website info) is not investment, legal or tax advice. The information and opinions from me and any guests may be incorrect. The numbers and information may be wrong. The views expressed may no longer be relevant or accurate. This is not investment advice. Investing is risky. Do your own research.

Jeffrey Towson 陶迅

Consultant and speaker on digital and AI agentic strategy