The Hard Parts of Building an AI Product

A convincing answer is only the demo

I thought putting useful AI inside Leversy would mainly be a model problem. Choose a capable model, give it enough product knowledge, and build a good interface around the answer. That produced a convincing demo quickly. It did not produce something I would trust with a creator’s work.

A real product has to answer harder questions. Which context is relevant? Can the user verify a recommendation? When is the AI suggesting something, and when is it changing their work? What happens when it is wrong, slow, or too expensive for the value it creates?

The model still matters. But most of my work moved to the decisions around it.

Context is a product decision

A blank chat makes the user repeat everything. A product already knows what they are working on, so the obvious answer is to give the model as much of that information as possible. I learned that more context is not automatically better context.

Extra information can distract the model, carry old assumptions forward, slow the answer, and increase its cost. Too little context produces generic advice. Too much can make a specific question harder to answer.

So I treat context as part of the interface. The user should understand what the AI is considering and be able to change that frame. The product should know enough to be useful without pretending that everything it knows is relevant.

Evidence matters more than confidence

Language models are very good at sounding certain. That is useful when the answer is right and dangerous when the user cannot tell which part came from evidence and which part came from interpretation.

Leversy helps creators make decisions about their work. I do not want the AI to win an argument through confident writing. I want the creator to see why a recommendation was made, inspect the material behind it, and disagree when their judgment is better.

Sources do not eliminate mistakes. They make the answer inspectable. That changed how I design AI features: the answer is not the end of the interface. The path back to the evidence is part of the answer.

Actions need a higher bar

An AI answer can be ignored. An AI action can overwrite, publish, spend, or create work the user never wanted. The risk changes as soon as the model can do more than talk.

I learned not to treat a vague request as broad permission. The product needs to make the proposed action clear, keep consequential decisions under the user’s control, and make the result easy to understand or recover from.

The quality of an agent is not the number of actions it can perform. It is how clearly the product defines the boundary between helping and taking over.

Credits make the economics visible

Most software products can make a button feel free at the point of use. AI makes that harder. Two requests that look similar in the interface can require very different amounts of work, and the most expensive answer is not always the most useful one.

I introduced credits in Leversy because both the user and the business need an understandable budget. Credits create one product unit for paid AI work without forcing the user to think about tokens, model prices, or infrastructure providers.

But credits only work when the value is clear. If someone cannot connect the credits they spent to time saved or a better decision, the system feels like an arbitrary tax. Pricing AI forced me to judge features by their outcome, not by how impressive the model call looked.

Failure is part of the experience

An AI request can fail because information is missing, a provider is unavailable, the model misunderstood the task, or the answer is technically valid but not useful. Only some of those failures look like errors in the code.

That makes failure a design problem. The user needs to know whether to wait, retry, change the request, inspect the evidence, or make the decision themselves. A polished loading state cannot repair an answer that solved the wrong problem.

I no longer define reliability as never failing. For an AI product, reliability means making uncertainty visible and giving the user a useful way forward.

Evaluation has no finish line

Traditional tests can tell me whether a button still works. They cannot tell me whether creative advice is relevant, whether the evidence supports it, or whether the answer was worth its cost. There may be several good answers and many convincing bad ones.

The evaluation work is ongoing. I judge AI behavior against the product promise: use the right context, stay grounded, follow the user’s intent, and create enough value to justify the time and credits. Model changes and new user behavior can move that standard, so one good result proves very little.

I started by asking what the model could do. Now I ask what the user can verify, what remains under their control, and whether the outcome is worth its cost. Those questions are less impressive in a demo. They are the work of turning AI into a product.