Choosing an artificial intelligence model often begins with a simple comparison of prices. Input token cost, output token cost, and deployment availability are easy to place into a spreadsheet, which can make the least expensive model appear to be the safest and most practical choice. During this proof of concept, I discovered that this assumption did not reflect the complete cost or the actual experience of using the application.
The project involved building an intelligent reporting tool for Pastoral Reach. Users can ask business questions in natural language and receive structured reports based on the information available to their organization. Behind that apparently simple interaction, the application must understand the request, generate a valid SQL query, enforce strict security rules, execute the query safely, and interpret the results in a useful format.
Model selection had a direct effect on every part of that workflow. A model could produce a technically correct query and still be unsuitable for an interactive reporting experience if it required too much time to respond. Correctness was essential, but it was only one part of the outcome I needed to evaluate.
I initially selected a smaller and less expensive model. The decision seemed reasonable because the model offered a significantly lower published cost per token and was capable of following the reporting instructions. During early testing, it generated valid results, understood the reporting schema, and completed the expected workflow.
The main problem was performance. Report generation took much longer than expected, especially when the model had to process a detailed reporting schema, security requirements, organization specific catalogues, and structured output rules. Even when the final result was correct, the waiting time made the application feel less responsive than an interactive reporting tool should.
This revealed an important distinction between token price and effective cost. A model with a lower price per token can still be inefficient if it consumes more tokens, requires more reasoning time, or increases the likelihood that another model call will be needed to review or correct its output. The cost of the workflow includes latency, token consumption, reliability, and the number of steps required to reach a trustworthy result.
Instead of continuing to optimize around the original model, I decided to evaluate several alternatives under the same conditions. I added a model selector to the application so that each reporting scenario could be tested against different Azure OpenAI deployments. I also expanded the diagnostic information to capture duration, input tokens, output tokens, estimated cost, and the role performed by every model call.
This instrumentation changed the conversation from assumption to evidence. I could run the same type of report using different models and compare how they behaved inside the actual application. The comparison was no longer limited to published prices or general statements about model capability.
A newer model with greater capacity produced the same required results in considerably less time. It followed the reporting rules consistently, used the supplied context more effectively, and reduced the delay experienced by the user. Although its published price per token was higher, the overall workflow was faster and better aligned with the purpose of the product.
The result did not prove that the more expensive model will always be the correct choice. It demonstrated that the right model depends on the work it must perform and the experience the application needs to deliver. A simple classification task, a complex reporting workflow, and a long analytical request may each benefit from a different model.
For this proof of concept, the model needed to do more than generate text. It had to understand a controlled data schema, distinguish between related business concepts, select trusted identifiers, create secure SQL, respect organization boundaries, and return structured output that the application could validate. The newer model completed this combination of tasks more efficiently.
The experience also reinforced the role of human judgment in development supported by artificial intelligence. Artificial intelligence can assist with implementation, analysis, testing, and experimentation, but it does not define the business objective or decide which tradeoffs are acceptable. Those decisions still require observation, experience, and an understanding of how the complete system behaves.
I needed to recognize that a correct result was not sufficient if the user had to wait too long for it. I also needed to understand that the least expensive token did not necessarily produce the least expensive workflow. Creating the right measurements made it possible to challenge the original assumption and choose a model based on actual evidence.
The model selector and diagnostic metrics also made the proof of concept more adaptable. As new models become available, I can evaluate them without restructuring the application or relying on isolated benchmarks. Each model can be compared using the same reporting scenarios, security requirements, token measurements, and user experience expectations.
This approach is particularly valuable because model performance can vary depending on the complexity of the prompt and the structure of the workflow. A model that performs well in a general benchmark may behave differently when it receives a large schema, strict SQL rules, dynamic organization catalogues, and a structured response contract. Testing inside the real system provides information that a pricing table or benchmark cannot provide on its own.
The final decision was therefore based on more than speed. The selected model provided a stronger combination of reliability, token efficiency, response time, and output quality. It delivered the same functional outcome as the less expensive model while producing a much better experience for the user.
The most important lesson from this proof of concept is that model selection should be treated as an engineering decision. Published token prices are useful, but they should be considered alongside latency, consistency, token usage, workflow complexity, and the cost of additional validation calls. The best model is the one that supports the complete objective of the application.
Artificial intelligence accelerated several parts of the development process, but experience guided the decisions that shaped the final result. By measuring real behavior and remaining willing to reconsider the original approach, I was able to select a model that better served the needs of the application. That combination of intelligent tooling and practical judgment produced a result that was faster, more reliable, and more valuable.
