LLM consulting: model choice, evaluation, and the cost of both.
Choosing a model is a five-minute decision that sets your cost base, your latency floor, and your switching cost for the next two years. Most teams make it once, informally, and live with it.
- A model chosen because it was the one everybody had heard of.
- No way to compare two models on your own task rather than a public leaderboard.
- Spend rising with usage and no clear line to the value returned.
- A prompt nobody will touch because the effect of changing it is unknown.
What’s included
Model selection
Compared on your task and your data, against quality, latency, cost, and where data may travel.
Buy, build, or fine-tune
The decision made explicitly, with the reasoning written down and revisitable.
Evaluation harness
A test set from your real traffic and a scoring method that makes changes comparable.
Prompt and context engineering
Treated as versioned artefacts under regression tests, not strings edited in production.
Cost and latency modelling
What the system costs per unit of work today, and what it costs at ten times the volume.
Team enablement
Your engineers left able to run the harness, read the results, and make the next call themselves.
How we work
- Map~1 week · fixed fee
AI Opportunity Sprint
Find where AI creates real value — and what it’s worth.
- Workshop with your team
- Prioritized use-case backlog
- ROI model for the top opportunities
- Prove3–4 weeks
Production PoC
Build one use case for real — and measure it.
- One use case built production-grade
- Evaluated against a real KPI
- Evidence-based go / no-go
- ScaleOngoing
Scale & Enable
Expand across workflows — and level up your team.
- Rollout across teams and workflows
- Evals & monitoring in place
- Team enablement + fractional AI leadership
Buy, build, or fine-tune
Buy — use a foundation model as it comes — when the task is general and your advantage is speed rather than differentiation. Summarizing, drafting, classifying into common categories. Lowest cost to start, fastest to ship, and the right default far more often than teams assume.
Build with retrieval when the value is in knowledge you own: policies, documents, product data, ticket history. Most enterprise use cases live here. You keep data current without retraining and answers stay grounded in sources you control.
Fine-tune when a specific gap survives the first two — an output format prompting cannot hold reliably, a domain vocabulary the base model mangles, or a latency budget only a smaller tuned model can meet. It is a real tool and a poor starting point, because it freezes knowledge at training time and adds a retraining cycle to every change.
Building an evaluation harness
Public benchmarks tell you how a model performs on someone else’s task. They cannot tell you whether it handles your customers’ phrasing, your document formats, or the failure your compliance team actually worries about.
A usable harness is smaller than people expect. A few dozen real inputs with agreed-good outputs, a scoring method matched to the task — exact match where there is a right answer, a rubric with a model as judge where there is not, human review on the sample that decides anything important — and a run that executes on every change.
The payoff is that model and prompt decisions stop being arguments about taste. You change something, you run the harness, and you can say whether it improved.
Cost and latency are design constraints, not invoices
Token cost and response time are set by architecture long before they show up on a bill. How much context you retrieve, whether you cache, how many calls a request makes, which model handles which step — those choices decide the economics.
Modelling them early changes the design. A system that routes simple requests to a small model and escalates only the hard ones can cost a fraction of one that sends everything to the largest available model, and users notice the latency difference more than they notice the quality one.
- Which LLM should we use?
- It depends on the task, the latency you need, what you can spend, and where your data is permitted to go. We compare candidates on your workload rather than recommending a default, and build so the model can be swapped later.
- Is fine-tuning worth it?
- Sometimes, and later than most teams try it. Fine-tuning teaches style, format, and narrow tasks well, and facts poorly. If the goal is for the model to know your content, retrieval is usually the better instrument.
- How do we stop hallucinations?
- You reduce them by grounding answers in retrieved sources, constraining the task, and adding refusal behaviour for questions outside scope. You detect the remainder with evaluation. Anyone promising elimination is overselling.
- Can this run in our own environment?
- Yes. Open-weight models can be deployed inside your cloud or on-premise where data residency requires it. That constraint changes the model shortlist, so it belongs in the conversation early.
- Do you hand over the evaluation setup?
- Yes. The harness and the reasoning behind the model choice are yours, and enabling your team to run them is an explicit deliverable rather than a bonus.