Before You Train Your Own AI Model, Try These Five Things
Most AI features don't need a custom model. A practical order of decisions, from APIs to distillation.
Sooner or later, every team building an AI feature hears the question: “Shouldn’t we just train our own model from scratch?”
It sounds ambitious. It can also burn months before anyone knows whether the feature works.
Suppose you’re building a support assistant for an online store. Customers write in about refunds, deliveries, and payment disputes, and the assistant needs to sort each request and answer policy questions. Here are the options in the order I’d try them, and what tends to go wrong at each step.
1. Start with an API, but build the test set first
The fastest working version is a single API call: send the customer’s message to a hosted model, ask for the issue category as structured JSON, and pass it to the backend. It can work on day one.
The common mistake is judging it by eye. A model that looks great on ten hand-picked messages can fail on the eleventh. Before changing anything, collect a few hundred real, anonymized messages and label them. That set becomes the scoreboard for every decision that follows.
If there’s one habit worth building: the evaluation set outlives every model you try. Models get swapped every few months. A good test set lasts.
Under the hood
Keep separate data for development and final evaluation. Use a validation set to compare models, tune thresholds, and diagnose failures. Reserve a held-out test set, with carefully checked labels, for the final comparison only. If every decision uses the same test set, you end up tuning to that set rather than to real traffic.
Score per category, not just overall. A model with 92% accuracy can still get refund disputes wrong half the time if they’re only 5% of traffic. A confusion matrix, which shows which categories get mistaken for which, makes this visible immediately.
2. Most requests don’t need your best model
Look closely at your validation results and a pattern usually appears. Say 60% of messages are like “Where is my order?” A small, cheap model may classify those almost as well as a large one. The rest look like “I was refunded once but charged twice, and my replacement got cancelled.” Those genuinely need a stronger model.
That’s where routing comes in: a lightweight step decides which model handles each request. Managed versions exist too, such as Amazon Bedrock’s Intelligent Prompt Routing.
A common misconception: routing doesn’t mean running only the “refund part” of a big model. You can’t select layers or neurons through an API. You’re choosing which whole model gets the request.
The trade-off people underestimate: a misrouted hard request costs more than an overpaid easy one. An angry customer is more expensive than a few cents of compute, so when the router is unsure, it should escalate.
Under the hood
There are two ways to route. A lightweight classifier can look at the request before any model runs, which is how Bedrock’s router works. Or you can try the small model first and escalate when its answer looks unreliable. That second approach is a fallback cascade, and escalated requests pay for both calls.
Either way, confidence scores need care. Calibrate a classifier’s scores against validation data, and don’t treat a chat model’s self-reported confidence as a real probability. Tune the threshold on the validation set. A higher threshold sends more traffic to the large model, which only helps if the large model actually does better on those requests.
The cost difference is easy to estimate. One million requests at about 500 tokens each is 500 million tokens. At illustrative prices of $3 per million tokens for a large model and $0.25 for a small one, that’s $1,500 versus $125. In a cascade where 60% of traffic stays on the small model, the bill drops to about $725, since escalated requests pay twice. This assumes every request uses the same number of tokens at a blended price, and output tokens usually cost more than input, so treat it as a rough estimate. The ratio is what matters. Cascades can reduce the total bill, but escalated requests also wait for two model calls, so measure latency alongside cost.
3. When the model is “wrong,” check what it was given
Suppose users report that the assistant gives outdated refund answers. The instinct is to blame the model. Often the real problem is that it never saw the current refund policy.
Retrieval-augmented generation (RAG) addresses this. Fetch the relevant policy section at answer time and include it in the prompt. When the policy changes, you update a document, not a model.
A useful debugging rule: many “the model is dumb” complaints turn out to be retrieval problems. The wrong document was fetched, or none was. Check the context before touching the model.
Under the hood
Measure retrieval separately from the answer. A standard metric is Recall@k: the share of relevant documents that appear in the top k results. When each query has exactly one right answer, that simplifies to how often the right document shows up in the top k. If the correct policy isn’t among the retrieved documents, the model can’t reliably ground its answer in it, however capable the model is.
In a known-duplicate evaluation in my open-source project DevSignal, keyword search (BM25) found the known duplicate issue in the top 5 about 48% of the time. Hybrid retrieval, which combines keyword and semantic search, raised that to about 61%. That gain came from retrieval alone, before any model saw the context.
4. Fine-tune for behavior, not for facts
Retrieval gives the model information it wasn’t trained on, like your current policies. But even with the right context, a model can struggle with dozens of company-specific issue categories or a strict output format, drifting on edge cases despite well-written prompts.
Fine-tuning an existing model on labeled examples can make it consistent. LoRA keeps this affordable: it trains small add-on weight matrices while the original model stays frozen, so iterating is cheap.
A rule of thumb: fine-tune for format, tone, and categories. Don’t fine-tune in facts that change, like prices or policies. Those belong in retrieval.
Under the hood
Before fine-tuning just to fix formatting, try structured outputs that constrain responses to a schema, plus validation. Many malformed-JSON problems disappear there.
When fine-tuning is warranted, a few hundred to a few thousand high-quality examples is a common starting range. A smaller, cleaner dataset can outperform a larger noisy one, depending on how complex the task is and how well the data covers it. In the original LoRA paper, training only the small low-rank matrices cut the number of trainable parameters for GPT-3 by about 10,000 times. That reduces training cost, not model size or inference cost.
Where you train matters too. OpenAI is winding down its self-serve fine-tuning platform, with new training jobs ending for remaining eligible customers on January 6, 2027. For teams that want control over training, LoRA on an open-weight model is one practical alternative.
5. Distillation: when the big model works and the bill doesn’t
Now suppose the large model handles the hard cases well, but at tens of thousands of requests a day the cost adds up. That’s exactly the situation distillation is built for.
A typical workflow:
- Use the strong “teacher” model to label and generate examples for the specific task.
- Filter hard. Review a sample by hand and drop anything the teacher got wrong.
- Fine-tune a smaller “student” model on what remains.
- Test the student on an untouched test set with independently checked labels, never on teacher-generated data.
If the student gets close enough to the teacher on that test set, it can take over most of the traffic at a fraction of the cost. The savings only make sense if they outweigh training, hosting, and maintenance costs over the volume you expect to serve.
The idea is well established. DistilBERT (2019) was 40% smaller and 60% faster than BERT while retaining about 97% of its reported language-understanding performance. Distillation can also be done by generating training examples with a stronger model and fine-tuning a smaller one, where training access and provider terms permit.
What’s easy to miss:
- The student inherits the teacher’s mistakes. Skip the filtering and you train in the errors.
- It struggles with situations missing from its training data, so keep the router as a fallback.
- Check the provider’s terms before training on its outputs. Some restrict it.
- A distilled model is now yours to monitor, retrain, and maintain.
Under the hood
There are two main ways to distill.
Training on teacher outputs (synthetic labels). The student learns from the teacher’s final answers. This can work with a text-output API, because it needs no access to the teacher’s weights or probabilities. It’s the common approach in practice, subject to the provider’s terms.
Matching the teacher’s probabilities (soft labels). The student learns from the teacher’s full probability distribution over possible answers, not just its top choice. That carries more information, such as “this message is probably a refund, but could be a payment dispute.” It needs the teacher’s raw probabilities, which usually means an open-weight model, though some APIs expose the top few token probabilities. DistilBERT used soft-target distillation as one part of its training, alongside language-modeling and cosine-distance objectives.
Define “close enough” before training. For example: within two percentage points of the teacher’s overall accuracy, while meeting a separately defined minimum recall target for high-cost categories. Check latency, cost, and the uncertainty in your evaluation results before replacing the teacher.
The order I’d follow
- Build labeled validation and test sets from real data.
- Try an existing model through an API, including the small ones.
- Route easy and hard requests to different models.
- Missing knowledge? Add retrieval.
- Inconsistent behavior? Fine-tune.
- Works well but too expensive at scale? Distill.
- Only then ask whether training from scratch makes sense.
This isn’t a strict sequence, and many products rightly stop at step 2 or 3. That’s not a failure. That’s the point.
The real gains in AI products rarely come from owning a bigger model. They come from understanding users, measuring honestly, and picking the cheapest tool that meets the bar. Building a model is one way to solve a problem. It’s rarely the first one worth trying.