Fine-Tuning & Distillation

LLM Fine-Tuning Services

The repetitive 80% of your traffic does not need a frontier model

Most production LLM traffic is the same handful of tasks repeated a million times — classify, extract, rewrite, route. Those tasks rarely need a frontier model. We fine-tune small open-weight models on your own data, prove they hold accuracy against a real evaluation set, and serve them at a fraction of the per-token cost. You keep the weights.

LORAQLORADISTILLATIONDPOEVALSSLMQUANTISATIONVLLM

Evals first

The scoring set is built before any training run starts

Open weights

Permissively licensed bases you can host and modify

Honest baseline

Every model scored against the API it would replace

3–8 weeks

Typical time from dataset design to a served endpoint

Why Zenthos

Most teams who ask for fine-tuning need better retrieval

Fine-tuning changes how a model behaves. It is a poor way to teach it what is true. When a team tells us their model ‘doesn’t know our product catalogue’ or ‘gets our refund policy wrong’, the fix is almost always retrieval — those facts belong in an index you can update on a Tuesday afternoon, not baked into weights you would have to retrain. Fine-tune a fact in and it goes stale the moment the fact does.

The second most common request is output shape: responses too long, tone slightly off, JSON that is valid four times out of five. Try a tighter prompt first, then few-shot examples, then a structured-output schema or constrained decoding. Those cost an afternoon. A fine-tune costs weeks and hands you an artefact you now have to version, evaluate, host and monitor forever. Do the cheap thing until it demonstrably stops working.

Then there are the cases where fine-tuning is clearly right, and they are worth being precise about. Cost at volume: when one narrow, repetitive task runs at tens of millions of tokens a month, the gap between frontier API pricing and a small model on hardware you amortise stops being close. Latency: a quantised 7B model on a GPU you control answers in a fraction of the time, with no network hop and no shared rate limit. And behaviour that resists prompting: rigid output formats held at high reliability, a house style prompts keep drifting away from, or domain vocabulary — clinical, legal, industrial — the base model has barely seen. If your problem is on that list, the economics usually work. If it is not, we will say so on the first call.

What We Deliver

Core Capabilities

[01]

Training Dataset Construction

The dataset is the project. We mine your logs, tickets and documents for real examples, deduplicate, decontaminate against the held-out split, fix label noise and balance the classes that actually matter. A thousand clean, representative pairs routinely beat fifty thousand scraped ones.

[02]

LoRA & QLoRA Fine-Tuning

Parameter-efficient tuning on open-weight bases — Llama, Qwen, Mistral, Gemma. Rank, target modules, learning rate and schedule are chosen by short sweeps against your task, not copied from a blog post. QLoRA when the model has to fit on a single GPU.

[03]

Distillation From a Frontier Model

Use the expensive model where it earns its keep: producing training data. We sample your real traffic, have a frontier model generate reference outputs, filter with rejection sampling and human review, then teach a small student to match on the distribution you actually serve. Teacher licence terms are checked before we start.

[04]

Evaluation Harness & Regression Gates

A scored test set drawn from real traffic, with metrics that fit the task — exact match, F1, schema validity, an LLM judge only where nothing better exists — plus the frontier baseline you are replacing. Regressions fail the run. No checkpoint ships on vibes.

[05]

Quantisation & Serving

GPTQ or AWQ quantisation, vLLM or TGI with continuous batching, and a load test at your real concurrency so throughput and p95 latency are measured rather than assumed. Deployed into your VPC, your cluster or your on-premise GPUs.

[06]

Continuous Retraining

Traffic drifts and accuracy follows it down. We wire in feedback capture, drift detection on the input distribution, and a retraining pipeline that can rebuild, re-evaluate and promote a checkpoint on a schedule without someone babysitting a notebook.

How It Compares

Fine-tuning vs. RAG vs. prompt engineering

Three ways to make a model do what you want. They solve different problems, cost different amounts, and fail in different places.

Prompt engineeringRAGFine-tuning
Best whenThe task is close to something the base model already does wellThe answer depends on facts that live in your documents or databaseA narrow task runs at high volume, or behaviour must be rigidly consistent
Cost to set upHours. One engineer, one afternoon.Days to weeks — chunking, embeddings, an index, a retrieval evalWeeks — dataset construction dominates, not GPU time
Cost per request at volumeHighest — frontier prices on every call, and the prompt only growsHigh — retrieved context inflates input tokens on every single requestLowest — small model, short prompts, fixed hardware you amortise
Handles new factsOnly what you paste into the promptYes — update the index, no retrainingPoorly. New facts mean a new training run, and they go stale between runs
Handles new format or tonePartly — holds until the edge cases pile upNo — retrieval does not change how the model writesYes — this is what fine-tuning is genuinely good at
LatencyFrontier API round trip, plus a long prompt to processSlowest — a retrieval hop before the model call, every timeFastest — small model, short input, no network hop if self-hosted
Maintenance burdenLow, until the provider changes the model underneath youOngoing — index freshness, chunking strategy, retrieval qualityHighest — versioned weights, an eval suite, and serving infrastructure
Fails whenVolume makes per-call cost hurt, or consistency has to be guaranteedThe problem was never about missing factsThe task is broad, the data is thin, or requirements change monthly

Try them in that order — prompt, then retrieval, then fine-tuning — and stop as soon as the numbers are good enough. They also compose: the strongest production systems we see run a fine-tuned small model over the routine majority, with retrieval supplying the facts and a frontier model held in reserve for the hard tail. Fine-tuning is the last of the three to reach for, and anyone recommending it before you have tried the other two is selling rather than diagnosing.

How We Work

Our Process

01

Feasibility & Cost Model

Before anything is spent on GPUs we answer two questions against a sample of your real traffic: can a small model hold the accuracy bar, and does the arithmetic work. The shape is simple — monthly token volume multiplied by the price gap between the frontier API and self-hosted inference, set against a one-off training cost plus a fixed monthly hosting bill. Below a crossover volume the fixed cost never amortises and you should stay on the API. We tell you when that is the answer.

02

Dataset Construction

We assemble and clean the training set from your logs, tickets and documents, cutting the evaluation split before any training begins and decontaminating the training data against it. Where volume is thin, a frontier model generates candidates under rejection sampling and a human reviews them. Most of the calendar goes here, and that is the right place for it.

03

Train & Iterate

LoRA or QLoRA adapters on an open-weight base, with short sweeps over rank, learning rate and target modules rather than one hopeful long run. Every checkpoint is scored on the held-out set and every run is logged, so you can trace the artefact you deploy back to the exact configuration that produced it.

04

Benchmark Against the Incumbent

The only comparison that matters is against the system you are replacing. We score the fine-tuned model and your current frontier prompt on the same set with the same metrics, and report task accuracy, cost per thousand requests and p95 latency side by side. If the small model loses, that is a legitimate outcome — and you found out for the price of an experiment rather than a migration.

05

Serve, Monitor & Hand Over

Quantisation, vLLM serving, a load test at your real concurrency, and deployment into your infrastructure. You receive the weights and adapters, the training and evaluation code, the dataset pipeline and a retraining runbook. Drift monitoring and a scheduled retrain are wired in if you want them.

Tools & Technologies

Tech Stack

We choose the right tool for the job — not the trendiest one.

PyTorch logoPyTorch
Python logoPython
Hugging Face
LoRA / PEFT
Axolotl
vLLM
Weights & Biases
Modal
AWS logoAWS
Questions

Frequently Asked Questions

Do we actually need fine-tuning?
Probably not, and we would rather say that in week one than month three. If the complaint is that the model does not know something, you need retrieval. If the output is the wrong shape, try a tighter prompt and a structured-output schema first. Fine-tuning earns its place in three situations: high-volume narrow tasks where per-token cost dominates, latency budgets a hosted API cannot meet, and behaviour that prompting keeps drifting away from. Send us a traffic sample and we will tell you which of those you are in, if any.
How much training data do we need?
Less than most people expect for format and style work — a few hundred to a few thousand clean, representative examples will often move a narrow task. Considerably more for anything that needs broad reasoning. Quality dominates quantity: a small curated set beats a large scraped one, because the noisy set teaches the model your labelling mistakes alongside your task. If you do not have enough real data, distillation from a frontier model is usually the cheapest way to manufacture it.
How long does a fine-tuning project take?
Three to eight weeks for most engagements. GPU time is rarely the bottleneck — a LoRA run on a 7B model is hours, not days. The calendar goes into dataset construction, building an evaluation set your team will actually trust, and the serving work at the end. Projects that run long almost always run long on data, not on training.
What does it cost — to train, and to serve?
Two separate numbers that people routinely conflate. Training a LoRA adapter on a small open-weight model is the cheaper one: rented GPU hours for a typical run are a minor line item, and the engineering time around the data is the real expense. Serving is the recurring cost — a dedicated GPU instance bills the same whether you send it one request or a million. That is precisely why volume decides the question. Below your crossover point the API wins on cost; above it, the fixed monthly bill amortises across every request and keeps amortising. We model that curve with your own traffic numbers before you commit to anything.
Will a fine-tuned small model beat GPT-class models?
On a narrow task with good data, often yes — a small fine-tuned model can match or beat a frontier model at classification, extraction, routing and format-constrained generation, at a fraction of the cost and latency. In general, no, and you should be sceptical of anyone who tells you otherwise. A 7B model will not out-reason a frontier model on open-ended work. The architecture that usually wins is both: the small model handles the routine majority, and the hard tail routes up to the frontier model.
Who owns the weights?
You do. We work with open-weight base models under permissive licences, and the adapters, merged checkpoints, training code, dataset pipeline and evaluation harness are all delivered into your repository and your model registry. There is no runtime dependency on Zenthos and no per-inference fee. We do read base-model and teacher-model licence terms before starting, because some restrict commercial deployment or the use of outputs for training — that is a week-one conversation, not a week-ten surprise.
Can you run it on our own infrastructure?
Yes, and that is the usual arrangement. We deploy into your VPC, your Kubernetes cluster or your on-premise GPUs, and for regulated environments the entire pipeline — data preparation, training and inference — can run without anything leaving your boundary. If you would rather not operate GPUs at all, we will set you up on a serverless inference platform and be honest about the cold-start behaviour that comes with it.
Do you work with companies in the US, UK and EU?
Most of our engagements are with teams in the US, UK and EU. We work overlapping hours with both US coasts and European time zones, and we are used to operating inside client security reviews, DPAs and procurement processes. For EU clients we can keep training data and inference inside EU regions where residency requirements apply.
Insights

Related Articles

View all →

Find out whether fine-tuning is worth it

Send us a sample of your real traffic and your current monthly inference bill. We will model the crossover and tell you honestly whether to fine-tune, fix your retrieval, or leave it alone for now — before you spend anything on GPUs.