What fine-tuning data really needs to look like, the 3 signs your data is not ready yet, and what nobody tells you about the labelling that produces it.
Artificial Intelligence Solutions
Looking for a artificial intelligence partner?
We build domain-led systems tailored to your industry and workflow. 12 years. 2,100+ engagements.
Every product team that decides to fine-tune a model runs into the same wall a few weeks in. "We thought we had the data." The training run gets scheduled, the vendor sends the specification, somebody on the team starts pulling examples from the database, and the sinking realisation hits: the examples are messy, incomplete, unlabelled, or not really representative of what the product does today.
The fine-tune either gets postponed while the data gets fixed, or it runs on bad inputs and produces a model that performs worse than the foundation model it was meant to improve. Both outcomes are expensive; both are avoidable if the data-readiness question is answered honestly before the fine-tune is scheduled.
So what does the data actually need to look like before you push the fine-tune button? That is the point of this piece. You will see what fine-tuning data really is, without the assumptions the vendor spec makes about your setup.
You will see the 4 data layers every fine-tune actually needs. You will see the 3 signs your data is not ready yet, and the honest thing to do when it is not. You will see the 4 things nobody tells you about labelling that data, and what it really costs.
You will see a simple pattern for turning what you have today into a fine-tune-ready dataset. And you will see the 3 mistakes that kill fine-tuning projects before they even start. All of it is written for the product owner making the call, not the data scientist running the training job, because the product owner is the one who has to justify the invoice on a fine-tune that did not work.
Why does this matter more this year than a few years back? Because fine-tuning has gotten cheaper to run and much easier to buy, which means more teams are running fine-tunes without the data preparation that used to be forced on them by the difficulty of the training itself. The vendor makes it look like a menu option.
The training runs. The model comes out. And then the team discovers the model is worse than what a good prompt on the foundation model produces, because the input data was not what fine-tuning needed. This is the fastest-growing failure mode in AI product work right now.
4
Data layers every fine-tune needs: representative examples, consistent labels, clean formatting, and a held-out evaluation set. Missing any one kills the training.
3
Signs your data is not ready yet: not enough examples, inconsistent labels between examples, and no held-out test set to measure the fine-tune's actual lift.
4
Things nobody tells you about labelling: it takes longer than you estimated, it costs more than you budgeted, and 2 people labelling the same thing disagree more than you expected.
1
Pattern that turns messy operational data into fine-tune-ready examples without a full data-engineering rebuild.
The rest of this piece walks the answer in the order the questions come up during a real conversation about fine-tuning readiness. What does the data really need to be? Which layers matter?
How do you know when you are not ready? What does labelling actually cost? How do you get from messy data to training-ready data?
And what kills these projects before they start? Boring on purpose, because the boring reading is the one that saves the fine-tune from being an expensive lesson.
Fine-Tuning Data, Without the Assumptions
What does fine-tuning data actually mean? A fine-tuning dataset is a collection of paired examples: an input the model will receive and the correct output you want it to produce in response. Each pair teaches the model the shape of your task.
Fine-tuning does not create new intelligence; it shapes the existing intelligence of the foundation model toward your specific pattern. Everything about whether the fine-tune works comes down to whether those input-output pairs are numerous enough, consistent enough, and representative enough of what your product will actually see in production.
So why is this harder than it sounds? Because the data most product teams have was never collected with fine-tuning in mind. Support tickets have the customer's question but not the answer that was actually given.
Historical decisions have the outcome but not the input that led to it. Documents are stored as PDFs with no structured extraction. Communication history sits in personal WhatsApp threads or somebody's email.
Turning that operational reality into paired input-output examples takes work, and the work has to happen before the fine-tune, not during. Vendors selling fine-tuning as a service rarely spell this out because it complicates the sales pitch.
What is the difference between fine-tuning data and retrieval data? Retrieval data is the pool of domain documents the model reads at question time to give you a grounded answer; it can be messy and updated freely because the retrieval layer picks what to send. Fine-tuning data is training material that goes into the model itself; it has to be clean, consistent, and paired correctly because the model is going to internalise the patterns in it, mistakes and all.
A bad retrieval library gives you occasionally bad answers you can fix by improving the library. A bad fine-tuning dataset gives you a model that has learned the wrong patterns, and undoing that requires training again on clean data. The tolerance for messiness is very different between the 2.
The Data-Readiness Question
If you cannot show a vendor 100 clean input-output pairs from your real product before the fine-tuning conversation gets serious, your data is probably not ready. The training run they are pitching will produce a model, but the model will underperform in ways nobody wants to explain in the post-mortem. Answer the data-readiness question honestly first; the fine-tune conversation gets much simpler on the other side.
The 4 Data Layers Every Fine-Tune Actually Needs
Which layers of your data have to be in place before a fine-tune has a chance of working? The 4 below are not optional; skipping any one of them is what turns a fine-tune into a model that performs worse than the foundation it started from. The map below shows what each layer looks like and why it matters.
The 4 Layers
What Every Fine-Tuning Dataset Actually Requires
Layer 1
Representative Examples
Real inputs your product actually receives, spanning the range of cases and edge conditions the model will meet in production. Not just the easy ones.
Layer 2
Consistent Labels
The correct output for each input, labelled the same way across the whole dataset. Two labellers should agree; inconsistent labels teach the model contradictions.
Layer 3
Clean Formatting
Consistent structure across every example: the same input shape, the same output shape, the same character encoding. Small format differences produce large training issues.
Layer 4
Held-Out Evaluation Set
A separate set of examples the training never sees, so you can measure whether the fine-tuned model actually got better or just memorised the training set.
Missing Any of These Kills the Fine-Tune
Layer 1 missing: the model learns a version of your task that will not generalise to real traffic. Layer 2 missing: the model learns to be inconsistent. Layer 3 missing: the training either fails or the model behaves strangely on production inputs. Layer 4 missing: you have no way to prove the fine-tune worked, or spot the moment it stopped working.
Why does missing one layer break the whole training run? Because fine-tuning is an amplifier. Whatever pattern is in the training data gets pushed harder into the model's behaviour.
Good pattern amplified equals a sharper, more useful model. Bad pattern amplified equals a model that confidently produces wrong outputs, and the confidence is worse than a foundation model that hedges when unsure. The 4 layers above are what keep the amplification pointing in the right direction; skipping any one lets the amplification work against you.
3 Signs Your Data Is Not Ready for Fine-Tuning Yet
How do you know whether your data is actually ready for a fine-tune, or whether you should wait? The 3 signs below are the ones that keep appearing in projects that ran the training too early. If you see more than one, the honest thing to do is pause the fine-tune, fix the data foundation, and come back. The training run that runs on ready data is dramatically cheaper than the one that runs on messy data and then has to be redone.
01
You Have Fewer Than a Few Hundred Clean Examples
Fine-tuning for a simple format or tone adjustment can start moving a model with a few hundred examples. Anything more complex needs meaningfully more. If you have 20 or 30 examples and are pitching for a fine-tune, you do not have a training dataset; you have a starting-prompt library. Use those examples as demonstrations in a good prompt with a foundation model first; that will get you further, faster, and cheaper than a fine-tune on the same small pile.
02
Two People Labelling the Same Input Disagree Often
Take 20 of your example inputs, have 2 people on your team label the correct output independently, and compare. If they agree less than 80 percent of the time, your task is not defined clearly enough to fine-tune yet. The model cannot learn a task the humans doing it disagree about. Fix the label definitions, run the exercise again, and only proceed when labelling agreement is high. This is the single most under-run pre-fine-tune check and the single most predictive of whether the training will actually work.
03
You Have No Held-Out Test Set That the Training Cannot See
If every example is going into the training set, there is no way to measure whether the fine-tune actually improved anything. Set aside a genuinely representative test set (a fifth to a tenth of your total examples) that the training never sees, and evaluate the fine-tuned model against it. Without this, "the fine-tune worked" is a claim without evidence, and you will discover the truth only when real users start complaining. Set the test set aside first, before the training, not after.
The Honest Wait
If any of these 3 apply to your data today, the honest move is to pause the fine-tune, fix the data, and revisit in a few weeks. This feels like slowing down; it is actually the fastest path to a working fine-tuned model. Teams that push through end up doing 2 training runs and paying twice; teams that pause do one training run and get a model that works.
4 Things Nobody Tells You About Labelling Fine-Tuning Data
What does labelling data actually look like in practice, once you get past the theory? The 4 truths below are the ones that surprise almost every team doing labelling for the first time, and the ones vendors tend to skip when pitching a fine-tune. Knowing them upfront turns labelling from a scoping surprise into a planned activity.
01
It Takes Longer Than You Estimated
Whatever your estimate is for how long it takes to label a batch of examples, roughly double it. The first pass is faster than the second, because the second forces you to standardise decisions you made ad-hoc the first time. Edge cases surface constantly and each one requires a labelling-guideline decision that then has to be back-applied to earlier examples. Complex tasks take longer per example than simple ones by an order of magnitude, not a little. Plan the labelling effort as a real project, not as a side task somebody does between other work.
02
It Costs More Than the Fine-Tune Itself Usually Does
The training compute for a fine-tune is often a modest bill. The people-time to produce clean labels for the training set is usually the larger cost, especially if you use domain experts (which you often need to). This surprises teams whose mental model of fine-tuning is dominated by the training cost. Plan the labelling budget alongside the training budget from the start, and price the fine-tune project as the sum of both, not just the flashy part.
03
Two People Labelling the Same Thing Disagree More Than You Expected
Have 2 people label the same 20 examples independently and compare. The agreement rate is almost always lower than the team predicted before running the exercise, especially on subjective or nuanced tasks. This is real information about how well-defined your task actually is. Low agreement means you have to sharpen the labelling guidelines before you can proceed. High agreement means the task is well-scoped and the fine-tune has a chance. Do not skip this exercise; it is the cheapest fine-tune insurance you can buy.
04
Labelling Guidelines Are a Real Document, Not a Note
Every fine-tune that produced a good model had a written labelling guideline that grew during the labelling process: definitions, edge case rulings, examples of correct and incorrect labels, decisions about ambiguous inputs. Teams that skip this end up with inconsistent labels because different labellers made different decisions on the same edge case. Treat the labelling guideline as a first-class artefact; it is what turns a group of people labelling into a coherent dataset the model can learn from.
The Cost Nobody Shows You
All 4 of these show up as line items on a real fine-tuning budget and none of them show up in the vendor pitch. Plan for labelling to be the largest cost, plan for it to take longer than expected, and plan to build the labelling guideline as you go. Teams that plan honestly hit the fine-tune within the budget they set; teams that trust the vendor's initial estimate blow past it and quietly compromise on data quality to hit the number.
The Data-Prep Path
5 Stages Between Raw Data and Fine-Tune-Ready
1
Collect
Pull the raw examples from real product traffic, tickets, and logs.
2
Clean
Deduplicate, strip PII, remove failed edge cases and formatting noise.
3
Label
Attach the desired output. This is where projects usually underinvest.
4
Validate
Second-reviewer pass on a sample, agreement rate, edge-case audit.
5
Version
Snapshot the dataset with metadata so future runs stay reproducible.
Where Fine-Tuning Projects Actually Stall
Stage 1 is usually done. Stage 2 is skipped. Stage 3 is underinvested. Stage 4 is skipped entirely. Stage 5 is forgotten until reproducibility becomes a problem. Every stage matters; skipping any of them shows up in the fine-tuned model's behaviour.
A Pattern for Building a Fine-Tune-Ready Dataset From What You Have Today
What if your data is messy, spread across systems, and does not look like fine-tuning data at all? Not fancy. The pattern below is the arrangement that lets your team turn operational reality (support tickets, chat history, product decisions, past outputs) into a dataset that a fine-tune can actually learn from, without doing a full data-engineering rebuild first.
Every layer has one job. When new data arrives, it flows through the same layers automatically.
Architecture
A Shape That Turns Messy Operational Data Into Fine-Tune-Ready Examples
Layer 1
Raw Collection
Pulls candidate examples from your operational systems: tickets, chat logs, past decisions, expert outputs. Stored as raw, timestamped, source-tagged.
Layer 2
Pair Extraction
Turns each raw record into an input-output pair. Support ticket becomes customer question plus agent answer; past decision becomes context plus outcome; expert output becomes brief plus final work.
Layer 3
Labelling and Review
Human reviewers apply the labelling guideline, correct outputs where needed, flag ambiguous cases for expert judgement. This is where the dataset earns its quality.
Layer 4
Split and Format
Splits the labelled pairs into training and held-out evaluation sets, formats them to the training vendor's specification, versions the dataset so the exact training data is reproducible.
↓
Cross-Cutting
3 Practices That Keep the Dataset Honest
Written Guideline
A living document that captures every labelling decision as it is made, so future labellers stay consistent.
Inter-Labeller Check
Regular exercises where 2 labellers work the same batch and compare, catching drift before it corrupts the dataset.
Dataset Versioning
Every dataset used in training gets a version. When a fine-tune regresses, you can identify which dataset produced it.
How to Turn Messy Data Into Fine-Tune Data
The pattern above is what turns "we have data, sort of" into "we have a training dataset". None of the layers is exotic; all of them are essential. Teams that build them once find every subsequent fine-tune becomes cheaper because the pipeline runs the same way with new data. Teams that improvise per training run pay the setup cost every time.
Why build this pattern before the first fine-tune, rather than after? Because the first fine-tune is exactly when you need the discipline the pattern enforces. Skipping it for the first run creates habits your team then has to unlearn for every subsequent run.
Building it once, upfront, means the second and third fine-tune project happens faster because the pipeline is already there. This is the leverage that turns fine-tuning from a one-off project into a repeatable capability.
3 Mistakes That Kill Fine-Tuning Projects Before They Even Start
What are the mistakes that make fine-tuning projects fail before the training even runs? The 3 below keep showing up, and each one is avoidable if the team knows to watch for it. Recognising them early is the difference between a fine-tune that ends in a working model and one that ends in a quiet write-off.
01
Treating Fine-Tuning as a Substitute for a Good Prompt
The team gets frustrated that a foundation model with a naive prompt is not producing what they want, and jumps straight to fine-tuning. Almost always, a well-designed prompt (with the right examples, the right instruction, the right structure) would have moved the model most of the way there. Fine-tuning should be the escalation when prompting has been genuinely tried and provably not enough. Skipping the prompting work means you are paying for a fine-tune to solve a problem the foundation model could have solved with better instructions, and the fine-tune will not add much over that baseline.
02
Skipping the Baseline Measurement
The team fine-tunes without first measuring how well a foundation model with a strong prompt performs on the same held-out test set. Then, after the fine-tune, they cannot say whether the fine-tuned model is better, the same, or worse than what they had before. Every fine-tune needs a baseline number captured before the training runs, and the fine-tuned model's score compared to it after. Without the baseline, the fine-tune has no evidence of improvement, and any regression will hide until it hurts a real user.
03
Not Planning for the Foundation Change Underneath
The team fine-tunes on a specific version of a specific foundation model. The provider deprecates that version or shifts behaviour on the successor. The fine-tune's advantage evaporates because the base model underneath moved. Every fine-tuning project needs a plan for re-training on the successor foundation when the current one gets retired, and a versioned dataset ready to re-run the training. Skipping this makes the fine-tune a one-time bet on a version that will not be there forever.
The Kill-Rate Filter
These 3 mistakes account for the majority of fine-tuning projects that fail without producing anything useful. Run each of them against your fine-tuning plan before the training gets scheduled. Have you tried prompting seriously?
Have you captured the baseline? Have you planned for the foundation change? Answering yes to all 3 puts you in the minority that produce a fine-tune worth keeping; answering no on any is a signal to fix that gap first.
Frequently Asked Questions
How many examples do you really need to fine-tune a useful model?
Depends heavily on what you are fine-tuning for. Simple format or tone adjustments can start moving a model with a few hundred well-labelled examples. Behaviour shaping on a complex task typically needs several thousand quality examples. Deep domain adaptation that would replace an industry-trained product can require tens of thousands or more, often with domain-expert labelling. What matters more than raw volume is whether the examples cover the range of real production inputs and whether the labels are consistent. A smaller clean dataset almost always beats a larger messy one.
Can you use synthetic data (data generated by an AI) to fine-tune?
Sometimes usefully, sometimes disastrously. Synthetic data works when it augments a real dataset to cover edge cases you do not have enough of, when the synthetic data is reviewed by humans before use, and when the underlying task is well-defined enough that the AI generating it produces mostly correct outputs. It fails when it becomes the majority of the training set (because the model learns whatever quirks are in the AI that generated the data), when it goes unreviewed, and when it is used to skip the effort of collecting real examples. The rule of thumb: synthetic data as a supplement to real data usually helps; synthetic data as a substitute for real data usually hurts.
How do you protect customer data used in fine-tuning?
Depends on your customer contracts, the applicable data-protection rules in the regions where your customers are, and whether you are using a closed-source vendor's fine-tuning service or fine-tuning open-source models yourself. In general: anonymise identifying information before it enters the training set, keep the raw customer data separated from the fine-tuning dataset, verify the training platform's data handling meets your compliance requirements, and document what data was used to train what model version. Vendors vary widely in how transparent they are about fine-tuning data handling; read the fine print, and consider open-source alternatives if the data sensitivity is high.
Should you use in-house labellers or a labelling vendor?
Depends on task complexity and domain expertise. Simple, well-defined labelling tasks (classification into obvious categories, extraction of clearly-defined fields) can work well with labelling vendors who handle the operational scale. Domain-heavy tasks (legal, medical, financial, or specialised operational work) usually require in-house labellers or vendors with domain-specific labeller teams. The wrong choice ends up with a large volume of inconsistent labels that hurt the fine-tune. Pilot with a small batch first, measure inter-labeller agreement, and scale up only if the pilot produces clean labels.
What is a held-out evaluation set and how do you build one?
A held-out evaluation set is a group of labelled examples the training process never sees, kept aside specifically to measure whether the fine-tuned model actually got better. Build it by taking a representative sample (usually a fifth to a tenth of your total labelled examples) that spans the range of production inputs, and setting it aside before training. The training runs on the remaining examples. After training, the fine-tuned model is scored on the held-out set and compared to the pre-fine-tune baseline on the same set. Without this, you have no way to prove the fine-tune worked. Build the held-out set first, always, before the training.
What happens if your training data becomes stale as your product evolves?
The fine-tuned model becomes stale with it. If your product's tasks shift, your customers change, or the pattern of inputs the model sees in production drifts away from what the training data captured, the fine-tune's advantage erodes. The pattern above (raw collection, pair extraction, labelling, split and format) is designed to be re-runnable, so refreshing the training data with recent examples is a routine cycle, not a one-time project. Plan the refresh cadence based on how fast your product is changing; static products can go longer between refreshes, fast-changing ones need the pipeline running continuously.
Can Entexis help you build the data foundation before a fine-tune?
Yes. Entexis designs and builds the data foundations that fine-tuning depends on. That work starts with the honest data-readiness assessment against the 4 layers above, so you know whether your data is ready today, close, or a real project away. We then design the pattern (raw collection, pair extraction, labelling and review, split and format) that turns your existing operational data into training-ready examples, help you develop the labelling guideline that keeps the dataset consistent, and wire the held-out evaluation set that lets you prove the fine-tune worked. We handle the training run itself only when the data foundation is genuinely ready; when it is not, we tell you honestly and help you fix the data first. Reach out with what you want the fine-tune to do, what data you think you have, and what your current AI baseline is, and we can walk through whether the fine-tune is the right next step for your product.
So where does that leave your fine-tuning project? The 4 data layers above are what a training run actually needs; skipping any one produces a model that performs worse than the foundation it started from. The 3 data-readiness signs tell you when to wait rather than train.
The 4 labelling truths are what most vendors do not put in the sales pitch. The pattern for building the dataset from what you have today is the alternative to a full data-engineering rebuild. The 3 project-killing mistakes are the ones that account for most failed fine-tunes and are all avoidable.
Answer the data-readiness question honestly first, plan for labelling as the largest cost, and only run the training when the data is ready. Teams that do this produce fine-tuned models that outperform the foundation baseline meaningfully; teams that push the training button before the data is ready produce models that quietly disappoint.
Want a Fine-Tune Built on Data That Actually Works?
At Entexis, we build the data foundations that fine-tuning depends on. We start with the honest data-readiness assessment against the 4 layers above, design the pipeline that turns your operational data into training-ready examples, develop the labelling guideline that keeps the dataset consistent, wire the held-out evaluation set that lets you prove the fine-tune worked, and run the training only when the data is genuinely ready. When your data is not ready, we tell you honestly and help you fix it first. Your fine-tune ends with a model that beats the foundation baseline on your specific tasks, not one that quietly disappoints. Start the conversation with Entexis.
Ready to Add AI to Your Business?
From intelligent chatbots to workflow automation, we build AI solutions that understand your domain, your data, and your users. Tell us what you need.
We'll get back within one business day.
Thank You!
We've received your message and will get back to you within one business day.
Try the AI workflows we build, for real, right now.
Same workflow patterns Entexis rolls into client setups. Try them in your browser, no signup. If one feels like it'd help your team, we build a private version tuned to your data.
