The VP of Data at a subscription software company in Amsterdam is asked by the leadership team to “use machine learning to cut churn”. After six weeks of vendor calls, three proposals arrive. All promise a model. All show slides from other industries. One even quotes a figure: “churn prediction accuracy of 94%”. None answers the obvious follow-up question. Ninety-four percent compared to what? The customer success team’s current habit of calling every account whose usage dropped last month? Predicting that nobody leaves? Nothing at all?
The contract is signed, a model is built, and the final presentation shows a lift chart in which the model’s curve sits comfortably above “random”. Six months later the customer success team still works from its usage-drop list. The model lives in a notebook on the consultant’s laptop, the consultant has moved to another client, and nobody in-house can retrain it on new data.
This article is for CTOs and heads of data considering external machine learning consulting services. It defends one position: you should buy ML consulting as a decision, not as a model. The first phase of an engagement should end with a go or no-go call, measured on a business metric against an honest baseline. “Don’t do ML” should be written into the contract as a legitimate, paid outcome. If a consultant cannot give that answer, do not hire them, because their income depends on you building more.
Why is “build us a model” a badly framed order?
Because it describes the means, not the goal. A model is a tool for improving a decision: which customer gets a retention offer, how much stock to order, which claim to review first, which payment to hold for manual review. If the order does not name that decision, the consultant optimises what they can measure on their own, which is a model quality score. A model quality score and business value are different things.
Martin Zinkevich at Google collected the lessons of years of building ML into Google products in a document called “Rules of Machine Learning”. Rule number one reads: “Don’t be afraid to launch a product without machine learning.” Rule number two: first, design and implement metrics. The order is not accidental. Before anything is trained, you need to know what happens today and how to measure it. “Build us a model” reverses that order and starts with the solution.
The second problem is financial. If the contract pays the consultant for delivering a model, a model will be delivered whether or not it makes sense. No bad faith is required. It is enough that everyone on the project knows that “this does not pay off” ends the engagement, while “we have a promising prototype” opens the next phase. A consultant who recommends stopping does so against their own interest, so the contract has to neutralise that interest.
The third problem is the definition of success. Without a business metric and a threshold at which deployment pays off, agreed in advance, every result can be presented as a win. An AUC of 0.87 sounds good until someone asks what the current rule scores and what each error costs in each direction.
A good order for ML consulting therefore reads differently: “We want to know whether a model can improve decision X enough to cover the cost of building and running it. If yes, we want a working pipeline and a rollout plan. If no, we want the measurement that shows it and a recommendation on what to do instead.”
When does machine learning consulting make sense, and when is an analyst or a rule enough?
ML consulting makes sense when four conditions hold at once. First, there is a repeated decision made many times: hundreds of times a day, not a few times a year. Models earn their keep through scale, because each individual improvement is small. Second, there is historical data about the decision and its consequences. Knowing who received an offer is not enough; you need to know who accepted it. Third, the current way of deciding has a measurable cost of error. Fourth, you do not know whether ML will help, and that uncertainty is blocking a budget decision.
If the first condition is missing, the decision is rare and strategic, and an analyst with a good spreadsheet model will serve you better than a learning system. If the second is missing, the first engagement should be about data, not models; we covered that in our piece on data management services. If the third is missing, you cannot calculate whether a model pays off. If the fourth is missing because you already know you need a model, you are buying ML development services or team augmentation, not consulting.
A separate case is the board that wants “an AI project” without anyone having named a decision to improve. ML consulting should not start with a model there. It should start with a review of processes and the choice of one place where the effect can be measured. It is not unusual for the biggest gain to come not from a model but from a simple rule nobody had written down, or from a process change that removes manual re-keying of data.
Zinkevich’s third rule cuts the other way: a complex heuristic is unmaintainable. If your systems have accumulated years of rules and exceptions until nobody can explain why a customer received a particular discount, a model trained on data may be simpler to maintain than the rulebook. A good consultant should be able to weigh both sides of that equation.
How does ML consulting differ from ML development services and from deploying an off-the-shelf AI tool?
The three are often sold under one label, but they differ in who carries the risk and what the deliverable is.
ML consulting answers “whether and how”. The deliverable is a decision backed by measurement and, if the decision is positive, a working prototype, a target architecture and a rollout plan. The risk that ML does not pay off is explicitly on the table, and resolving it is what you pay for. The engagement is short, measured in weeks rather than quarters.
ML development services answer “build it”. They assume the decision to deploy has been made. The deliverable is a production system with a training pipeline, monitoring and documentation. The risk is mostly engineering risk: deadlines, code quality, integrations. If you buy ML development before resolving whether ML pays off, you pay to build something that may turn out to be unnecessary.
Deploying an off-the-shelf AI tool, such as a forecasting module in your ERP or a managed cloud service with a pre-trained model, answers “configure it”. It is cheapest at the start but gives you the least control over what the model does and what it learns from. We covered how to evaluate vendors of such tools in our article on AI vendor selection.
A good consulting engagement ends with a recommendation for one of these three paths, or a recommendation to take none of them. A consultant who always recommends building from scratch is selling ML development under a consulting label. One who always recommends the same product may be its reseller. Ask directly what recommendations they made in recent engagements and how often they were negative.
What should the brief contain before the first conversation with a consultant?
The brief is a document the client writes, not the consultant. The better the brief, the shorter and cheaper the first phase, because the consultant does not have to rediscover what someone in your company already knows. One or two pages are enough, as long as they are specific.
The decision to improve. One sentence: “Every Monday the customer success team decides which 120 of our 9,000 accounts to contact proactively this week.” Not “we want to reduce churn”.
How the decision is made today. A rule, a spreadsheet, an expert’s judgement, a vendor module. This will be the first baseline, so it must be described precisely enough to be reproduced on historical data.
The cost of error in both directions. What a missed churner costs (lost annual contract value, say EUR 18,000 on average) and what a wasted call costs (an hour of a customer success manager’s time, plus the risk of discount leakage if the call ends in an unnecessary concession). These costs are rarely symmetric, and the asymmetry changes everything about how a model should be evaluated.
The available data. Which systems, which period, which known quality problems. Who owns the data, and whether the consultant will have access in week one rather than week six. Delayed data access stretches a first phase more than any modelling difficulty, because the consultant has nothing to work on in the meantime.
Constraints. Whether the model must be explainable to customers or regulators, whether the decision concerns natural persons, what response time is required, whether data may leave your infrastructure or region.
An owner on your side. Someone with time for weekly reviews, who knows the process and has the authority to accept the phase outcome. Without that person the consultant optimises for their own assumptions.
The minimum effect that justifies deployment. This is the hardest item and the one most often skipped. If the model saves five extra accounts a quarter, does that cover the cost of running it? Fifteen? Even a rough threshold written down before work starts prevents every result from being declared a success.
Why is the baseline the first deliverable rather than a formality?
A baseline is the result the same data produces without the new model. Without one, any model score is a number without context. A churn model with 95% accuracy looks excellent until you learn that 5% of customers churn, and a “model” predicting that nobody leaves also scores 95%.
This is not a theoretical worry. scikit-learn, one of the foundational libraries of the Python ecosystem, ships a component called DummyClassifier that makes predictions ignoring the input features entirely. Its documentation describes it as a “simple baseline to compare against other more complex classifiers”. If a consultant does not show a naive result next to their own, you cannot tell how much of the score comes from the model and how much from the structure of the data.
A good consultant builds at least three baselines, not one:
- A naive baseline: always the most frequent class, last period’s average, the value from the same week last year. It shows how much information sits in the structure of the data alone.
- A current-process baseline: today’s rule or decision, reconstructed on historical data. This is the model’s real competitor, because it is what the model must beat for deployment to make sense.
- A simple-model baseline: linear or logistic regression, a shallow decision tree, a model with a handful of obvious features. It shows what machine learning itself contributes before anyone reaches for complex architectures.
Zinkevich’s fourth rule says to keep the first model simple and get the infrastructure right, because the first model provides the biggest boost. If a simple model beats the current process by 80% of what a complex model achieves, choosing between them is a decision about maintenance cost, not quality.
The current-process baseline has a further benefit: it forces the company to write down how it actually decides today. Sometimes writing down the rule the team applies intuitively exposes its weaknesses on its own. Sometimes reconstructing an expert’s decisions on historical data shows that the expert is far better than anyone thought, leaving the model very little room to improve. Both findings are valuable, and both belong in the report from the first weeks, before any complex model exists.
How do you translate a model metric into a business metric you can actually account for?
Model metrics such as accuracy, precision, recall, AUC or mean absolute error describe how well the model predicts. They say nothing about what that is worth. The translation needs three inputs: the cost of each type of error, the number of decisions in a period, and the way the model’s output changes what people or systems do.
Take the churn example. The model raises precision among the top 120 accounts flagged each week from 22% (the usage-drop list) to 34%. What does that mean for the company? It depends on what the customer success team does with the list. If a proactive call saves, say, one in four genuine churners, twelve extra percentage points of precision means about 14 more true churners reached per week and three or four more saved. At EUR 18,000 average contract value, that is a number the CFO can check. If the team ignores the list and calls whoever it was going to call anyway, the improvement is worth nothing until the process changes. These figures are illustrative; the point is the arithmetic, not the values.
The second trap is cost asymmetry. In fraud detection, a missed fraud costs the transaction value, while a false alarm costs an analyst’s time and customer goodwill. A model with higher overall accuracy can be worse for the business if it errs more often in the expensive direction. The decision threshold, the probability above which a case is flagged, should therefore be set from costs, not left at the default 0.5.
The third trap is process capacity. If a model flags 500 suspicious claims a day and the team can review 80, what matters is precision in the top 80, not performance across the whole set. A consultant who reports AUC without reference to team capacity is reporting a metric nobody in your company will use.
| Model metric | Business question to ask | Translation into value |
|---|---|---|
| Accuracy | How often do we err in each direction today, and what does each error cost? | Errors of each type × unit cost, compared with the current-process baseline |
| Precision in the top N | How many cases can the team handle per day or week? | Correct interventions within capacity × value of one intervention |
| Recall | What does a case we failed to catch cost? | Value of cases caught minus the cost of false alarms |
| Forecast error | How does the forecast drive stock, orders or staffing? | Change in safety stock or overtime resulting from lower error |
| Latency | Is the decision made in real time or in a nightly batch? | Infrastructure cost required to meet the requirement |
Zinkevich advises choosing a first objective that is simple, observable and attributable, so that it is clear the model, and not something else, moved the result. That is a good rule for the consulting contract as well: the metric agreed at the start should be one your company can compute on its own after the engagement ends, without the consultant’s tooling. If it is too complex to compute in a spreadsheet, simplify it, even at the cost of some information.
What does honest offline evaluation look like, and where does it fool itself most often?
Offline evaluation tests the model on historical data it did not see during training. It is necessary, cheap and fast, but it has one weakness: it is very easy to run it in a way that makes the result look better than reality. This is rarely deliberate. It usually comes from small decisions whose consequences nobody checked.
The most dangerous mistake is data leakage: the training data contains information the model will not have at the moment of decision. A classic example: a churn model uses “number of contacts with the retention team” as a feature, but the retention team only contacts customers after they have asked to cancel. The model looks brilliant because it predicts the outcome from its symptom. Sayash Kapoor and Arvind Narayanan of Princeton University surveyed ML-based scientific literature in 2022 and found leakage-related errors in 17 fields, affecting 329 papers. If researchers make this mistake in peer-reviewed work, a consulting project written under deadline pressure can make it too.
The second mistake is random splitting in time-dependent problems. If a model forecasts the future and the data has been split randomly into training and test sets, the model learns from March to “predict” February. In production it will never see the future. Honest evaluation in such problems splits data chronologically, training on older periods and testing on newer ones, ideally across several time windows to see whether the result is stable.
The third mistake is repeatedly peeking at the test set. Every time the team checks the test score and adjusts the model in response, the test set becomes part of training. After twenty iterations the test score is optimistic. The remedy is a separate holdout set nobody touches until the end of the phase, ideally stored on your side rather than the consultant’s.
The fourth mistake is training-serving skew. Historical data is often cleaned, backfilled and joined in a warehouse, while at decision time the system sees only part of it, sometimes late. Zinkevich devotes several rules to this, including saving the set of features used at serving time and training on it, and warning that data in joined tables may change between training and serving.
What follows for the buyer? An offline evaluation report should include a description of the split with dates, a list of features stating when each becomes available in the process, a result on a holdout set used exactly once, and a comparison with all three baselines on that same set. If any of these is missing, ask for it before deciding on the next phase. That is not distrust. It is the standard a good consultant applies anyway.
When is offline evaluation not enough, and how should you plan an online test?
Offline evaluation tells you whether the model would have predicted the past well. It does not tell you whether deploying it will change a business outcome. Between the two stand people, processes and customer behaviour, all of which respond to the model’s decisions. A recommendation model that accurately predicted purchases on historical data may, once deployed, recommend products customers would have bought anyway, adding nothing to revenue.
That is why the “go” criterion should rest on a test on live traffic. Ron Kohavi, Diane Tang and Ya Xu set out the methodology of online controlled experiments in “Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing” (Cambridge University Press, 2020), drawing on practice at companies such as Microsoft, Google and LinkedIn. The core idea is simple: some cases are handled by the new approach, some by the existing one, assignment is random, and the metric and minimum effect are fixed before the test begins.
In ML consulting, the practical sequence usually looks like this:
- Shadow mode. The model scores live data but does not affect decisions. This shows whether production results match offline evaluation, whether data arrives on time and whether training-serving skew exists. It typically runs for several weeks, long enough to cover normal weekly or monthly cycles.
- A limited experiment. The model makes or supports decisions for a share of cases: some regions, some account segments, a random share of customers. The rest follow the existing process. The comparison is on the business metric, not the model metric.
- A rollout decision. Based on the experiment and a threshold agreed in advance, the company decides on full rollout, improvement or withdrawal.
Not every problem allows a classic A/B test. In demand forecasting it is hard to randomly assign interdependent products to “model” and “spreadsheet”. You can compare regions or warehouses, use a control period or test on part of the range instead. The consultant should propose a test design that fits your process and be candid about its limits, because each of these alternatives produces weaker evidence than a randomised experiment.
Online tests also involve people. If the model supports employees’ decisions, you need to know how often they follow its suggestions and how often they override them. We covered how to design the point at which a person accepts or rejects a model’s output in our article on human in the loop. The override rate is often the fastest signal that the model is solving a different problem from the one people actually have.
What exactly must be handed over so the model does not die when the consultant leaves?
The opening scene, a model in a notebook on the consultant’s laptop, is the typical ending of an engagement in which nobody defined the handover. Google’s MLOps architecture documentation calls this maturity level 0: a manual, script-driven and interactive process in which data scientists hand over a trained model as an artifact to the engineering team, a new model version is deployed only a couple of times per year, and predictions are not tracked or logged. In that state, a decline in model quality can go unnoticed for months.
The same documentation notes that only a small fraction of a real-world ML system is ML code; the rest is configuration, data verification, serving and monitoring. That observation comes from “Hidden Technical Debt in Machine Learning Systems” by D. Sculley and colleagues (NIPS 2015), in which a Google team warned that quick wins in ML do not come for free and carry high ongoing maintenance costs: entangled features, hidden feedback loops, data dependencies, changes in the external world. A consultant who hands over only the model hands over the smallest part of the system.
The handover list should be part of the contract and cover:
- Data preparation code from raw sources to the training set, runnable in your infrastructure, with dependencies and library versions documented.
- Training and evaluation code, including the code that computes the baselines, so every future retrain can be compared against the same reference point.
- A pipeline, not just scripts: a sequence of steps that runs with one command or on a schedule, on new data, without manual fixes.
- A model card. Margaret Mitchell and colleagues proposed in 2018 a standard document describing a model: its intended use, the conditions under which it was evaluated, results across groups and its limitations. For the buyer, it is the document that says what the model must not be used for.
- A monitoring specification: which indicators to track (input distribution, prediction distribution, the business metric), which thresholds signal a problem and what to do when they are crossed.
- A retraining plan: how often, on what data, and who decides whether a new version goes live.
- Knowledge-transfer sessions with the people who will maintain the system, ending with your team running the pipeline on its own, without the consultant.
The last item is the test of the whole list. If your team cannot retrain the model on its own and compare the new version with the previous one, the handover has not happened, however many documents were produced. We wrote more about what happens to a model after go-live, and why data quality rather than the model drives cost in later years, in the real cost of AI. The practical side of moving from notebook to production is covered in MLOps in practice, and the vocabulary in our glossary entry on MLOps.
Who will maintain the system? If you have no ML engineers, the handover lands with a team that cannot use it. Part of the consultant’s recommendation should then be the missing skills: often one Python engineer with data pipeline experience is enough, not a full data science team. Such specialists can also join your in-house team as Python developers on an augmentation basis.
Why can “don’t do ML” be the most valuable outcome of an engagement, and how do you write it into the contract?
Because building a model is only the start of the spending. A production model needs monitoring, retraining, pipeline maintenance, and responses to changes in source data and in the world that make past patterns stop working. We collected cost estimates from prototype to production in our article on AI implementation cost. If a model improves a decision by less than it costs to run, deploying it is a loss, even if it works technically.
“Don’t do ML” comes in several typical forms, each valuable:
- A simple rule is enough. The simple-model baseline, or a well-specified rule, achieves almost as much as the complex model. Recommendation: implement the rule in an existing system, with no new infrastructure.
- The data carries no signal. No model clearly beats the naive baseline, because the data does not contain information that predicts the outcome. Recommendation: start collecting the missing data and revisit after a defined period.
- The data quality is too poor. A signal probably exists, but errors, gaps and inconsistencies drown it. Recommendation: a data quality project first, then a fresh assessment.
- The process will not use the model. The model works, but the process has no place for its output, or people have no way to act on it. Recommendation: change the process before building the model.
- Regulatory risk outweighs the benefit. The decision affects natural persons in a way that requires explainability, oversight and documentation whose cost exceeds the gain.
For such an outcome to be possible, the contract must anticipate it. The first phase should have a fixed price and a scope defined as “a decision”, not “a model”. Fees must not depend on whether the recommendation is positive. The recommendation criteria, the minimum effect on the business metric, should be written down before work starts. It is also worth stating that, whatever the recommendation, you receive all code, intermediate data and the report, because a negative result is an asset too: it prevents the same study from being repeated two years later by the next vendor.
There is a less obvious argument as well. A consultant who says at the outset that one possible recommendation is to drop ML is signalling that they sell an answer, not an outcome. That makes a good selection test: ask under what conditions they would recommend stopping. If they cannot name any, they will probably never recommend it.
How should you structure phases, milestones and fees?
Structure an ML consulting engagement in phases, each ending with your decision rather than an automatic move to the next one. This limits risk on both sides and gives each phase either a fixed price or a clearly capped time-and-materials budget.
| Phase | Typical duration | What is produced | Criterion for moving on |
|---|---|---|---|
| 0. Brief and data access | 1–2 weeks (client side) | Decision description, error costs, data access, named owner | Consultant confirms the data is sufficient for a first assessment |
| 1. Feasibility assessment | 3–6 weeks | Three baselines, candidate model, offline evaluation with split description, value of the difference | Model beats the current-process baseline by more than the agreed threshold; otherwise a “do not deploy” report and stop |
| 2. Production prototype | 4–8 weeks | Pipeline in your infrastructure, shadow mode, monitoring, model card | Shadow-mode results consistent with offline evaluation; no training-serving skew |
| 3. Experiment | 4–12 weeks | Test on a share of cases, measurement of the business metric | Effect on the business metric exceeds the threshold; the team uses the output |
| 4. Handover | 2–4 weeks | Documentation, sessions, independent retrain by your team | Your team retrains and compares a new model version on its own |
The durations in the table are an estimate, not a market norm. They depend mainly on three factors: how quickly you grant data access, how long the business cycle is (a weekly forecast needs a longer shadow period than a decision made every minute), and whether the model must be integrated with an existing transactional system.
Match the fee model to the phase. Phase 1 works best at a fixed price, because its scope is a decision, not an outcome. Phases 2 and 3 are often better billed on time and materials with a cap, because their course depends on how systems and people on your side behave. The handover phase should have an acceptance criterion defined as your team’s capability, not a page count.
Not every engagement will go through all phases. Some will end after phase 1 with “do not deploy”, and that is a good result if it cost a fraction of what building would have. Some will pass phases 2 and 3 and then show a smaller business effect than offline evaluation suggested. That is also knowledge worth paying for before full rollout rather than after it.
Which regulations change the scope of an ML engagement in the European Union?
If a model will influence decisions about natural persons, widen the scope to cover legal requirements before the first model is built. For companies operating in or selling into the EU, two instruments matter most.
The first is the GDPR. Article 22 gives the data subject the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning them or similarly significantly affects them. For an ML engagement this means establishing whether the model decides alone or supports a human decision, and designing the process so that human involvement is real rather than a rubber stamp.
The second is the EU AI Act, Regulation (EU) 2024/1689. Its Annex III lists areas in which AI systems are considered high-risk. They include systems intended for recruiting or selecting candidates, systems that shape decisions on promotion, termination or task allocation, and systems that evaluate the creditworthiness of natural persons or establish their credit score, with fraud detection excluded. If a project touches one of these areas, the requirements for documentation, data governance, human oversight and quality are considerably higher, and their cost should appear in the phase 1 feasibility assessment.
The practical conclusion is simple: the brief and phase 1 must answer whether the decision concerns natural persons and whether it falls within Annex III. The answer can change the recommendation. A scoring model that offers only a small improvement over a rule may become uneconomic once compliance costs are added, and a rule that is easier to explain to customers and supervisors then wins not on technical grounds but on economic ones. If you also operate in the US, ask your counsel which sector-specific rules apply to automated decisions in your industry before phase 1 begins, and include the answer in the brief.
How can you tell that an ML consultant is working for your outcome, not for the next phase?
Several signals appear in the first weeks and show whether the engagement is heading in the right direction.
The first report contains baselines before it contains a model. If after two weeks you see a complex model’s score but not the current process reconstructed on data, the order is reversed.
The consultant asks about error costs and process capacity. Questions about what each type of mistake costs and how many cases people can handle show they are thinking about the decision, not the metric.
The code runs in your infrastructure from day one. Working on data copied to the consultant’s laptop is faster at the start but leads straight to the opening scene of this article. A repository on your side and an environment the consultant works in also make handover far easier.
The report describes what the model cannot do. Every model has areas where it performs poorly: new products with no history, rare customer segments, unusual periods. A report that lists none has either not been checked or is hiding something.
The recommendation is conditional and numeric. “We recommend rollout if, in shadow mode, precision in the top 120 stays above X for four weeks” is a recommendation you can verify. “The model is promising and worth developing further” is an invitation to the next engagement.
The consultant talks about running costs. If the report includes build cost but no estimate of monitoring, retraining and pipeline maintenance, the board’s decision will rest on half the bill.
Warning signs that deserve a response: no client access to the code until the end of the phase, results reported only as model metrics, no description of the data split, the success metric changed mid-phase without the business owner’s agreement, and a recommendation for further work with no criterion for when that work would end.
What are the key takeaways before hiring machine learning consulting?
Machine learning consulting is worth its price when you buy a decision from it, not an artifact. A model is one possible outcome, not the purpose of the engagement. The purpose is an answer to whether machine learning will improve a specific decision enough to pay for building and running it, backed by a measurement your company can reproduce on its own.
Remember seven things. First, write a brief that names the decision to improve, the cost of error in both directions and the minimum effect that justifies deployment. Second, require three baselines: naive, current process and simple model, computed on the same data as the candidate model. Third, translate model metrics into money or hours, accounting for cost asymmetry and process capacity. Fourth, check offline evaluation for leakage, chronological splitting and a single-use holdout set, and base the deployment decision on an online test. Fifth, write a handover list into the contract whose acceptance test is your team retraining the model unaided. Sixth, make “do not deploy ML” a legitimate phase 1 outcome, paid the same as a positive one. Seventh, if the decision concerns natural persons, check GDPR Article 22 and Annex III of the AI Act before the first model is built.
If the recommendation is positive and you need a team to build the pipeline, integrate the model with your systems and hand it over for maintenance, explore our custom software development services, or book a call and we will go through your brief and tell you what is still missing for the first phase to produce an answer rather than more questions.