Chapter 08

Predictive Modeling for Marketing Decisions

From Explaining the Past to Scoring the Future

Dr. Jose Mendoza, Academic Director and Clinical Associate Professor

Version 1.0 · July 2026

Except where otherwise noted, this chapter is licensed under CC BY 4.0.

Chapter Information

ABSTRACT

This chapter develops predictive modeling as a distinct analytic practice with its own obligations and failure modes. It builds the predictive frame — unit, target, features, and horizon — and the label as a manufactured measurement, with a snapshot date placing a wall in time between what a model may know and what it must predict. Leakage is defined as the canonical error and cataloged by route. Honest evaluation follows train/test splits and the in-sample flattery they defeat; mean and last-value baselines operationalized as stated opponents; MAE and RMSE read in dollars beside capture at the program’s capacity as the decision metric; overfitting and underfitting read from the error signature; and training-only cross-validation as the instrument that keeps the test set sealed. The labs build a spend-prediction pipeline, compare candidates on a frozen leaderboard, manufacture and catch a leaking model, and grade a vendor’s near-perfect accuracy claim.

KEYWORDS

predictive modeling; leakage; train/test split; baselines; overfitting; cross-validation; MAE; RMSE; expected value; model drift

VERSION AND DATE

Version 1.0 · July 2026 · Language: English (United States)

SUGGESTED CITATION

Mendoza, J. (2026). Predictive modeling for marketing decisions. In Applied business analytics for marketing decision-making: Business analytics and data visualization (Chapter 8, Version 1.0) [Open educational resource]. CC BY 4.0.

LICENSE AND RIGHTS

Copyright © 2026 Jose Mendoza. Except where otherwise noted, this work is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). You may share and adapt this material for any purpose, provided appropriate credit is given. Third-party trademarks, screenshots, figures, and other materials remain subject to their respective rights and licenses.

Google Colab is a product of Google LLC. “Python” and the Python logos are trademarks or registered trademarks of the Python Software Foundation. pandas, NumPy, Matplotlib, and scikit-learn are sponsored or affiliated projects of NumFOCUS, a 501(c)(3) nonprofit charity in the United States. statsmodels is a community-developed project distributed under the modified BSD license. ChatGPT is a product of OpenAI, Claude of Anthropic, Gemini and NotebookLM of Google LLC, and GitHub Copilot of GitHub, Inc. Product names are used for identification only and do not imply endorsement. StyleCraft Collective is a fictional company created for instruction.

COMPANION REPOSITORY

Datasets, notebooks, and figure sources for this chapter: https://github.com/jrmst102/businessanalytics

GENERATIVE AI USE

Generative artificial intelligence and other AI-assisted tools were used in the research, writing, revision, and production of this chapter, including literature discovery, source organization, outlining, preliminary drafts, prose revision, support for code and analytical examples, and document formatting. These tools were used under the author’s direction and are not credited as authors, researchers, or sources. The author determined the chapter’s scope, learning objectives, methods, interpretations, and recommendations, and reviewed and approved all AI-assisted material: factual claims and citations were checked against the underlying sources rather than accepted from AI-generated summaries, and code and analytical outputs were tested or otherwise reviewed for accuracy. Responsibility for the accuracy, originality, and final form of this chapter rests entirely with the author. A fuller statement appears in the front matter of the complete guide.

Chapter Learning Objectives

• By the end of this chapter, students should be able to:
  
• Distinguish predictive from explanatory modeling as goals with different obligations, citing the Section 7.10 distinction, and state what changes when a model’s outputs become the product.
• Specify a complete predictive frame — unit of prediction, target, feature set, and horizon — in writing before fitting, as the predictive extension of the analytic specification of Section 2.5, and declare its decision metric and baselines alongside it.
• Define a prediction label precisely, including the snapshot date, feature window, outcome window, and eligibility rule, and explain why the label is a measurement decision rather than a given.
• Define leakage, catalog its common routes in marketing data, and audit a feature list for information that crosses the snapshot line.
• Construct an honest train/test evaluation, explain in-sample flattery, state what a random customer holdout does and does not validate, and apply the split hygiene rules that protect it.
• Compute and interpret MAE and RMSE in dollars, compute top-decile spend capture as the decision metric of a ranked-list program, and choose the metric whose weighting matches the decision at stake.
• Operationalize baselines — mean, majority-class, and last-value — as stated opponents and express a model’s value as its margin over the strongest honest baseline.
• Diagnose overfitting and underfitting from the training-versus-validation error signature and explain the bias–variance tradeoff at an intuitive level.
• Run k-fold cross-validation on the training data to compare candidate models and select model complexity, and apply the comparison discipline of same folds, same metric, same baselines.

• Translate predictions into decisions: build ranked targeting lists under a fixed budget, apply expected-value framing at concept level, distinguish predicted spend from spend caused by a treatment, and state the deployment realities of data drift, concept drift, performance decay, and retraining.
• Verify an AI-built or vendor-supplied predictive model with the five-point audit — frame, leakage, split hygiene, baseline, and error in decision units — treating “too good” as a red flag.

Chapter 7 ended with a deliverable and a warning. The deliverable was the drivers analysis: spend an average order value modeled on the feature table, segment dummies read against the Urban Loyal Core, the designed interaction priced in dollars, and every coefficient guarded by a verb that stopped at the evidence. The warning came in the chapter’s last paragraph: the drivers model was built to explain, and the moment a model’s outputs — rather than its coefficients — become the product, the obligations change. This chapter is about that change. The question is no longer what moved spend in the window we observed; it is what spend will be in the window we have not yet seen, customer by customer, accurately enough to act on. Answering it requires machinery Chapter 7 never needed: a target defined in time rather than assumed, a dataset split so the model can be graded on customers it never met, baselines that convert Chapter 2’s interpretive discipline into a working opponent, error metrics denominated in dollars and in the decision’s own currency, and a new class of silent failure — leakage, the model that grades itself on information it should never have had — whose most reliable symptom is the one result every stakeholder wants to hear: a model that looks too good.

concept

What This Chapter Is Really About

Chapter 7 closed by separating two modeling goals: explanation, where the coefficients are the deliverable, and prediction, where the outputs are. The separation sounds administrative and is anything but, because the two goals fail differently. An explanatory model fails in public — a sign that contradicts known structure, a causal verb that overreaches — and Chapter 7 built the audit that catches it. A predictive model fails in private. It can be arithmetically flawless, statistically significant, beautifully fitted, and worthless, because everything that certified it happened on data it will never see again.

The predictive disciplines of this chapter — the frame declared before fitting, the wall in time between features and outcome, the test set touched once, the opponent named before the race — are all versions of a single idea: the model must be graded under the conditions of its actual job, predicting what it has not seen, against the alternative of not using it at all. And the chapter’s characteristic red flag inverts every instinct a student brings to it: in predictive work, the result that looks too good almost never is, and the analyst’s most valuable reflex is suspicion of their own success.

Source: Course concept developed for this guide, informed by Shmueli (2010) and Kaufman et al. (2012).

8.1 Marketing Decision Context: The Scored List and the Vendor’s Promise

Chapter 7’s drivers deliverable survived the budget-lock meeting: the install campaign was funded as a test rather than a certainty, the discount ladder survived on the within-segment exhibit, and the fall retention program’s incremental budget was allocated across the four journeys the treatment map defined. Now the program is being built, and its most expensive component has a capacity constraint. The top journey tier — internally, the Backstage Preview — pairs early access to the holiday drops with a printed lookbook and a small gifting touch. Merchandising and print work out to roughly twenty dollars per treated customer, and the budget covers eight hundred customers: the top decile of a base of roughly eight thousand. Eight hundred slots, eight thousand customers. Someone must decide who gets one, and the CRM manager has asked the obvious question in its modern form: can we predict, customer by customer, who will spend the most over the next six months — and use that prediction to fill the list?

Three answers are already in the building, and each conceals a problem this chapter exists to expose. The first is the incumbent rule, defended by the CRM team’s operations lead: “Rank customers by what they spent in the last six months and take the top eight hundred. Past spend is the best predictor of future spend. We do not need a model; we need a spreadsheet sorted in descending order.” The second arrives from outside, in a vendor deck. StyleCraft’s customer-data-platform vendor has been piloting a predictive-spend module, and its pitch slide is the kind that ends meetings: “Our AI model predicts each customer’s next-six-month spend with 96 percent accuracy (R² = 0.96).” The annual license costs more than the entire Backstage print budget. The third comes from inside the analytics team, and it is the most tempting because it is nearly right: “We already built a spend model in the drivers analysis. Its signs survived the four-point audit — score everyone with it and take the top eight hundred.”

Each answer has a real virtue and a disqualifying flaw, and stating the flaws is the fastest tour of this chapter’s territory. The incumbent rule is cheap, transparent, and — this chapter will insist — the legitimate baseline every model must beat before it earns a budget line. But Chapter 5’s certified findings already show why it misfires: the base is lumpy. Suburban Occasion customers concentrate a season’s spending into one or two large baskets, so a customer with zero recent spend may be three weeks from a four-hundred-dollar occasionwear order, and a customer who just placed one may be done for the year. A backward-looking sort systematically gifts the customers who’s spending just happened and misses the customers whose spending is about to. The vendor claim fails differently: not on its arithmetic, which no one at StyleCraft can inspect, but on its vocabulary and its plausibility. Chapter 7 established that individual human spending is noisy and that fit statistics near 1 on individual behavior are a symptom rather than an achievement (Section 7.7). The in-house answer fails most instructively of all. The drivers model was built, in Section 7.10’s terms, to explain: fitted on the full window, graded on the data it was fitted to, its variables chosen to make coefficients defensible rather than predictions accurate. Nothing about surviving the four-point audit certifies that its predictions hold on customers and months it has never seen.

One boundary belongs in the brief rather than in a footnote three sections later, because it constrains what the whole project can claim. A model of future spend can prioritize the customers expected to be most valuable. It cannot identify the customers whose spending will increase because they receive the Backstage treatment. Those are different quantities, and only the second is what a marketer means by the return on a gift. The program is therefore commissioned as a recognition and prioritization program — treat the customers we expect to matter most — with the incremental question named as unanswered and Chapter 11’s experimental machinery named as the instrument that would answer it.

So, the VP of Marketing has commissioned the analyst — again, you — with a deliverable and a deadline. The deliverable: a scored list of eight hundred customers, produced by a method chosen honestly among the three candidates or a better fourth, with the evidence for the choice stated in a form the CFO can interrogate. The deadline: the scoring method must be locked in thirty days, when the holiday-drop calendar goes to print. Sections 8.2 through 8.4 build the frame, the label, and the leakage discipline. Sections 8.5 through 8.9 build honest evaluation: splits, error metrics, baselines, overfitting, and the comparison rules. Section 8.10 carries the score to the decision and into deployment. Section 8.11 turns to the AI assistant, which will build any pipeline fluently — including, without complaint, a leaky one. The labs in Section 8.12 build the honest pipeline, compare candidates on a frozen leaderboard, manufacture the leaky one on purpose, and catch it. Sections 8.13 through 8.15 rehearse the meeting.

8.2 From Explanation to Prediction: The Predictive Frame

Section 7.10 drew the distinction this chapter now inhabits: explanation and prediction are different modeling goals, served differently by the same equation, with different success measures and different characteristic failures. One sentence of reminder suffices — an explanatory model is graded on whether its coefficients are honest structure; a predictive model is graded on whether its outputs are accurate for cases it has not seen — and the rest of this section develops what the second goal demands before any model is fitted at all. The demand is a specification, and it is the direct descendant of the analytic specification of Section 2.5: just as no analysis in this guide begins without a written statement of the decision, the alternatives, and the deliverable, no predictive model begins without a written statement of four commitments this guide calls the predictive frame.

definition

Predictive Modeling and the Predictive Frame

Predictive modeling is the construction of a model whose purpose is to output accurate estimates of an unknown or future quantity for individual cases, evaluated on cases not used to build it. Its specification — the predictive frame — consists of four declared commitments: the unit of prediction (what entity receives a prediction), the target (the quantity predicted, precisely defined), the features (the information the model may use, fixed as of a stated moment), and the horizon (how far beyond that moment the target lies).

Source: Adapted from Provost and Fawcett (2013) and Kuhn and Johnson (2013).

Each commitment is a decision with consequences, and each connects to machinery this guide has already built. The unit of prediction is the grain question of Section 3.3 pointed forward: StyleCraft’s scoring problem lives at the customer grain — one prediction per customer — but the same business could ask for predictions at the store grain (next quarter’s revenue per store), the campaign grain (expected response per send), or the day grain (Chapter 10’s subject). Declaring the unit settles what a row means in every table the pipeline touches, and undeclared units produce the same silent damage here that undeclared grain produced in Chapter 4’s joins. The target is the subject of Section 8.3, because it deserves its own section: “future spend” is not a column that exists but a measurement that must be constructed, and every construction choice changes what the model learns to do. The features are the model’s permitted information, and the discipline they require — nothing the model would not know at prediction time — is the subject of Section 8.4. The word itself was defined in Section 3.5 among the variable roles; what is new here is not the role but the restriction. The horizon is the forward distance of the claim: the Backstage decision needs six months, because the program will be judged on the fall-and-holiday arc, while a win-back trigger might need only thirty days. The horizon disciplines everything downstream — a model with a six-month horizon cannot be validated in two weeks, and Section 8.10 will show that the horizon also sets the clock on how often the model must be retrained.

Table 8.1 states the frame for the Backstage decision in full, and its final column carries the point of the exercise: every row is a choice that could have been made differently, and each alternative would produce a different model answering a different question. In other words, “predict customer spend” is not yet a modeling problem — it is a family of modeling problems, and the frame is the act of choosing one member and committing to it in writing. The commitment is what makes verification possible later: a model can only be audited against a frame that exists.

Table 8.1
The predictive frame for the Backstage scoring decision

One more inheritance completes the frame’s paperwork, and Table 8.1’s last three rows record it. Section 2.5’s specification discipline required that the deliverable be named before the analysis begins; the predictive version has a sharper edge, because the frame must also name, in advance, the metric the model will be graded on and the baselines it must beat. Declaring them before fitting is not ceremony. It is the predictive analog of predict-then-verify (Section 1.7): a model builder who chooses the success metric after seeing the results has the same conflict of interest as an analyst who writes the hypothesis after seeing the data, and Section 8.9’s comparison discipline exists because the temptation is real, routine, and mostly unconscious.

8.3 Defining the Target: Labels, Snapshots, and Windows

The frame’s second commitment deserves its own section because it is the one student most reliably treat as given when it is in fact constructed. The drivers analysis of Chapter 7 modeled monetary — total spend across the observed window — a column that existed in the feature table before any model was imagined. The scoring decision has no such column. “What this customer will spend in the next six months” exists nowhere in StyleCraft’s data today; for the model to learn it, the analyst must manufacture historical examples of it, and the manufacturing is a measurement act with all of Chapter 3’s obligations attached. The manufactured quantity is called the label: the recorded value of the target for a case whose outcome is already known, from which the model learns the relationship between features and future.

definition

Label (Target Variable)

A label is the constructed, recorded value of the target for a historical case — the answer the model trains on. Constructing labels requires choosing a snapshot date that splits time into a feature window (the past the model may know) and an outcome window (the future the label summarizes), together with eligibility rules for which cases receive labels at all. The label is a measurement decision: its windows, its filters, and its eligibility rules are choices that change what the model learns, and they are declared, not discovered.

Source: Adapted from Provost and Fawcett (2013) and Kuhn and Johnson (2013).

The construction is easiest to see drawn on StyleCraft’s own calendar. The certified transaction history runs twenty-four months, from July 1, 2024, through June 30, 2026 — the window every lab since Chapter 4 has used. To manufacture labels, the analyst plants a snapshot date inside that history: December 31, 2025. The feature window is the eighteen months from July 1, 2024, through December 31, 2025, and the customer feature table is rebuilt from it, exactly per Table 6.5’s derivation rules but with the analysis date moved back to the snapshot. The outcome window is the six months from January 1 through June 30, 2026, and each customer’s label, call it spend_next6m, is their total certified spend inside it: numerator, the sum of line_revenue on their transactions in the outcome window; denominator, none — it is a total, not a rate; window, January 1 through June 30, 2026; filters, none beyond the eligibility rule below. The four-element discipline of Section 3.5 applies to labels exactly as it applied to metrics, and writing the label in that form is not an homage — it is what makes two analysts’ pipelines produce the same target.

Notice that both windows have two ends, and that both ends are load-bearing. A feature window declared as eighteen months but implemented as “everything before the snapshot” silently includes whatever history the file happens to carry, so two analysts working from the same declaration produce different recency and different monetary values. An outcome window declared as six months but implemented as “everything after the snapshot” silently lengthens the horizon for any customer whose file extends further, so the label stops meaning what the frame says it means. Lab 8.1 therefore names four dates rather than one and asserts that the minimum and maximum transaction dates on each side fall inside their declared window. This is the four-element metric discipline turned into two assertions.

Three consequences of the construction deserve a paragraph each, because each becomes a lab step. First, the snapshot creates a wall in time, and the wall is what the whole chapter protects. On the feature side, the model may know everything StyleCraft knew on December 31, 2025: eighteen months of recency, frequency, spend, discount behavior, channel mix, and engagement. On the outcome side, the model may know nothing — not one transaction, not one field updated after the snapshot — because now the real scoring decision is made, the future is exactly that unavailable. Every feature must therefore be re-derived as of the snapshot: recency_days recomputed against December 31, tenure_days truncated at the wall, monetary summing only feature-window transactions. Reusing the June 2026 feature table, the one Chapters 6 and 7 worked with, would hand the model six months of the very future it is being asked to predict — the error Section 8.4 names.

Second, eligibility is part of the label, and this chapter states its rule in the form the code can enforce. A customer receives a labeled row when two conditions hold: their signup_date falls on or before the snapshot, and they placed at least one order inside the feature window. The first condition is obvious — a customer acquired in March 2026 has no feature-window history and would enter the training data as a row of zeros wearing a label they never had a chance to earn. The second is the one drafts forget, and it is not a convenience: recency, average order value, discount share, and store share are all undefined for a customer with no feature-window purchases, because each of them divides by a quantity that is zero. This is the same population rule Chapter 6 declared for the clustering base (Section 6.6), inherited here deliberately rather than reinvented, and it carries the same obligation: the excluded customers are counted and named, not silently dropped.

Third, a label of zero is a label, not a gap. Customers eligible at the snapshot who bought nothing in the outcome window have spend_next6m equal to zero — and they are the single most important group in the training data, because they are the customers the incumbent sort-by-past-spend rule cannot distinguish from the lumpy occasion shoppers about to place a large order. Dropping zero-label rows, a mistake AI assistants make with some regularity when asked to “clean” a training table, deletes precisely the contrast the model exists to learn. The missing-data disciplines of Chapter 4 apply, but the prior question is definitional: here, absence of transactions is the outcome, not a defect in recording it.

8.4 Leakage: The Canonical Error

Chapter 7 closed its multicollinearity section with a designed trap: regress monetary on aov and frequency, watch R² leap toward 1, and recognize that the model had been handed the answer — monetary is, by Table 6.5’s own derivation rules, their product. Chapter 7 promised that Chapter 8 would name the general disease. Here is the name. Leakage is the use, in training or evaluating a predictive model, of information that would not be available now the model makes its real predictions — information from the future, from the target’s own construction, or from the evaluation data itself. It is the canonical error of predictive modeling: the most common serious failure, the hardest to see in a finished pipeline, and the one whose symptom is a model that performs implausibly well right up until it is deployed, at which point the borrowed information is no longer there to borrow and performance collapses to what the legitimate features always supported (Kaufman et al., 2012).

definition

Leakage

Leakage is the presence, in a model’s training features or evaluation procedure, of information that will not legitimately be available when the model is used for its real predictions — most often information from after the prediction moment, information arithmetically derived from the target, or information shared between training and test data. A leaky model’s measured performance reflects the leaked information rather than genuine predictive relationships, and its deployed performance falls to the level the legitimate features support.

Source: Adapted from Kaufman et al. (2012).

In other words: a leaky model is not a bad model that can be tuned into a good one. It is an honest measurement of the wrong thing — of how well the future predicts itself — and no number of refitting repairs it, because the defect is in the data the model was allowed to see. That is why leakage is caught by audit rather than by statistics: no diagnostic printed by the fitting software distinguishes leaked performance from earned performance, and the only reliable detector is an analyst tracing where each feature came from and when it became knowable. The trace has a natural checklist, because leakage in marketing data arrives by a small number of well-worn routes. Table 8.2 catalogs the five this guide’s students will meet, each with its StyleCraft instance and its repair; the first two account for most real cases.

Table 8.2
Common leakage routes in marketing data

Three of the table’s rows repay a closer look, because their surface innocence is the lesson. Outcome-window contamination is temporal leakage’s subtle cousin: no one typed “use the future” — someone reused a convenient existing table whose columns were computed at the wrong date, and every derived value silently absorbed six months of the answer. The tell is often a range check, which is why the audit in Section 8.11 includes one: at a December 31, 2025, snapshot, no eligible customer can have a recency of fewer than 181 days when recency is computed correctly, so a table full of smaller values has been computed at a later date, and a single impossible value convicts the whole column.

Split contamination is smaller in effect than the first two rows and larger in principle, and it needs its boundary drawn precisely, because the boundary is where students most often over-apply the rule. What contaminates a split is a step that learns something from the data and is fitted before the split: a standardizer that estimates means and spreads, an imputer that estimates fill values, a feature selector that ranks columns by their relationship to the target, a clustering model that estimates centroids, a target encoder that averages the label within categories. Each of those carries a whisper of the test customers’ distribution into training, and the whispers compound across steps. A transformation that learns nothing — taking a logarithm, multiplying two columns, expanding a predictor into fixed polynomial powers — does not contaminate anything, because applying it before or after the split produces identical values for every row. The rule to memorize is therefore about estimation rather than about ordering: anything estimated from data — a mean, a scale, an imputation value, a centroid, a category encoding — is estimated from the training data only, which in practice means it lives inside a pipeline that is refitted in every fold (scikit-learn developers, 2026a).

Population leakage is the one that hides inside a sentence rather than inside a column. A filter is a modeling decision, per Section 3.5, and a filter that consults the outcome window chooses the training population using the answer. “Customers who were active during the study” sounds like housekeeping and is a prediction: it quietly removes the customers who went dark, which is the outcome the model is supposed to be able to anticipate. The repair is mechanical — every eligibility condition must be evaluable using only information available on or before the snapshot — and this chapter’s eligibility rule was written in Section 8.3 to satisfy exactly that test.

What makes leakage the chapter’s central discipline rather than one hazard among several is its relationship to the chapter’s red flag. Every route in Table 8.2 improves measured performance; none improves real performance; therefore, unexplained excellence is evidence of a leak. The logic deserves a box, because it inverts the instinct every student brings to model building — and because it is the exact instrument the vendor claim of Section 8.1 requires.

8.5 The Out-of-Sample Standard: Train/Test Splits and In-Sample Flattery

Leakage corrupts what the model knows; this section addresses how the model is graded, and the two disciplines are halves of one standard. A predictive model’s job is to be accurate on cases it has not seen. Its performance on the cases it was fitted to — in-sample performance — is therefore not merely a weaker version of the evidence; it is systematically misleading evidence, and misleading in a known direction. Least squares, by construction, chooses the coefficients that minimize error on the fitting data (Section 7.6). The fitted model is thus tailored to that data’s every quirk: its whales, its noise, its accidental patterns. Grade it on the same data and the tailoring is graded as skill. This guide calls the phenomenon in-sample flattery, and its size is not constant — it grows with model flexibility, shrinks with sample size, and vanishes only in the one place it cannot be observed: the future. Every impressive statistic computed on training data should be read with this discount in mind, and the discount cannot be computed from the training data itself.

The repair is structural, not statistical: hold data back. Before any fitting, the labeled customers are split at random into a training set, which the model may learn from without restriction, and a test set — also called a holdout — which is sealed until evaluation and consulted exactly once, for the finalist. The split is performed at the unit of prediction — whole customers go to one side or the other, per the frame’s grain declaration — with a fixed random seed so the split is reproducible, and with proportions this guide sets at 80/20: enough training data to fit stable models on eight thousand customers, enough test data to grade them with useful precision.

definition

Train/Test Split and the Test Set

A train/test split randomly partitions labeled cases, at the unit of prediction, into a training set used to fit and select models and a test set held out to evaluate the selected model. Test-set performance estimates out-of-sample performance — how the model will do on cases it has not seen — which is the only performance that matters for a predictive model’s real job. The estimate is honest only if the test set contributes nothing to fitting, preprocessing, or model selection, and is consulted once for the final model rather than repeatedly during development.

Source: Adapted from James et al. (2021) and scikit-learn developers (2026b).

The definition’s last clause carries the discipline most often violated in practice, so it earns its own paragraph. A test set consulted repeatedly stops being a test set. If the analyst fits a model, checks test error, adjusts, checks again, and repeats, the adjustments are being steered by the test data — a slow-motion version of training on it, and a cousin of Table 8.2’s split contamination. By the tenth consultation, the “held-out” error is flattery with extra steps. The working protocol, which the labs of Section 8.12 follow to the letter: development decisions — which features, which model family, how much flexibility — are made using the training data alone, with Section 8.9’s cross-validation supplying out-of-sample estimates during development; the test set is opened at the end, once, to grade the finalist; and if development resumes after that, the honest analyst treats the old test error as spent. On StyleCraft’s eight thousand customers this protocol costs little and buys the deliverable its central sentence: “on sixteen hundred customers the model never saw, the average error was X dollars” — a sentence the CFO can trust precisely because of everything the protocol forbade.

One qualification completes the section, and it is more important than it looks, because the chapter later teaches drift. It is tempting to say that the test customers “stand in for the future.” They do not, quiet. A random customer holdout estimates cross-sectional generalization at the historical snapshot: how the model performs on other customers drawn from the same population, at the same moment in the business’s life, under the same acquisition mix, pricing, and merchandising. It does not measure time-forward transport — whether a relationship estimated at a December 2025 snapshot still holds at a June 2026 scoring date after the world has moved. Measuring that requires a different design: fit on an earlier snapshot, evaluate on a later one, or, once deployed, monitor realized error as each fresh outcome window matures. Standard random folds assume a stable sampling distribution, and where the intended generalization is explicitly into later periods, a time-ordered evaluation is the honest instrument (scikit-learn developers, 2026b).

The random split is nonetheless the right design here, and the reason is worth stating so the qualification is not mistaken for a defect. The frame already handled time: the wall of Section 8.3 lives inside the label construction — every customer’s features stop at the snapshot, and every label starts after it — so shuffling customers cannot shuffle the future into the past. What the shuffle cannot do is manufacture a second point in time, which is why the deliverable’s performance sentence names its population (“customers like these, at this snapshot”) and the monitoring plan of Section 8.10 exists to cover the rest. When the unit of prediction is itself a time, as in Chapter 10’s daily forecasting, random splitting breaks even the first guarantee, and a time-aware holdout is mandatory rather than supplementary.

8.6 Measuring Error in Dollars and in Decisions: MAE, RMSE, and Capture

An honest test set answers where to measure; this section answers what to measure, and the answer has two parts, because the Backstage program asks two different questions of the same predictions. The first question is calibration: how close is a customer’s predicted dollar amount to what they spend? The second is the decision: which eight hundred customers appear above the cutoff? Those are not the same question, a model can improve on one while worsening on the other, and a frame that declares only the first has quietly agreed to be graded on something other than its job.

Take calibration first, because it supplies the vocabulary. The raw material is the per-customer error — actual label minus predicted value, the out-of-sample sibling of Section 7.6’s residual — and the reporting need is a single number that summarizes sixteen hundred such errors in units a decision can use. Two summaries dominate numeric prediction, both built by this guide’s oldest recipe (aggregate the individual deviations, per Chapter 5), and their difference is a policy choice about big misses. The mean absolute error takes each customer’s error, strips its sign, and averages. Its virtues are bluntness and units: an MAE of 54 means the model’s predictions miss actual six-month spend by 54 dollars on average, a sentence any stakeholder can absorb, denominated in the same dollars as the budget. The root means squared error squares each error before averaging, then takes the square root to return to dollars. The squaring is the policy: it weights a single 300-dollar miss like nine separate 100-dollar misses, so RMSE is dragged upward by the largest errors far more than MAE is. Neither metric is more correct; they encode different sensitivities, and the gap between them is itself diagnostic — RMSE materially above MAE on the same predictions announces that the error is concentrated in a few large misses rather than spread evenly (Willmott & Matsuura, 2005), which on StyleCraft’s data means one thing before anyone plots it: the whales and the lumpy occasion baskets of Chapter 5’s concentration analysis are where the model hurts.

definition

MAE and RMSE

Mean absolute error (MAE) is the average of the absolute differences between actual and predicted values: the typical miss, in the target’s native units, with every error weighted equally. Root mean squared error (RMSE) is the square root of the average squared difference: also in native units, but disproportionately sensitive to large errors, so that it exceeds MAE whenever misses are uneven and exceeds it greatly when error is concentrated in a few cases. Both are computed out of sample; the choice between them is a statement about which misses the decision cares about most.

Source: Adapted from Willmott and Matsuura (2005) and James et al. (2021).

Now the decision question, and the correction it forces on a tempting argument. It is easy to justify leading with MAE by saying that a wasted twenty-dollar slot costs the same whether the prediction behind it was moderately or catastrophically wrong, so the program’s economics do not square the pain. The premise is true and the inference does not follow, because the cost of the decision is not a function of the point-prediction error at all. What costs money is whether an error moved a customer across the selection threshold, who was displaced from the list when it did, and how far the ranking shifted. A catastrophic prediction error four hundred ranks below the cutoff costs exactly nothing; a small error at rank 800 changes who receives a lookbook. Error metrics summarize distance; the program acts on order.

The metric that grades order under a fixed capacity is capture. Score every customer, sort descending, take as many customers as the budget funds, and compute what share of the population’s actual outcome-window spends those customers turned out to account for. A model whose funded customers capture 34 percent of next-window spend is ordering customers better than one whose selection captures 28 percent, and the difference converts directly into dollars by multiplying the percentage-point gap by total outcome spend. Because the incumbent rule produces its own ranked list, the two can be graded head-to-head on exactly the artifact the meeting is deciding between, which makes the incremental capture over the incumbent the single most persuasive number the project produces.

One quantity inside that paragraph must be settled rather than assumed, because it decides what the metric means: how many customers is “as many as the budget funds,” as a share of the population being ranked? The Backstage budget funds eight hundred treatments. That is a capacity, not a proportion — it does not grow if the eligible population turns out to be larger, and it does not shrink if the eligibility rule of Section 8.3 excludes more customers than anyone expected. The selection share is therefore computed rather than typed: eight hundred divided by the population the model can honestly score. On StyleCraft it lands near a tenth, which is why this guide is comfortable calling the exhibit top-decile capture in conversation and why Chapter 9 generalizes it to deciles. The labs still compute it, because a proportion typed into a metric is a proportion nobody re-derives when the population changes.

The frame of Table 8.1 therefore declares three metrics with three different jobs, and the labs report all three for every candidate. Capture at the capacity share is the decision metric, because the program acts through a fixed ranked cutoff. MAE is the lead calibration metric, because it gives an interpretable typical miss in the currency of the budget and keeps the deliverable’s language honest about how much the model does not know. RMSE is reported beside MAE as the concentration alarm. A different decision could reorder them: if predictions fed inventory commitments, where one enormous miss costs more than many small ones combined, RMSE’s sensitivity is the point, and if every customer received a personalized offer amount, calibration would become the decision metric rather than its companion. The general rule follows the asymmetric-cost discipline of Section 2.7 — choose the metric whose weighting matches the decision’s actual cost structure — and note, for honesty, that percentage-based error metrics are deliberately absent here; they have a characteristic failure on near-zero actuals, and this guide takes them up where they natively live, in Chapter 10’s forecasting.

8.7 Baselines, Operationalized

Section 2.7 introduced baselines as interpretive discipline: no number means anything until you know what it is being compared against. Section 7.7 revealed R² as secretly a baseline comparison — the model against guessing the mean. This section completes the thread’s promised operationalization: in predictive modeling, the baseline stops being a lens and becomes an opponent — a fully specified, deliberately simple prediction rule, evaluated on the same folds and the same test customers with the same metrics as the model, whose defeat is the model’s entry fee. The reframing sounds small and changes everything about how results are reported: a model with a test MAE of 54 dollars is not good or bad in itself; it is good if the best naive rule scores 80 and embarrassing if the best naive rule scores 55.

definition

Baseline Model

A baseline model is a deliberately simple prediction rule — requiring no fitting beyond, at most, one summary statistic — evaluated under exactly the conditions of the candidate model: same folds, same test set, same metrics. Standard baselines include the mean baseline (predict the training mean for everyone; numeric targets), the majority-class baseline (predict the most common class for everyone; categorical targets), and the last-value baseline (predict that the last observed value, or window, repeats). A predictive model’s demonstrated value is its margin over the strongest applicable baseline, not its raw score.

Source: Adapted from Provost and Fawcett (2013).

Table 8.3 gives the three standard baselines their StyleCraft form. The mean baseline is the floor: predicting the training-set average of spend_next6m for every customer uses no customer information at all, and a model that cannot beat it has learned nothing — it is R²'s comparison (Section 7.7), now made to stand in the open and fight on held-out customers. It is also the baseline that needs one qualification on the decision metric, and the qualification is more instructive than it looks. A rule that gives every customer the same score produces no ranking at all, so its capture is not a property of the rule; it is a property of whatever order the rows happened to arrive in and reported carelessly that is row order masquerading as performance. Reported honestly — by breaking the ties on a declared random draw, which is what the lab does — it becomes the quantity everyone wanted: the capture a program would get by selecting customers at random, which is simply the selection share itself. That is the floor every candidate must clear, and it is a fact about random ordering rather than about the mean. The majority-class baseline is the floor’s categorical sibling, recorded here at concept level because this chapter’s target is numeric; Chapter 9 will show it doing serious work, where “predict that nobody churns” turns out to embarrass many a classifier. The last-value baseline is the one with teeth in this chapter, because it is not a strawman — it is the incumbent. “Predict that the next six months look like the last six” is exactly the CRM operations lead’s sorted spreadsheet, expressed as a prediction rule, and it is often strong: trailing spend genuinely carries signal, which is why the rule survives in so many businesses. The lab will discover its designed weakness — the miniature of Lab 8.1 shows it losing to the mean baseline on lumpy customers — but the weakness must be demonstrated, not asserted, and on the same field the model plays on.

Table 8.3
The three standard baselines, in StyleCraft form

The section closes with the thread’s forward arc stated once, in this chapter’s terms: baselines began as interpretation (Section 2.7), operationalized here as opponents, extend in Chapter 10 to time — where naive and seasonal-naive rules inherit the last-value role — and take visual form in Chapter 12 as reference lines. One idea, four chapters, each adding machinery and none re-teaching it.

8.8 Overfitting, Underfitting, and the Bias–Variance Intuition

The holdout discipline of Section 8.5 exists to measure a gap; this section explains where the gap comes from and what its size means. Fit a sequence of models of increasing flexibility to the same training data — a straight line, then a curve, then a wigglier curve, then one flexible enough to pass near every point — and watch two error curves. Training error falls, or at least does not rise, with every step: more flexibility can only fit the fitting data better. Held-out error typically falls, then turns, then climbs. The turn is the most important shape in predictive modeling, and its two sides have names. A model on the left side, too simple to capture real structure — the mean baseline is the extreme case — is underfitting it misses systematically, in the same direction for whole regions of customers, and both its training and its held-out errors are high. A model on the right side has begun learning its training data’s noise: the accidental quirks, the specific whales, the coincidences that will not recur. It is overfitting, and its signature is the diverging pair — training error excellent and still improving, held-out error worsening — because every quirk memorized is a point of training flattery and a liability on customers whose quirks are different (James et al., 2021).

One precision about that first curve, because the lab will print it and the claim is easy to overstate. What is guaranteed, when nested polynomial terms are added to a least squares model, is that the training sum of squared errors cannot increase — least squares minimize squared error, so a richer model can always reproduce the poorer one’s fit and usually does better. Training RMSE therefore cannot rise with degree. Training MAE is a different summary of the same residuals, and nothing in the fitting criterion protects it: a richer model can reduce squared error while nudging the typical absolute miss slightly upward. The lab tracks training RMSE for the monotone claim and reports training MAE beside it as the quantity that generally improves without being required to.

definition

Overfitting and Underfitting

A model overfits when it captures patterns specific to its training data — noise, outliers, coincidence — that do not generalize, so that its training performance substantially exceeds its held-out performance and additional flexibility widens the gap. A model underfits when it is too simple to capture structure that does generalize, so that both training and held-out performance are poor. The two are diagnosed jointly from the training-versus-held-out error signature, not from either number alone.

Source: Adapted from James et al. (2021) and Domingos (2012).

The intuition underneath the U-shape is the bias–variance tradeoff, which this guide states at intuition level and uses forever after. A too-simple model is wrong on average — its errors have bias, systematic and shared across any sample you might have drawn. A too-flexible model is wrong differently every time — refit it on a different sample of customers and it memorizes different noise, so its predictions have variance, swinging with the accidents of the draw. Total error carries both, flexibility trades one for the other, and the honest minimum lies between the extremes — which is why “more sophisticated” is not a direction of improvement in predictive modeling, and why Domingos (2012) lists the belief that it is among the field’s canonical beginner errors. Two of this guide’s running facts sharpen the intuition. Sample size moves the turn: with more training customers, noise averages out more, and flexibility becomes safer — eight thousand customers tolerate a richer model than eight hundred. And irreducible uncertainty sets a limit: StyleCraft’s spend labels contain genuine randomness — the wedding that did or did not happen in the outcome window — that no feature measured at the snapshot can carry.

That limit deserves a careful sentence, because it is easy to turn into a number the analysis has not earned. This guide does not estimate an irreducible error and does not claim to know where the limit sits. What it claims is weaker and more usable: individual human purchasing is volatile, spend is concentrated in a small number of customers, repeated windows of the same customer’s history disagree with each other substantially, and honest baselines on this data land in a known range — so a result far stronger than all of that context would suggest is implausible on its face and belongs in the audit queue rather than in the deck. Implausible relative to domain variability, outcome concentration, repeated-history stability, and the honest baselines: that is the standard, and it is a judgment supported by evidence rather than a threshold computed from a formula.

Table 8.4 collects the diagnostic signatures, because the pair of error numbers is the instrument the lab will read. The table’s repair menu sits at the level this chapter needs: against overfitting — simplify the model, reduce the feature list to the frame’s justified variables, or get more data; against underfitting — add real structure, which in this guide’s practice usually means a feature the frame should have included, not a fancier algorithm. The menu’s modesty is deliberate. The regularization machinery that automates the flexibility dial belongs to later coursework; what transfers to every future tool is the signature-reading skill. Note the column headings: during development the second number is a cross-validated estimate computed inside the training data (Section 8.9), not a test-set figure, because reading this table off the test set is how the test set gets spent.

Table 8.4
Reading the training-versus-validation error signature

8.9 Cross-Validation and the Discipline of Model Comparison

Section 8.5’s protocol left a gap this section fills: development needs out-of-sample estimates too. Choosing between candidate models — the explanatory specification or a prediction-oriented one, the shorter or the longer feature list, a straight line or a curve — requires comparing their out-of-sample errors, but the test set is sealed until the end and consulting it per candidate would spend it. The standard answer is cross-validation: rotate the holdout inside the training data. Partition the training customers into k equal folds — five is this guide’s default; split at the unit of prediction, as always. Then, k times over, fit the candidate on all folds but one and evaluate it on the one left out, so every training customer serves once as validation and never validates a model that saw them. The k scores average into a single cross-validated estimate per candidate, with a bonus the single split cannot give: the spread across folds, which reads like every spread since Chapter 5 — a candidate whose error swings widely from fold to fold is telling you its performance depends on which whales it drew, the variance half of Section 8.8 made visible.

definition

Cross-Validation

Cross-validation estimates a model’s out-of-sample performance using training data alone, by repeatedly refitting the model with one of k folds held out and averaging the k held-out scores. It is the standard instrument for development-time decisions — feature choices, model family, settings, and the amount of flexibility — allowing the true test set to stay sealed for the final grade. The averaged estimate remains honest only if every learned preprocessing step is refit inside each fold, and dishonest the moment any fold’s validation data influences its own fitting.

Source: Adapted from James et al. (2021) and scikit-learn developers (2026b).

The definition’s last two clauses are the reason this chapter’s labs put every model inside a pipeline rather than fitting steps by hand. A pipeline is an object that chains learned steps — a standardizer, an imputer, a clustering model, the regression itself — and refits all of them together whenever it is fitted. Handed to a cross-validation routine, it is refitted inside each fold automatically, so the standardizer of fold three never sees fold three’s validation customers and the segmentation of fold three never learned its centroids from them. Doing the same work by hand is possible and reliably forgotten under deadline; scikit-learn’s own guidance is to use pipelines precisely so that this discipline is structural rather than remembered (scikit-learn developers, 2026a). The consequence for Chapter 6’s segmentation is direct and worth stating before the lab reaches it: a segment label is a learned feature, not a raw column, so a segmentation refitted on all labeled customers before the split lets the test customers shape the scaler and the centroids that training then uses. Inside a pipeline, the same segmentation is honest.

What must be owned by the analyst is the comparison discipline cross-validation serves, because model comparison is where predictive projects most often go quietly wrong — not through any single dishonest act but through drift in the conditions of comparison. The discipline has three clauses, each closing a real loophole. Same split: every candidate is estimated on the same folds, generated once with a fixed seed, because reshuffling between candidates lets chance pick the winner — with a mediocre model and enough reshuffles, some split flatters it. Same metric: the metrics declared in the frame (Section 8.2) grade every candidate, with the decision metric leading; a candidate that wins on RMSE after losing on the declared capture and MAE has not won — the goalposts have moved and moving goalposts after results are visible is the model-comparison version of writing the hypothesis after the data. Same baseline: the baselines of Section 8.7 appear in every comparison table, all the way to the final one, because the question is never only “which model is best” but “does the best model beat the spreadsheet by enough to matter.”

The three clauses have a physical form in this chapter’s labs: one leaderboard, built once, with a row per candidate and a column per declared metric, the two baselines occupying the first two rows, and every figure computed on the same five folds of the same training customers. Because the folds are shared, the leaderboard supports a comparison a table of separate averages cannot: the paired per-fold difference between a candidate and the incumbent. Pairing matters because folds differ in difficulty — a fold holding three whales is hard for everyone — so each candidate’s own standard deviation across folds is inflated by difficulty the comparison should cancel. The honest question is not whether a candidate’s mean beats the incumbent’s mean by more than its own spread; it is whether the candidate beat the incumbent fold by fold. Only when the leaderboard is complete, and a selection rule written before the race has been applied to it, does the analyst name a finalist, refit it on all the training customers, and open the sealed test set — once. Everything before that moment is development; everything after it is reporting.

8.10 From Score to Decision: Expected Value and Deployment Realities

A graded model is still not a decision. The model’s output is a score — a predicted spend_next6m per customer — and the Backstage question is an action: which eight hundred customers get the treatment. This section builds the translation and then follows the model past the decision into the part of its life that analytics courses traditionally ignore and analytics careers cannot: deployment.

The translation’s first form is the one the fixed budget dictates: rank and cut. Score every eligible customer, sort descending, take the top eight hundred. Under a hard capacity constraint the model’s job is ordering quality, not clairvoyance — it does not matter whether the top customer’s prediction is 700 or 900 dollars if the ranking puts the right customers above the line — and the deliverable therefore leads with the capture exhibit Section 8.6 declared: what share of actual outcome-window spend the model’s funded customers capture, against the share the incumbent’s funded customers capture at the same capacity, with the count of customers appearing on both lists. That head-to-head, on sealed test customers, is the single most persuasive table the project produces, because it grades the exact artifact the meeting is deciding about. One operational detail belongs in the deliverable rather than in the code comments: at the cutoff, ties must break the same way every run, or the eight hundredth name changes between the analysis and the campaign build. The lab declares a secondary sort of key for exactly this reason.

The translation’s second form asks the question the budget’s size begs: is eight hundred even the right number? Answering it requires attaching money to predictions, and the instrument is expected-value framing: for any candidate action on any customer, expected value is each possible outcome’s value weighted by its probability, minus the action’s cost, and the action is worth taking where the expectation is positive (Provost & Fawcett, 2013). At concept level — which is where this guide holds it, one notch above arithmetic and well short of decision theory — the framing does two jobs for the Backstage program. It converts the score threshold from an arbitrary rank into an economic line: treating a customer is justified where the treatment’s expected incremental value exceeds twenty dollars.

The word incremental in that paragraph is doing load-bearing work, and Section 8.1 flagged it in the brief so that it could be enforced here. This chapter’s model predicts what a customer will spend. It does not predict what a customer will spend because of the gift, and no amount of out-of-sample accuracy converts the first quantity into the second. A perfectly calibrated spend model can rank the base beautifully and still send lookbooks to eight hundred customers who would have spent identically without one. That is not a flaw in the model; it is the boundary of what a predictive frame can establish, the same boundary Section 7.13’s verb discipline enforced for coefficients, and only Chapter 11’s instrument can cross it. The honest deliverable therefore prices the program two ways: the capture gap, which the analysis certifies, and the incremental return, which it explicitly does not — with a holdout design named as the purchase that would.

Performance decay is the observable consequence of either — measured predictive quality eroding as one or both accumulate, sometimes gradually, sometimes at a policy change’s step edge. The disciplines that answer them follow directly. Monitoring means tracking the deployed model’s realized error on each period’s fresh labels as they mature, against the benchmark the test set established, exactly as the test set once was — and, cheaply and separately, watching the input distributions, because an input alarm arrives months before an outcome alarm can. Retraining runs on a declared cadence tied to the horizon: a six-month-horizon model can only be fully graded six months after each scoring run, so the monitoring calendar is part of the frame, written before deployment, not improvised after the first bad quarter. And explicit triggers — a performance floor, a known structural change such as a repricing or a merchandising overhaul — force revalidation off-cadence, because concept drift’s most expensive property is that it announces itself in the business before it announces itself in the metrics. Sculley et al. (2015), in the paper that named the field’s hidden costs, put the engineering point this guide converts to an analyst’s point: the model is the smallest part of the system, and the system — data feeds, definitions, monitoring, retraining, ownership — is what actually predicts. The analyst of record’s obligations does not end at the test set. They end when the model is retired.

definition

Data Drift, Concept Drift, and Performance Decay

Data drift is post-deployment change in the distribution or quality of a model’s input features, so that scored cases increasingly differ from training cases; covariate shift is its special case, in which the predictor distribution changes while the conditional relationship to the outcome remains stable. Concept drift is change over time in the relationship between the inputs and the target, so that the learned mapping no longer describes how the outcome is generated, and it cannot be diagnosed from the inputs alone. Performance decay is the resulting erosion of predictive quality, observed by monitoring realized error as fresh outcomes mature. All three are managed rather than prevented: by monitoring inputs and outcomes against the training-window benchmark, retraining on a cadence tied to the horizon, and declaring triggers that force revalidation when a structural change is known.

Source: Adapted from Gama et al. (2014) and Sculley et al. (2015).

8.11 AI as a Predictive-Modeling Assistant

Chapter 7’s AI section ended with a division of labor: the assistant proposes relationships; the analyst tests whether they are honestly stated. Predictive work sharpens the division again, because the assistant’s fluency now extends end to end. Prompted with a feature table and “build me a model that predicts customer spend,” a current assistant will construct the label, engineer features, split the data, fit several model families, tune them, and report a polished comparison table — an entire pipeline, delivered in one response, runnable as pasted. The productivity is real and this guide uses it. The hazard is equally real and has a precise shape: every failure mode this chapter has named is a failure the assistant commits fluently, silently, and with congratulations. A leaky pipeline does not look different from an honest one in the chat window. It looks better.

The characteristic failure modes, named so the audit can target them. The convenient-table leak: asked to predict future spend, the assistant builds features from whatever table it was given — including columns computed across the outcome window — because nothing in the prompt drew the snapshot wall; temporal leakage by default, Table 8.2’s first row, and the single most common defect in AI-drafted pipelines. The celebration of the impossible: the leaky pipeline reports R² of 0.97, and the assistant’s narration calls it excellent — assistants inherit their training data’s enthusiasm for high fit statistics, not this chapter’s suspicion of them, so the too-good reflex is precisely the judgment that does not transfer. The missing opponent: the report quotes MAE, RMSE, and R² with no baseline anywhere, leaving no way to know whether the model beats the sorted spreadsheet it exists to replace. The metric mismatch: distance metrics reported for a ranked-list decision, with no capture figure and no head-to-head against the incumbent’s list, because nothing in the prompt named the act. The vocabulary shuffle: “96 percent accuracy” applied to a numeric target — accuracy is classification vocabulary (Chapter 9’s, where its own failures await), and its appearance in a numeric-prediction claim is a tell that the claimant is quoting whatever number sounded best. The hygiene slips: preprocessing fitted on pooled data, the test set consulted in a tuning loop, complexity chosen by watching test error, the comparison metric switching mid-report — each individually small, each a clause of Sections 8.5 and 8.9 violated without announcement. And the deletion helpfulness: zero-spend labels “cleaned” away as missing data, quietly deleting the customers the model most needs to learn (Section 8.3).

Where the assistant genuinely helps, use it deliberately. Pipeline mechanics from a written frame — the split with a stated seed, the fold generator, the metric functions, the baseline estimators, the leaderboard table — are exactly the well-specified boilerplate assistants draft reliably. Feature derivation code is safe when, and only when, the prompt supplies the derivation rules and the window boundaries explicitly; the lab’s prompts do. Audit exhibits — the feature-window trace, the range checks of Section 8.4, the signature table of Section 8.8 — are tedious, mechanical, and ideal for delegation. Error analysis is a strength: asked “which customers does this model miss worst, and what do they share?”, an assistant surveys the residuals faster than any manual pass, and the answer — on StyleCraft’s data, the lumpy occasion shoppers — feeds the next framing conversation. And the assistant remains a good rehearsal partner for the vendor meeting.

The audit that governs all of it extends Chapter 7’s four points to the predictive setting. Five points, in order, run on every model — the assistant’s, the vendor’s, and yours — before any result is believed or repeated. Table 8.5 states the checklist; the box operationalizes it as the two-prompt pattern this guide has used since Chapter 6, with the audit standing between mechanics and narration.

Table 8.5
The five-point audit of a predictive model

ai in practice

The Pipeline That Must Survive Its Audit

A two-prompt pattern with the audit between, extending Section 7.11’s. Prompt one, the frame as specification: paste the seven declarations of Table 8.1, the label definition with its four dates and eligibility rule, the feature list with derivation rules and the instruction that every feature be computed from transactions inside the declared feature window, the declared metrics and baselines, and the split and cross-validation protocol with its seed — then ask for mechanics only: “Build this pipeline exactly as specified; select among the candidates using five-fold cross-validation on the training customers only; report one leaderboard with capture, MAE, and RMSE for every candidate and both baselines; do not touch the test set; change nothing in the specification; narrate nothing yet.”

Then run Table 8.5 yourself, in order: confirm the frame came back unedited; trace three features to their windows and run the recency range check; confirm every learned step sits inside a pipeline and the test set is untouched; confirm both baselines are present on every metric and the margin is computed; read the signature against Table 8.4, and if anything looks too good, stop and trace. Prompt two, only after the pipeline survives and the finalist has been graded once on the test set: “Draft the comparison paragraph for a marketing VP: the model’s capture at the program’s capacity against the incumbent list, its test MAE against both baselines in dollars, and no claim that any predicted difference is caused by anything.” Audit the draft’s verbs against Section 7.2’s registry — prediction language earns “expected” and “predicted,” never “will drive.” Predictions first, per Section 1.7; the analyst of record signs the scores.

Source: Course concept developed for this guide, informed by scikit-learn developers (2026a) and Kaufman et al. (2012).

8.12 Marketing Interpretation and Managerial Insight

The lab’s outputs are leaderboards and ranked lists; the thirty-day deadline runs on sentences. This section translates — and, per this guide’s standing practice, it does so partly by exhibiting wrong managerial readings and correcting them, because the predictive setting mints new misreadings faster than the associational one did, and three of them will be voiced in the scoring meeting by people whose only error is applying last chapter’s intuitions to this chapter’s numbers.

The first wrong reading will come from whoever liked the vendor deck: “Our in-house model’s average miss is fifty-odd dollars. The vendor’s model is at 96 percent. Ours is a C-minus student; theirs is an A. Buy the license.” The comparison is meaningless before it is even unfair, and the first problem is vocabulary rather than evidence. “Ninety-six percent accuracy” is undefined for a numeric prediction. Accuracy is a classification statistic — the share of cases assigned to the correct class — and future spend in dollars has no classes to be correct about. If the vendor means a test R² of 0.96, they should say so and disclose the evaluation design, because R² is a proportion of variance accounted for and not a percentage of predictions that were right; the two sentences sound alike and describe different quantities. It is worth knowing, and worth saying in the meeting, that R² computed on data a model was not fitted to is not even bounded below by zero: a model that predicts held-out cases worse than their own mean produces a negative value (scikit-learn developers, 2026d). A statistic that can go negative out of sample is not a percentage of anything.

The second problem is the one Section 8.4 armed you for. Even read charitably as R² = 0.96 on individual six-month spend, the number is implausible against everything the lab documented about this outcome: a quarter of eligible customers spend nothing, spend is concentrated in a small minority, and the honest baselines land in a known range. The corrected reading, which the deliverable should print in exactly this shape: “Our model’s grade was earned under exam conditions we can show you — a declared frame, a sealed test set opened once, two baselines on the same customers. The vendor’s number, as presented, was not earned under any conditions we can inspect, and the statistic it names does not exist for this kind of target. Before the two numbers can share a sentence, the vendor must answer five questions” — and the five questions are Table 8.5 read aloud: What is the frame, and what exactly does the number measure? Where is the snapshot, and can you prove no feature crosses it? What was held out, how often was it consulted, and was model complexity chosen against it? What baseline was beaten, by how much, on which metric? What is the error in dollars, on our customers, and what happens to it over time? A vendor with good answers becomes a genuine candidate — build-versus-buy is a real decision, and Section 8.14’s third vignette shows the buy side winning honestly. A vendor without them is selling in-sample flattery at an annual license price.

The second wrong reading is gentler and more corrosive: “The model says C004471 will spend 585 dollars, so put her on the list — the model knows.” The model does not know. The score is a conditional average — what customers with C004471’s snapshot profile spent, on average, in past outcome windows — wearing the precision of its decimal places, and the test MAE is the printed reminder that individual customers sit tens to hundreds of dollars from their scores routinely. The corrected reading changes the object of confidence from the customer to the list: the model is not trusted to be right about C004471; it is trusted to sort eight thousand customers well enough that the top eight hundred capture more of next season’s spend than any competing list — which is precisely the claim the capture exhibit tests and the reason Section 8.6 made capture the decision metric. This distinction — score as verdict versus score as ordering — is the predictive sibling of Chapter 7’s coefficient-is-not-a-lever box, and it changes what “the model was wrong” even means: one customer’s shortfall is noise; the list underperforming the spreadsheet is failure.

The third wrong reading is the most expensive, and it is the one the opening brief pre-empted on purpose: “The model’s list captures six points more spend than the spreadsheet’s, so the Backstage program will generate that much additional revenue.” It will not, and the gap between those two sentences is the difference between a prediction and a treatment effect. The capture figure says that the customers the model selected turned out to spend more than the customers the spreadsheet selected. It says nothing about how much either group would have spent without a lookbook. A program that gifts eight hundred customers who were always going to spend heavily has bought recognition, not growth, and the honest deliverable prices it that way: the model improves who receives the treatment, and only a holdout — a randomly withheld slice of the eligible list, measured six months later — can price what the treatment does. Naming that experiment in the deliverable is not a hedge. It is the analysis pricing its own upgrade, the same closing move Chapter 7’s drivers deliverable made.

The deliverable that survives all three misreadings has a fixed anatomy, and the lab produced every part: the frame, one page, seven declarations and the label definition with its four dates, because it is the document every later question lands on; the leaderboard from Code 8.11, cross-validated on training customers only, with both baselines and every candidate’s margin; the single test-set table from Code 8.14, with capture leading and MAE and RMSE in dollars beside it; the capture head-to-head from Code 8.16, model list against spreadsheet list, with the dollar gap and the count of names that change, because the meeting is deciding between those two artifacts and this table grades the actual choice; the expected-value sketch, with incremental flagged as the assumption the design cannot certify and the holdout test named as the instrument that could; the vendor verdict as a one-paragraph memo built on the five questions; and the monitoring plan of Section 8.10 — who checks the realized error when the outcome window matures in June, against what benchmark, which input distributions are watched in the meantime, and what number triggers a retrain — because the deliverable is not a scored list but a scored list with a maintenance contract, and only one of those survives its first drift.

8.13 Business Analytics in Practice

This section turns from the fictional case to how predictive models operate — and fail — in professional marketing organizations, where the models are commodities, the pipelines are products, and the failures that matter are almost never statistical. All three vignettes below are composites of recurring professional patterns rather than reports about a single named organization.

8.13.1 Lead Scoring and the Adoption Problem

The first vignette is lead scoring, the business-to-business world’s version of this chapter’s scored list. The setting recurs across enterprise software firms: marketing builds a propensity model scoring inbound leads on firmographic fit and behavioral signals, validated properly out of sample, demonstrably better than the alphabetical-and-instinct routing it replaced — and six months later, sales representatives are ignoring the scores. The post-mortems rarely find a modeling failure. They find an adoption failure with a recurring anatomy: nobody translated the score into the representative’s decision, so a lead marked 74 arrived with no instruction about what to do before lunch; nobody showed the representatives evidence that the ordering worked, so the first false positive — a high-scored lead who wasted an afternoon — confirmed every prior about ivory-tower analytics; and nobody instrumented the loop, so scores were computed, exported, and quietly unconsulted while the dashboard reported the model live.

The operational artifact that separates the teams who fix this from the teams who repeat it is unglamorous: an adoption metric, defined as precisely as any model metric and reported beside it. The working version is a contact-rate-by-score-decile report — of the leads the model placed in the top decile, what share were worked within the service-level window, and how does that share compare with the bottom decile? A model whose top and bottom deciles are worked at the same rate is not in production in any sense that matters, whatever the pipeline logs say. Teams that instrument this discover the gap in weeks rather than quarters, and the fix is usually not a better model but a translation layer: score bands with named actions rather than a continuous number, a weekly list rather than a field in a record, and a published back-test showing representatives what the ordering was worth on leads they remember. The practice lesson is the one Provost and Fawcett (2013) build their expected-value framing around: the model is an input to a decision system, and the system — thresholds in the user’s units, feedback that earns trust, telemetry on whether scores change behavior — is what produces value. A perfectly ranked list that no one acts on has an expected value of exactly zero.

8.13.2 Silent Decay After a Pricing Change

The second vignette is silent decay, composited from a pattern retention teams describe often enough to treat as a genre. A subscription commerce firm builds a customer-value model — a spend-horizon frame much like this chapter’s — and wires it into onboarding offers, service-tier routing, and paid-acquisition bidding. It works, and because it works, it stops being watched. Fourteen months later, someone notices acquisition economics sagging and pulls the thread: the model has been mis-scoring new customers for two quarters, ever since a pricing-and-packaging change restructured early purchase behavior — the model’s most important input. Nothing had errored. The scores still arrived, plausible-looking, on schedule; the model was simply answering a question about a relationship that no longer held.

Note which failure this is, in the vocabulary of Section 8.10. The inputs did shift, so there was data drift. But the expensive part was concept drift: the same early-purchase pattern now implied a different lifetime value, because the packaging change altered what an early purchase meant (Gama et al., 2014). No amount of watching the input distributions alone would have caught the second, which is why the repaired practice monitors both. The operational artifact here is a monitoring view with three panels and one rule. The panels: realized error on each cohort as its outcome window matures, plotted against the benchmark the original test set established; the distribution of the two or three most important inputs, this month against the training window; and the score distribution itself, which moves when either of the first two does. The rule is the part teams skip: a written trigger naming the number that forces a retrain — realized error exceeding the test benchmark by a stated margin for two consecutive cohorts — plus a standing clause that known structural changes, a repricing or a merchandising overhaul, force revalidation without waiting for the cadence. Sculley et al. (2015) systematized why this is engineering rather than diligence: the model is the smallest box in the system diagram. The lesson compresses to a sentence worth carrying into every deployment conversation: a model without a monitoring plan is not an asset; it is a liability with good first-quarter manners.

8.13.3 Build Versus Buy

The third vignette is built versus buy, composited from diligence processes that recur wherever marketing technology is purchased, because most working analysts will meet this chapter’s machinery not as builders but as evaluators of other people’s claims. The modern marketing stack sells predictions everywhere: customer-data platforms ship churn and spend scores, CRM suites ship propensity models, ad platforms ship value-based bidding — and the buy side is often right, because a vendor amortizing pipeline engineering across hundreds of clients can genuinely out-build a two-person analytics team. What separates a sound purchase from an expensive slide is the diligence, and the diligence is this chapter run as a questionnaire: the frame (what unit, target, and horizon — a “churn score” with an undisclosed churn definition is not yet a product claim, per Section 8.3’s label discipline); the evaluation (out of sample on whose data, and how was complexity chosen); the baseline (better than what — scores that barely beat trailing spend are repackaging your own spreadsheet); the leakage audit (what are the features, and can impossible values be range-checked); and the maintenance contract (who monitors, who retrains, what triggers).

The artifact that converts that questionnaire from a conversation into evidence is an acceptance test written into the pilot: the vendor scores a historical snapshot of the client’s own customers, chosen by the client, with the outcome window already complete but withheld from the vendor. The client then computes the metrics — capture at the client’s own capacity, error in the client’s own dollars — against the client’s own incumbent rule, using code the client wrote. Performance on someone else’s base is drift waiting to be discovered; performance on a vendor-selected sample is a sales artifact; performance on a client-held holdout is the only version that means what the contract will claim. Teams that run this report an unexpected benefit: the vendors worth buying agree to it easily, because the honest ones did the work and would rather demonstrate it than argue about it. The meeting where a vendor cannot name their own snapshot date is a license fee saved.

8.13.4 In Your First Analyst Job

In your first analyst job, these vignettes compress into one expectation: the modeling is the minority of the work. The frame conversations, the label negotiations, the leakage audits, the baseline politics — the spreadsheet’s defenders are stakeholders, not obstacles — the adoption design, the monitoring calendar, the vendor diligence: that is the job’s actual shape, and the fitted model sits inside it as one well-understood component. The industry’s software has made fitting nearly free. What it has not made free, and what this chapter trained, is knowing whether the number the software prints deserves to be believed — and the analysts who advance are the ones whose belief, granted or withheld, turns out to be worth something.

8.14 Ethics, Acting on Predictions About People

The Business Analytics in Practice section described organizations wiring predictions into decisions; this section examines the wiring as an ethical act, extending the guide’s running discussions — data use (Section 1.13), problem framing (Section 2.12), measurement design (Section 3.13), cleaning as editorial power (Section 4.14), honest summarization (Section 5.14), differential treatment of segments (Section 6.16), and causal language (Section 7.15) — to what this chapter adds: decisions made about individual people, automatically and at scale, on the strength of statements about people who resemble them. The chapter’s ethics is not a caution about misuse at the margins. Acting on predictions is the technique’s entire purpose, and the act carries obligations that the arithmetic, however honest, does not discharge on its own.

Begin with what a score is, because the ethics follows from the epistemology. The model’s 585 dollars for customer C004471 is, as Section 8.13 established, a conditional average over historically similar customers, carrying a typical error the test set measured in tens of dollars. Statistically, that is a fine instrument for building a list. Ethically, the shift happens now of action: the Backstage program does not gift a conditional average — it gifts, or declines to gift, a person, and the person receives the treatment their resemblance earned, not the treatment their actual future would have justified. At the individual grain, some of the model’s decisions are simply wrong, in both directions.

It is tempting to say that the test MAE tells you how often. It does not, and the precision matters because this is a chapter about not overstating what a number establishes. MAE is unsigned and it is an average magnitude: it reports how far the model misses on held-out customers, on average, in dollars. It does not report a frequency of wrong decisions, it does not say which side of the cutoff any customer fell on, and it does not distinguish a twenty-dollar miss that changed nothing from a two-hundred-dollar miss that displaced someone from the list. What the test MAE is, exactly, is the honest numerical size of the model’s individual-level ignorance. That is enough to dissolve the comfortable idea that a validated model has earned the right to be unexamined — and it is worth being equally precise about what validation bought. Validation did not certify the average in any general sense. It estimated average performance on a stated holdout population, at a stated snapshot, under a stated use, with sampling uncertainty attached and no claim about transport to a later period. The decisions, meanwhile, land one at a time.

Three obligations follow, each with a StyleCraft edge. The first is proportionality between the model’s demonstrated reliability and the stakes of the automated act. A capture improvement of a few points honestly justifies routing lookbooks; the same model, repurposed to decide which customers’ service complaints get escalated or whose returns get flagged for review, is the same arithmetic carrying stakes it was never validated for — and the repurposing conversation, which will arrive because scores are convenient and licenses are sunk costs, is the analyst’s to have. Function creep, not initial deployment, is where scoring programs most often cross their ethical lines.

The second is transparency of the automated targeting itself, in two registers. Internally, the deliverable’s frame and monitoring plan are the transparency instrument: anyone operating the program can learn what the score is, what it is not, and how wrong it typically runs — which is why Section 8.13 made the frame page one. Externally, the customers on neither side of the line were asked whether their transaction histories could audition them for gifts, and while a lookbook is benign, the guide’s standing test — would you be comfortable explaining the program’s mechanics, aloud, to the customers on both sides of the cut? — is worth running precisely because it is easy to pass here. A program that passes it at twenty dollars a slot and fails it when the same scores set service priorities has located its own boundary, and the analyst who ran the test knows where the boundary is before the repurposing meeting starts.

The third obligation is memory that the scores inherit the past. The model learned from behavior shaped by everything StyleCraft has already done — where stores were opened, who was marketed to, which segments got offers — so the scored list partly re-awards yesterday’s attention as tomorrow’s gifts; the disparate-impact frame introduced with Barocas and Selbst in Section 1.13 applies whenever resemblance-based decisions recycle historical patterns, and the StyleCraft dataset’s own designed trap — the newest stores, whose customers score poorly for no reason except the stores’ youth — is waiting in exactly this chapter’s machinery for any analyst who scores without asking what the training window could not yet know. The eligibility rule of Section 8.3 is where this bites hardest and least visibly: a population defined by having purchased in an eighteen-month window is a population defined, in part, by who was reachable during it.

8.15 Chapter Summary

This chapter crossed the line Chapter 7 drew and the guide had been approaching since Part II began: from models whose coefficients are the deliverable to models whose outputs are, and from grading analyses on their honesty about the past to grading them on their accuracy about a future they have not seen. The main point is a change of posture. Explanatory work earns trust through audited structure — signs, magnitudes, units, diagnostics. Predictive work earns trust structurally: a frame declared before fitting, a wall in time between what the model may know and what it must predict, an opponent named before the race, complexity chosen inside the training data, and a grade computed once, on sealed customers, in the decision’s own currency. Every impressive number produced outside that structure is flattery of one kind or another, and the chapter’s standing red flag inverted the beginner’s instinct in a sentence: in prediction, the result that looks too good almost never is.

The machinery arrived in three movements. The framing movement built the predictive frame — unit, target, features, horizon, and, in this chapter’s version, the metrics and baselines declared alongside them — and the label as a manufactured measurement: a snapshot date, a feature window and an outcome window with both ends enforced, an eligibility rule evaluable at the snapshot, and the insistence that a zero is a label rather than a gap. The modeling population was distinguished from the deployment population, and the customers the rule excludes were named as an activation problem rather than absorbed silently. Leakage was named as the canonical error and cataloged by route — temporal, target-derived, outcome-window contamination, split contamination, population — with the audit, not the software, as its only reliable detector, and with the split-contamination rule stated precisely enough to be usable: what contaminates a split is a step that estimates something from data, not any step performed before it.

The evaluation movement built the honest exam. Train/test splits defeat in-sample flattery, and the chapter was careful about what they buy: a random customer holdout estimates cross-sectional generalization at the historical snapshot and does not measure time-forward transport to a later scoring date. MAE and RMSE were read in dollars and separated from the decision metric, because the Backstage program acts through a fixed ranked cutoff and a model can improve its typical miss while worsening the ordering at the line — so top-decile capture leads the frame, and MAE leads the calibration reporting. Baselines were operationalized from Section 2.7’s interpretive discipline into working opponents, with the incumbent’s sorted spreadsheet dignified as the baseline with teeth and the direction of the error metrics stated plainly, since a model far below the baselines on MAE is the suspicious one and a model far above them is the broken one. Overfitting and underfitting were read jointly from the training-versus-validation signature, with the guarantee attached to training RMSE rather than to training MAE, and with the limit on honest performance described as a documented picture of the outcome’s volatility rather than a computed noise floor the chapter had not earned. Cross-validation stopped being a concept and became the instrument: one-fold generator, one leaderboard, every candidate and both baselines scored by the same call, the test set sealed throughout.

The deployment movement carried the score into the world: rank and cut under a fixed budget with a declared tiebreaker, the capture head-to-head against the incumbent list, expected-value framing at concept level with the word incremental flagged as unearned, and the deployed model’s real life stated in three terms rather than one — data drift in the inputs, concept drift in the relationship, performance decay as the observable consequence — each with its own monitoring implication and its own trigger. The AI section named the assistant-era failures — the convenient-table leak, the celebrated impossible fit, the missing opponent, the metric that does not match the act — and installed the five-point audit that the labs then ran three ways: on the honest pipeline, on a deliberately manufactured leak whose smoking gun was a single impossible range check, and on an assistant’s unframed enthusiasm. Sixteen numbered cells carried the argument into executable form: the miniature baselines with their one-slot capture, a snapshot-safe feature builder that takes its window as an argument, both windows enforced at both ends, the eligible population reconciled against the customer file, the labeled table audited before anything was fitted, the split frozen by customer identity, five candidates registered with every learned step inside a pipeline and a selection rule written before the race, the cross-validated leaderboard with its paired fold margins, the flexibility dial turned without touching the test set, the rule executed unedited to name a finalist, that finalist opened against the sealed customers once, the leak manufactured on the same customers and caught by arithmetic, and the two ranked lists compared deterministically. The interpretation, practice, and ethics sections carried the results into the meeting room, where the vendor’s 96 percent met the five questions and a vocabulary correction, the score was demoted from verdict to ordering, the capture gap was distinguished from a treatment effect, and the act of gifting eight hundred people on the strength of their resemblance to other people acquired its permitted-use note.

Looking ahead, the chapter’s target was a dollar amount, and dollars were a mercy: a numeric miss has a size, and MAE could price it. Most of marketing’s predictive questions refuse the mercy. Will this customer churn; will she respond; will he click, convert, return — yes or no, where a prediction is not close or far but right or wrong, and where being wrong has two faces with two different prices. The next chapter takes up classification, propensity, and churn: the logistic machinery that turns features into probabilities, the confusion matrix that gives the two error types of their business names, the lift and gains machinery this chapter borrowed one exhibit from, and the threshold decision — set not by statistics but by the asymmetric costs Section 2.7 planted and this chapter previewed — where the analyst’s model finally negotiates directly with the CFO’s economics.

8.16 Exercises for Practice

The following exercises practice the chapter’s main habits: declare the frame before touching data, define labels in time with both ends of both windows enforced, trace every feature for leakage, choose complexity inside the training data, grade models only out of sample and only against named baselines, read the training-versus-validation signature, translate scores into decisions with their economics attached, and treat too-good results as audits waiting to be run. They are organized into two groups. Core chapter practice is the required path and should be completed by every student.

8.16.1 Core Chapter PracticeExercise 8.1 Concept Check

Answer each in two or three sentences, in your own words.

• State what changes when a model’s outputs, rather than its coefficients, become the product, and name one practice that is acceptable in explanatory work and a failure in predictive work.
• Define the four commitments of the predictive frame and explain why the decision metric and the baselines must be declared before fitting rather than after.
• Explain why “future spend” is a manufactured label rather than an existing column, and state why each of the four declared dates is load-bearing rather than decorative.
• State this chapter’s eligibility rule, explain why both of its conditions can be evaluated at the snapshot, and name the population it excludes and what should happen to that population.
• Define leakage and explain why it cannot be detected by fitting statistics or by a train/test split, only by audit. What is its most reliable symptom?
• A colleague says that any transformation applied before the split contaminates it. Give one transformation for which this is true and one for which it is false and state the test that distinguishes them.
• An analyst reports that their model’s training MAE is 31 dollars and its five-fold cross-validated MAE is 78 dollars. Diagnose, using Table 8.4’s vocabulary, and prescribe.
• Why does this guide compute the mean baseline’s prediction from the training labels only, and which rule from Section 8.5 is that the smallest example of?
• RMSE substantially exceeds MAE on the same predictions. What does the gap announce, and which StyleCraft customers — by Chapter 5’s findings — are almost certainly responsible?
• Explain why capture and MAE can disagree about which candidate is better, and state what the deliverable should do when they do.
• State the three clauses of the model-comparison discipline, and for each, describe the loophole it closes.
• Distinguish data drift, concept drift, and performance decay, and state which of the three can be detected without waiting for outcomes to mature.

Exercise 8.2 Draw the Frame

For each brief below, write a complete predictive frame — unit of prediction, target with a fully defined label (snapshot, feature window with both ends, outcome window with both ends, filters, eligibility), horizon, decision metric, calibration metrics, and the strongest honest baseline — and then name the single most likely leakage route from Table 8.2 and how your frame forecloses it.

• The CRM manager wants to time win-back email for customers likely to lapse predict each customer’s days until next purchase.
• Merchandising wants to know, at the end of each customer’s first month, what they will spend over their first year, to set onboarding investment.
• The email team wants to predict, per customer, next quarter’s email-driven revenue — and someone proposes using each customer’s full-history open rate as a feature. Address the proposal explicitly.
• The retention team wants to score every customer monthly, including customers acquired last week. State what your frame can and cannot do for the newest customers, using the representation argument of Section 8.3 rather than a claim about technical impossibility.

Exercise 8.3 Spot the Leak

Each pipeline below contains at least one leakage route from Table 8.2, or none. Name the route (or certify the pipeline clean), cite the table row, and state the repair.

• To predict spend_next6m from a December 31 snapshot, an analyst includes loyalty_tier as of the June 30 database extract, reasoning that tier changes slowly.
• A churn model’s training population is “customers with at least three orders in the 24-month window,” with churn labeled in the final six months.
• An analyst standardizes all features on the full labeled table, then splits 80/20, fits, and reports test MAE against a mean baseline computed from training labels.
• An analyst expands two predictors into third-degree polynomial terms on the full labeled table, then splits 80/20 and fits. Be careful with this one and justify your verdict with the test from Section 8.4 rather than with the position of the split.
• To predict first-year, spend at the end of month one, the feature list includes number of orders in the first 30 days and average order value over the customer’s lifetime.
• A team refits Chapter 6’s segmentation on all labeled customers, adds the resulting segment indicators as features, then splits and cross-validates the model. Name what leaked and state the two structural repairs Section 8.9 offers.
• An assistant’s pipeline for the Backstage frame reports test R² of 0.95, and its feature importance list is led by a column named monetary.

Exercise 8.4 AI Pipeline Audit

Give an AI assistant the raw StyleCraft tables and this deliberately loose prompt: “Build me the best possible model to predict customer spend — maximize accuracy.” Then audit the response with Table 8.5’s five points, in order and in writing: reconstruct the frame the assistant implicitly chose and compare it to Table 8.1; trace every feature to its window, running the recency range check if applicable; check the split and preprocessing hygiene, including whether any complexity or tuning decision was made by consulting a test set; determine what baseline, if any, the pipeline was compared against, and compute the missing ones yourself on the same customers; and grade the reported performance for plausibility against the volatility figures you recorded in Code 8.6, stating whether the assistant’s own narration celebrated or questioned it. Separately, list every performance claim in the reply and mark each as earned under exam conditions or not, flagging any use of classification vocabulary for a numeric target. Conclude with two sentences on which audit point caught the most serious problem.

8.17 Glossary of Terms

This glossary includes only the terms introduced in this chapter. Each definition is tied to the sources used in the chapter rather than added for decoration.

Baseline model. A deliberately simple prediction rule — mean, majority class, or last value — evaluated on the same folds, the same test set, and the same metrics as a candidate model, whose margin of defeat is the measure of the model’s demonstrated value (adapted from Provost & Fawcett, 2013).

Bias–variance tradeoff. The intuition that prediction error decomposes into systematic error from oversimplification (bias) and instability from fitting sample-specific noise (variance), with model flexibility trading one for the other and honest performance minimized between the extremes (adapted from James et al., 2021; Domingos, 2012).

Concept drift. Change over time in the relationship between a model’s inputs and its target, so that the learned mapping no longer describes how the outcome is generated; invisible in the input distributions and detectable only as outcomes mature (adapted from Gama et al., 2014).

Cross-validation. Estimation of out-of-sample performance from training data alone, by k-fold rotation of a held-out fold and averaging of the k held-out scores; the standard instrument for development-time decisions, including model complexity, while the test set stays sealed (adapted from James et al., 2021; scikit-learn developers, 2026b).

Data drift. Post-deployment change in the distribution or quality of a model’s input features, so that scored cases increasingly differ from training cases; detectable by watching the inputs alone. Covariate shift is its special case, in which the predictor distribution changes while the conditional relationship to the outcome remains stable (adapted from Gama et al., 2014).

Expected value framing. The translation of predictions into actions by weighing each action’s possible outcomes by their probabilities and subtracting its costs, so that score thresholds become economic lines rather than arbitrary ranks (adapted from Provost & Fawcett, 2013).

Feature window. The declared span of time, bounded at both ends and closing at the snapshot date, from which a predictive model’s features may be computed; the past side of the label’s wall in time (adapted from Kuhn & Johnson, 2013).

Horizon. The forward distance between the snapshot date and the end of the outcome window: how far into the future the model’s claim extends, constraining validation timelines and retraining cadence (adapted from Provost & Fawcett, 2013).

In-sample flattery. This guide’s name for the systematic optimism of performance measured on the data a model was fitted to, which grows with model flexibility and cannot be estimated from the training data itself (adapted from James et al., 2021).

Label. The constructed, recorded value of the target for a historical case, manufactured by choosing a snapshot date, a feature window, an outcome window, and eligibility rules; the answers a predictive model trains on (adapted from Provost & Fawcett, 2013; Kuhn & Johnson, 2013).

Leakage. The presence in training features or evaluation procedure of information unavailable at real prediction time — from the future, from the target’s construction, or from the evaluation data — inflating measured performance that collapses on deployment (adapted from Kaufman et al., 2012).

MAE (mean absolute error). The average absolute difference between actual and predicted values, in the target’s native units, weighting all misses equally; an unsigned average magnitude rather than a count of wrong decisions, and this guide’s lead calibration metric (adapted from Willmott & Matsuura, 2005).

Outcome window. The declared span of time after the snapshot date, bounded at both ends, over which the label is measured; the future side of the wall, which no feature may touch (adapted from Kuhn & Johnson, 2013).

Overfitting. A model’s capture of training-data-specific noise that does not generalize, diagnosed by training performance substantially exceeding held-out performance and worsening with added flexibility (adapted from James et al., 2021; Domingos, 2012).

Performance decay. The erosion of a deployed model’s predictive quality over time as data drift, concept drift, or both accumulate, observed by monitoring realized error as fresh outcomes mature (adapted from Gama et al., 2014; Sculley et al., 2015).

Pipeline. An object chaining every learned step of a model — scaling, imputation, clustering, the estimator itself — so that all of them are refitted together on each training fold; the structural defense against split contamination (adapted from scikit-learn developers, 2026a).

Predictive frame. The four declared commitments of a predictive model — unit of prediction, target, features, and horizon — written before fitting, together with the decision metric, the calibration metrics, and the baselines, as the predictive extension of the analytic specification (adapted from Provost & Fawcett, 2013; Kuhn & Johnson, 2013).

Predictive modeling. The construction of models whose purpose is accurate output for individual unseen cases, evaluated out of sample, as distinct from explanatory modeling graded on the honesty of its coefficients (adapted from Shmueli, 2010; Provost & Fawcett, 2013).

RMSE (root mean squared error). The square root of the average squared difference between actual and predicted values, in native units but disproportionately sensitive to large misses; its excess over MAE announces error concentration (adapted from Willmott & Matsuura, 2005).

Snapshot date. The declared moment that splits time in a label’s construction: features may use only information available on or before it; the label is measured only after it (adapted from Kuhn & Johnson, 2013).

Split contamination. The leakage route in which a step that estimates something from data is fitted before the split, letting held-out cases influence the values the model trains on; distinguished from transformations that estimate nothing and are therefore unaffected by the order of operations (adapted from Kaufman et al., 2012; scikit-learn developers, 2026a).

Test set (holdout). The randomly held-out portion of labeled cases, sealed during development and consulted once for the selected model, whose performance estimates cross-sectional out-of-sample error at the historical snapshot (adapted from James et al., 2021).

Capture (at a selection share). The share of a population’s actual outcome-window spends held by the customers a model ranks inside the program’s capacity — the capacity divided by the population being ranked. It is the decision metric of a fixed-capacity ranked targeting program, and the metric on which a model’s list is compared with an incumbent’s; where the share is a tenth, it is the familiar top-decile case. A constant score carries no ranking content, so its capture is defined only once ties are broken on a declared rule, and a random tiebreak makes that figure the selection share itself (adapted from Provost & Fawcett, 2013).

Training set. The portion of labeled cases from which models, every learned preprocessing step, and every development decision including model complexity are derived (adapted from James et al., 2021).

Underfitting. A model’s failure to capture structure that does generalize, diagnosed by poor performance on training and held-out data alike (adapted from James et al., 2021).

8.18 References

Barocas, S., & Selbst, A. D. (2016). Big data’s disparate impact. California Law Review, 104(3), 671–732. https://doi.org/10.15779/Z38BG31

Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87. https://doi.org/10.1145/2347736.2347755

Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), Article 44. https://doi.org/10.1145/2523813

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning: With applications in R (2nd ed.). Springer. https://doi.org/10.1007/978-1-0716-1418-1

Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, methodology, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), Article 15. https://doi.org/10.1145/2382577.2382579

Kuhn, M., & Johnson, K. (2013). Applied predictive modeling. Springer. https://doi.org/10.1007/978-1-4614-6849-3

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

Provost, F., & Fawcett, T. (2013). Data science for business: What you need to know about data mining and data-analytic thinking. O’Reilly Media.

scikit-learn developers. (2026a). Common pitfalls and recommended practices [Software documentation]. https://scikit-learn.org/stable/common_pitfalls.html

scikit-learn developers. (2026b). Cross-validation: Evaluating estimator performance [Software documentation]. https://scikit-learn.org/stable/modules/cross_validation.html

scikit-learn developers. (2026c). PolynomialFeatures [Software documentation]. https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html

scikit-learn developers. (2026d). r2_score [Software documentation]. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems 28 (pp. 2503–2511). Curran Associates.

Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289–310. https://doi.org/10.1214/10-STS330

Willmott, C. J., & Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30(1), 79–82. https://doi.org/10.3354/cr030079