Chapter 09

Classification, Propensity, and Churn Models

From How Much to Whether: Probabilities, Verdicts, and the Price of Being Wrong in Both Directions

Dr. Jose Mendoza, Academic Director and Clinical Associate Professor

Version 1.0 · July 2026

Except where otherwise noted, this chapter is licensed under CC BY 4.0.

Chapter Information

ABSTRACT

This chapter develops classification — the prediction of yes/no outcomes — as the completion of the machinery Chapter 8 built for numeric targets, with churn as its application. It constructs the churn label as a measurement decision, builds logistic regression from the log-odds equation, and adds classification trees and k-nearest neighbors as structurally different scorers compared on a training-only cross-validated leaderboard. It then builds the evaluation vocabulary: the confusion matrix in campaign terms, accuracy’s failure under imbalance, precision and recall as trade-off instruments, ROC and AUC as threshold-free measures of ranking skill, and calibration as the separate property an economic threshold consumes. The destination is the decision threshold derived from asymmetric error costs, with propensity scoring, deciles, lift, and gains translating scores into budget-sized lists. The labs verify every figure by hand, keep the test set sealed until a finalist is chosen, and close with a group-level fairness screen.

KEYWORDS

classification; churn; logistic regression; confusion matrix; precision; recall; ROC/AUC; decision threshold; propensity; lift

VERSION AND DATE

Version 1.0 · July 2026 · Language: English (United States)

SUGGESTED CITATION

Mendoza, J. (2026). Classification, propensity, and churn models. In Applied business analytics for marketing decision-making: Business analytics and data visualization (Chapter 9, Version 1.0) [Open educational resource]. CC BY 4.0.

LICENSE AND RIGHTS

Copyright © 2026 Jose Mendoza. Except where otherwise noted, this work is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). You may share and adapt this material for any purpose, provided appropriate credit is given. Third-party trademarks, screenshots, figures, and other materials remain subject to their respective rights and licenses.

Google Colab is a product of Google LLC. "Python" and the Python logos are trademarks or registered trademarks of the Python Software Foundation. pandas, NumPy, Matplotlib, and scikit-learn are sponsored or affiliated projects of NumFOCUS, a 501(c)(3) nonprofit charity in the United States. statsmodels is a community-developed project distributed under the modified BSD license. ChatGPT is a product of OpenAI, Claude of Anthropic, Gemini and NotebookLM of Google LLC, and GitHub Copilot of GitHub, Inc. Product names are used for identification only and do not imply endorsement. StyleCraft Collective is a fictional company created for instruction.

COMPANION REPOSITORY

Datasets, notebooks, and figure sources for this chapter are in the Applied Business Analytics companion repository on GitHub.

GENERATIVE AI USE

Generative artificial intelligence and other AI-assisted tools were used in the research, writing, revision, and production of this chapter, including literature discovery, source organization, outlining, preliminary drafts, prose revision, support for code and analytical examples, and document formatting. These tools were used under the author’s direction and are not credited as authors, researchers, or sources. The author determined the chapter’s scope, learning objectives, methods, interpretations, and recommendations, and reviewed and approved all AI-assisted material: factual claims and citations were checked against the underlying sources rather than accepted from AI-generated summaries, and code and analytical outputs were tested or otherwise reviewed for accuracy. Responsibility for the accuracy, originality, and final form of this chapter rests entirely with the author. A fuller statement appears in the front matter of the complete guide.

Chapter Learning Objectives

• By the end of this chapter, students should be able to:
  
• Identify marketing’s binary prediction questions, distinguish classification from class-probability estimation, and explain at intuition level why linear regression is the wrong instrument for a binary target.

• Define a churn label completely for a non-contractual business — activity window, activity definition, eligibility, and the snapshot structure of Section 8.3 — and explain why the label is a measurement decision with program consequences, per the four-element discipline of Section 3.5.
• Explain the logistic function and log-odds at intuition level, fit and read a logistic regression, and translate its coefficients into plain-language statements about odds and probabilities without overclaiming causation.
• Describe how classification trees and k-nearest neighbors score customers, state each method’s characteristic strengths and failure modes, and explain why k-NN requires the feature scaling of Section 6.7.
• Construct a confusion matrix from a scored list and a threshold, name its four cells in business vocabulary, and verify its arithmetic — counts, accuracy, precision, recall — by hand.
• Explain accuracy’s failure under class imbalance, compute the majority-class baseline of Section 8.7 as the working floor, and state the imbalance-handling options at concept level.
• Compute and interpret precision, recall, and F1 in campaign terms, and choose which to lead with based on the decision’s cost structure.
• Read ROC curves and AUC at working level as measures of ranking skill, state the full range AUC occupies and what it does and does not certify, and treat implausibly high AUC as the leakage signal of Section 8.4.
• Distinguish a model’s ranking skill from its probability calibration, evaluate calibration out of fold on training data, and state the condition calibration imposes on any economically derived threshold.
• Select a model family and its settings using stratified cross-validation inside the training customers, per Section 8.9, and open the sealed test set once for the finalist.
• Derive a decision threshold from stated false-positive and false-negative costs using the expected-value framing of Section 8.10, explain why the software’s default of 0.5 is rather an inheritance than a decision, and reconcile an economic threshold with a budget constraint.
• Build ranked-targeting exhibits — decile tables, lift charts, and gains charts — and translate them into a treatment recommendation and a budget argument.
• Verify an AI-built or vendor-supplied classifier with the five-point audit of Section 8.11 plus the classification supplement — base rate and majority baseline, hand-checked confusion matrix at the economic threshold, threshold re-derived from stated costs, calibration evidence.
• Examine a classifier’s error rates across customer groups, explain how classification models can cause the outcomes they predict, and state the fairness and feedback-loop obligations that attach to acting on classifications at scale.

Chapter 8 ended with a mercy it named on its way out. The Backstage deliverable predicted a dollar amount, and a dollar amount can be missed by a little or by a lot: the mean absolute error priced the typical miss, the leaderboard priced the model’s margin over the sorted spreadsheet, and the whole apparatus of honest evaluation ran on the fact that a numeric error has a size. Most of marketing’s predictive questions refuse the mercy. Will this customer churn; will she respond to the offer; will he click, convert, come back — yes or no, where the decision the model feeds is not close or far but right or wrong, and where being wrong has two faces with two different prices. This chapter builds the machinery for those questions: the logistic model that turns features into probabilities, the trees and neighbors that offer second opinions, the confusion matrix that gives the two kinds of error their business names, the curves that grade a model’s ordering skill, the calibration check that grades its probabilities, and — the chapter’s destination, and the payoff of a thread planted seven chapters ago — the decision threshold, chosen not by statistical convention but by the asymmetric costs of the two mistakes. Every discipline Chapter 8 installed ride along unchanged: the frame is still declared before fitting, the label is still a manufactured measurement, leakage is still the canonical error, model selection still happens inside the training data, and the test set is still touched once. What is new is that the number the model produces is a probability, the number the meeting wants is a verdict, and the space between them is where this chapter’s analyst earns the seat at the table.

CONCEPT

What This Chapter Is Really About

Chapter 8’s machinery graded predictions by how far they missed. This chapter’s verdicts cannot miss by a distance — a customer predicted to churn either does or does not — and that small change of geometry rearranges everything downstream. One number can no longer summarize performance, because a binary verdict fails in two distinguishable ways, and the two failures bill different departments: the offer wasted on a customer who was staying anyway is a marketing cost, and the customer lost because no one intervened is a revenue cost, and nothing in the mathematics makes them equal. The chapter’s methods, however, do not produce verdicts. They produce probabilities, and a probability forecast can still be wrong — predicting 0.90 for a customer who stays is a larger probabilistic error than predicting 0.55, and 0.51 and 0.99 yield the same verdict at a 0.5 cut while carrying very different confidence. Class verdicts therefore create two error types; probability forecasts additionally require evaluation of their ranking and their calibration, and this chapter grades all three.

The chapter’s real subject is the sequence of separations this forces on the analyst: the score separated from the verdict, because the model produces a probability and someone must choose where to cut it; the model’s ranking separated from its calibration, because a well-ordered list of scores can still be systematically too high or too low; the model’s quality separated from the threshold’s wisdom, because a well-ranked list can be cut in a foolish place; and the statistical output separated from the economic decision, because the threshold that converts scores into treatments is not a modeling parameter at all — it is a business policy, derivable from the costs of the two errors, and the analyst who cannot re-derive it from stated costs has delegated the most consequential number in the pipeline to a software default. Chapter 2 planted the idea that the cost of being wrong is asymmetric. This chapter is where that idea stops being a lens and becomes arithmetic.

Source: Course concept developed for this guide, informed by Provost and Fawcett (2013).

9.1 Marketing Decision Context: The Save Offer and the Default Threshold

Chapter 8’s deliverable shipped, and the Backstage Preview list was only the top of the fall retention program. Its other end concerns the customers StyleCraft is losing. Chapter 5’s certified findings put a number on the worry that opened this guide: repeat-purchase rates are weakest exactly where the expansion bet is placed, and a large fraction of customers who buy once never buy again. The program’s answer is a save track — internally, the Comeback Edit: a personalized offer combining a $15-off-$75 voucher, free shipping, and a curated selection drawn from the customer’s purchase history. Finance has costed the treatment at roughly twelve dollars per treated customer once redemption rates and margin give-up are blended. The question is the list. Who gets the Comeback Edit — and, harder, who decides who gets it, on what evidence, cut at what line? The save track goes live with the November journeys; the targeting rule must be locked in twenty-eight days.

Three answers are already circulating, and each conceals a failure this chapter exists to expose. The first is the incumbent rule, defended by the same CRM operations lead whose sorted spreadsheet fought honorably in Chapter 8: “Anyone with no purchase in ninety days is lapsing. Send the offer to the ninety-day-lapsed list. We have run win-back this way for two years.” The second lives in the marketing platform StyleCraft already licenses, whose Churn Risk panel sits one click from activation behind the claim that ends meetings: “Our churn model is 89 percent accurate.” Activate the flag, treat the flagged, done. The third comes from the growth lead, and it has the charm of arithmetic honesty: “Email is nearly free. Why are we rationing a retention offer at all? Send the Comeback Edit to everyone who has ever bought. You cannot mis-target a blanket.” The CFO, attending because the offer is margin and margin is the CFO’s, has a preemptive answer: the voucher is not an email. Every redemption is real money, a blanket sends prices out at twelve dollars times the whole file, and the finance model says most of that spend would land on customers who were never leaving.

Each answer has a real virtue, and stating the flaws is this chapter’s tour. The ninety-day rule is cheap, transparent, and — the labs will show — genuinely informative, because long recency is the strongest churn signal in StyleCraft’s data by design. It is also a one-variable model wearing a policy costume, and its one variable is precisely the one Chapter 5’s lumpiness findings booby-trapped: the Suburban Occasion cohort routinely goes ninety quiet days between legitimate seasonal baskets, so the rule sprays vouchers across customers mid-cadence while missing the discount-dependent buyer whose recent small purchase disguises an exhausted relationship. The platform’s 89 percent invites the question Chapter 8 trained: accurate against what, measured how, at what threshold? This chapter adds a sharper one. If roughly seventy percent of eligible customers do not churn, then seventy percent accuracy is available for free by predicting that nobody churns, so 89 percent is a nineteen-point margin over the do-nothing rule — a margin that could be genuinely valuable or could be produced by the wrong mixture of errors, and the number cannot be interpreted at all without the base rate, the confusion matrix, the threshold, and the cost structure beside it. That is the chapter’s real complaint about accuracy: not that 89 percent necessarily fails, but that accuracy alone cannot establish whether the model helps the retention program. And the platform’s flag arrives as verdicts, not scores, which means someone else’s threshold — almost certainly the software default of 0.5 — has already made StyleCraft’s most consequential retention decision without attending a meeting. The blanket proposal fails on the CFO’s arithmetic, but its deeper failure returns in Section 9.14: treating everyone is not a neutral act with wasted cost, because a retention offer reaches customers it can annoy, train to wait for discounts, or — the industry’s most uncomfortable finding — remind to leave.

So, the VP of Marketing has commissioned the analyst — still you — with a deliverable in five parts. A churn definition in writing, because “churn” is not a column StyleCraft possesses but a measurement the analyst must construct and defend. A scored model, selected inside the training customers and graded once out of sample against stated baselines including the ninety-day rule — but graded with a new vocabulary, because a yes/no decision fails in two directions and one accuracy number hides the difference. A threshold derived from the program’s own economics, shown as arithmetic the CFO can re-run on a napkin and accompanied by the calibration evidence that entitles the arithmetic to be believed. A ranked list with its lift and gains exhibits, because the budget conversation will not be “is the model good” but “how deep into the file do we fund.” And a fairness appendix the VP did not ask for but will need, because the scored file includes the customers of the newest stores, and Section 9.15 will show that a naive churn model has already made up its mind about them for reasons that have nothing to do with their loyalty. The chapter builds those five parts in order: the target and the label, the scorers, the evaluation vocabulary, the threshold and the ranked-targeting exhibits, then the AI audit, the labs, and the three sections that rehearse the meeting.

9.2 From How Much to Whether: Binary Targets and Why the Line Fails

Chapter 8 defined the predictive frame — unit, target, features, horizon — and nothing in that definition required the target to be a dollar amount. The frame machinery transfers whole; one sentence of reminder suffices, and the reader who wants the full apparatus has Section 8.2. What changes in this chapter is the target’s type, in exactly Section 3.4’s sense: spend_next6m was a ratio-level quantity, and the targets of this chapter are binary categoricals — two classes, conventionally coded 1 for the class the business is watching for (the positive class) and 0 for its complement. Marketing manufactures such targets constantly, because marketing actions are mostly binary at the customer grain: send or do not send, and the customer responds or does not; show the ad or do not, and she clicks or does not; the contract month arrives, and he renews or does not. Will she churn, will he respond, will they convert, will it bounce — the yes/no questions are not a special case of marketing prediction; they are its majority.

Two tasks hide inside “predict a yes/no outcome,” and separating them now saves confusion for the rest of the chapter. Classification, strictly, assigns each case a predicted class: this customer will churn, that one will not. Class-probability estimation assigns each case a probability of the positive class: this customer’s churn probability is 0.62, that one’s is 0.08. The second task is the one this guide’s methods actually perform and the one marketing decisions actually need, because a probability preserves the ordering and the uncertainty that a verdict throws away — and because, as Section 9.9 will show, the conversion from probability to verdict is a business decision that should not be made inside the model at all. The distinction sounds academic and is the chapter’s load-bearing wall: nearly every misuse this chapter catalogs, from the platform’s pre-thresholder flag to the assistant’s celebrated accuracy, begins by collapsing the two tasks into one.

DEFINITION

Classification and Class-Probability Estimation

Classification is the prediction of a categorical target: assigning each case to one of a fixed set of classes, in this guide’s scope two. Class-probability estimation is the prediction, for each case, of the probability of the positive class — a score between 0 and 1 that ranks cases by likelihood and defers the assignment of verdicts to a separately chosen threshold. The methods of this chapter estimate probabilities; classifications are produced from them by thresholding, and the threshold is a decision, not a model property.

Source: Adapted from Provost and Fawcett (2013).

Why not simply reuse Chapter 8’s instrument? Linear regression will run on a 0/1 target without complaint — the software fits, the coefficients print, and the output even has a respectable name in older literature (the linear probability model). The failure is not that it refuses to work; it is that it works badly in ways an intuition-level argument makes visible. First, the line does not know the target’s boundaries: fit spend-style regression to a 0/1 churn outcome and it will cheerfully predict churn values of 1.3 for the most lapsed customers and −0.2 for the most engaged, numbers that cannot be probabilities and cannot be read as anything else. Second, the line’s constant slope is the wrong shape for how binary risk behaves: a straight line says that thirty additional days of recency change churn risk by the same amount whether the customer is at the engaged extreme, in the uncertain middle, or already almost certainly gone — but risk near the floor and ceiling has almost nowhere to move, and the real action lives in the middle. The curve that honest binary risk follows is flat near 0, steep in the middle, and flat again near 1 — an S-shape, and Section 9.4’s entire contribution is a principled machine for producing it. Third, the errors of a binary target misbehave in ways that undermine the least-squares machinery of Section 7.6 — a technical point this guide leaves at one sentence, because the first two arguments already carry the verdict: the right instrument for a bounded, S-shaped quantity is a model built for bounded, S-shaped quantities.

One more piece of vocabulary completes the setup, and it is a number rather than a method: the base rate, the share of the positive class in the population — the fraction of StyleCraft’s eligible customers who churn in the outcome window. The base rate is the single most important number in any classification project and the one most reliably missing from celebrated results. It is the context that makes every other metric legible: an accuracy of 89 percent means one thing when churners are half the file and something entirely different when they are a tenth of it; a model that finds churners at twice the base rate is doing real work, and the same model quoted without the base rate is doing unverifiable work. This guide’s rule, enforced in every lab and exercise: no classification result is reported without its base rate in the same exhibit, and Section 9.6 will show what happens to the analyst who forgets — the majority-class baseline of Section 8.7, which sat politely at concept level through the numeric chapter, is about to start winning arguments.

9.3 Defining Churn: The Label as a Measurement Decision

Section 8.3 established that a predictive label is manufactured, not found: a snapshot date splits time into a feature window the model may know and an outcome window the label summarizes, eligibility rules decide who gets a labeled row, and the whole construction is a measurement act carrying Chapter 3’s obligations. That machinery transfers to this chapter unchanged — same snapshot of December 31, 2025, same eighteen-month feature window opening July 1, 2024, same January-through-June 2026 outcome window, same two-condition eligibility rule — and none of it is re-derived here; Section 8.3 is the reference. What this section adds is the problem that makes churn the most definitionally treacherous label in marketing: for StyleCraft, churn does not exist until the analyst defines it, and every element of the definition changes who the model learns to find.

The reason is structural, and it divides the industry’s retention problems into two families. In a contractual business — wireless plans, streaming subscriptions, gym memberships — churn is an event: the customer cancels, the relationship has a legal off switch, and the data records the date it was flipped. In a non-contractual business like StyleCraft’s, there is no switch. No customer ever tells an apparel brand she has left; she simply does not come back, and “has not come back yet” and “is never coming back” look identical in the transaction log for months. Chapter 5 measured this at the descriptive grain — repeat rates and their windows were metric constructions, per the four-element discipline of Section 3.5 — and this chapter inherits the problem at the label grain: churn must be defined into existence as a threshold on observed silence, and the definition is a choice among defensible alternatives, not a discovery of a fact.

DEFINITION

Churn and the Churn Label

In a non-contractual business, churn is the sustained cessation of a customer’s purchasing relationship, operationalized rather than observed: a customer is labeled as churned when a defined activity — usually a completed purchase — fails to occur within a defined outcome window following the snapshot date. The churn label therefore requires every element of the label construction of Section 8.3 plus a definitional commitment specific to churn: what counts as activity, how long the silence must last to count as departure, and which customers are eligible to be labeled at all. Different defensible definitions produce different labels, different models, and different treated populations.

Source: Adapted from Neslin et al. (2006) and Provost and Fawcett (2013).

The StyleCraft definition, stated in the four-element form of Section 3.5 because a label is a metric with a modeling job: churned equals 1 for an eligible customer with zero completed purchase orders in the outcome window, January 1 through June 30, 2026. Numerator concept: customers with no outcome-window purchase order. Denominator: none — it is a per-customer flag, not a rate. Window: the six-month outcome window, inherited from the Section 8.3 snapshot structure. Filters and eligibility: the Chapter 8 rule in both of its conditions — the customer’s signup_date falls on or before December 31, 2025, and the customer completed at least one purchase between July 1, 2024, and December 31, 2025 — so that every labeled customer has a feature-window history to be scored on. Both conditions are load-bearing, and the second is the one drafts drop. Recency, average order value, discount share, and store share are all undefined for a customer with no feature-window purchases, because each divide by a quantity that is zero; a population defined only by signup date would therefore carry rows whose features do not exist. Customers who signed up and never purchased remain a named activation population, counted and reported, outside this churn model entirely.

One implementation detail carries more weight than its length suggests, and the labs enforce it. It is tempting to build the churn flag as the exact complement of Chapter 8’s spend label — churned when spend_next6m equals zero — because the two are usually the same customers and the code is one line shorter. They are not guaranteed to be the same customers. Zero net revenue is not the same event as zero purchase activity: an order can net to zero after a discount and a returned line, purchases can be offset by returns and adjustments, and a customer can generate negative net revenue while having unambiguously transacted. The definition above says activity, so the label is built from counted purchase orders in the outcome window, and the reconciliation against spend_next6m is reported as a separate diagnostic rather than assumed. Where the two disagree, the disagreement is a finding about the revenue column, not a bug in the label — and the deliverable should not claim the churn flag is the exact complement of the spend label unless the dataset specification guarantees that every completed purchase produces strictly positive net revenue.

Every element of the definition is a lever, and moving any one of them changes the program. Table 9.1 walks the three that matter most. Shorten the outcome window to ninety days and the label sweeps in every Suburban Occasion customer sitting mid-cadence between seasonal baskets — the base rate inflates, the model learns to find slow shoppers rather than departing ones, and the Comeback Edit ships vouchers to customers who were three weeks from a full-price occasionwear order. Lengthen it to a year and the label grows more truthful about permanent departure but the program loses its purpose, because a customer silent for twelve months is far harder to save than one silent for five — the definition trades label purity against intervention timing, and the trade is a marketing judgment, not a statistical one. Redefine activity to include non-purchase engagement — an app open, an email click — and the label stops measuring commercial churn and starts measuring attention churn, a different construct that may or may not be what a voucher can treat. None of these alternatives is wrong. What is wrong is choosing one silently, and the deliverable’s first page therefore states the definition, the rejected alternatives, and the reason — the same discipline the metric disputes of Chapter 3 taught, now with a model and a budget downstream of the choice.

Table 9.1
The churn definition’s levers and what each one changes

One consequence deserves its own paragraph because it will resurface in three later sections. The definition makes the base rate a constructed number: StyleCraft’s churn rate under this label is not a fact about StyleCraft so much as a fact about StyleCraft under this definition, and Chapter 5’s certified repeat-purchase findings suggest it will land somewhere near a third of the eligible file — a designed expectation the labs will confirm against the shipped answer key. That constructed-ness is why this chapter’s evaluation sections insist on carrying the base rate everywhere: two churn models built on two defensible definitions can post identical accuracies while chasing different customers, and only the definition-plus-base-rate pair makes a reported result auditable. It is also why the platform’s Churn Risk panel fails the first question before any statistics arrive: 89 percent accurate at predicting what definition of churn is not a rhetorical question, and a vendor who cannot answer it is selling a model of an undisclosed construct — the churn-flavored sibling of the undisclosed metric definitions that Chapter 3’s one-metric-three-numbers dispute made famous.

9.4 Logistic Regression: Probabilities from the Log-Odds Machine

The label exists; the first scorer can now be built, and this guide builds it from the instrument the reader already trusts. Section 9.2 established what the model must produce — an S-shaped probability, bounded between 0 and 1, steep in the uncertain middle and flat at the confident extremes — and logistic regression produces exactly that by the smallest possible modification of Chapter 7’s machinery: keep the linear equation, change what it computes. A linear regression says the outcome is an intercept plus weighted feature. A logistic regression says something one step removed: a quantity called the log-odds of the outcome is an intercept plus weighted features, and the probability is recovered from the log-odds by a fixed squashing curve — the logistic function — that maps any number the equation produces, however large or negative, into the interval the probability must occupy.

The intuition arrives fastest through odds; a currency marketing student already speak from every context where chances are quoted as ratios. A probability of 0.75 is odds of 3 to 1: three chances of the event for every one against. A probability of 0.5 is even odds, 1 to 1. A probability of 0.10 is odds of 1 to 9. Odds re-express probability on a scale that runs from zero to infinity instead of zero to one, and the logarithm of the odds runs the scale out symmetrically in both directions — negative log-odds for unlikely events, zero for even odds, positive for likely ones. That unbounded, symmetric scale is what the linear equation lives on: features move the log-odds up and down freely, the way they moved dollars in Chapter 7, and the logistic function translates the result back into a bounded probability at the end. The S-shape falls out automatically: a one-unit feature change always moves the log-odds by the same amount, but the same log-odds movement changes the probability a great deal near 0.5 and almost not at all near the extremes — which is precisely the risk behavior Section 9.2 said a straight line could not express.

DEFINITION

Logistic Function and Log-Odds

The odds of an event are the ratio of its probability to its complement’s: p ÷ (1 − p). The log-odds are the natural logarithm of the odds — an unbounded, symmetric rescaling of probability on which zero means even odds. The logistic function is the inverse translation: it converts any log-odds value back into a probability between 0 and 1, producing the characteristic S-curve that is flat near both extremes and steepest at even odds.

Source: Adapted from Hosmer et al. (2013).

DEFINITION

Logistic Regression

Logistic regression models the log-odds of a binary target as a linear function of the features — an intercept plus weighted feature values, exactly the equation form of Chapter 7 — and converts the modeled log-odds into a predicted probability through the logistic function. Its coefficients are estimated from labeled data (by maximum likelihood, machinery this guide leaves to the software), and each coefficient is read as the change in log-odds — equivalently, the multiplicative change in odds — per one-unit difference in its feature, holding the model’s other features constant.

Source: Adapted from Hosmer et al. (2013) and James et al. (2021).

Reading the coefficients is where Chapter 7’s disciplines transfer with one translation step, and the translation is the only genuinely new skill. A logistic coefficient lives on the log-odds scale, which no meeting speaks; exponentiating it produces an odds ratio, which careful plain language can carry. A coefficient of 0.009 on recency_days exponentiates to about 1.009: each additional day since last purchase multiplies the odds of churn by 1.009 — about nine-tenths of a percent higher odds per day, compounding, so that ninety days of additional silence multiply churn odds by roughly 2.2. A negative coefficient on app_user exponentiating to 0.55 says app users carry a little over half the churn odds of comparable non-users. Three disciplines attach immediately, all inherited. The ceteris paribus boundary of Section 7.8 applies verbatim: each statement holds the model’s other features constant and is silent on everything omitted. The verb registry of Section 7.2 applies verbatim: these are associations wearing precise multiplicative clothing, and “app users churn less” earns its verb while “the app retains customers” does not — the customers chose the app, and Section 7.8’s omitted-enthusiasm argument transfers without modification. And where categorical predictors do enter a model, the dummy machinery of Section 7.9 applies verbatim, with every odds ratio read against the declared reference category rather than absolutely. What does not transfer is any instinct to read effects on probability as constant: because of the S-curve, the same odds ratio moves a mid-risk customer’s probability substantially and a low-risk customer’s barely at all — statements about “percentage-point changes in churn probability” are only honest with a stated starting point, and the labs enforce the habit.

One design decision belongs here rather than in the lab, because it is a modeling judgment rather than a coding convenience. It is tempting to enter Chapter 6’s named segments into this chapter’s classifier as dummy variables, since the segments are memorable and the meeting speaks their names. This guide does not, for the reason Section 8.9 established: a segment assignment is a learned feature, produced by a scaler and a set of centroids estimated from data, so entering it as a precomputed column lets the customers being scored shape the definitions the model trains on — Table 8.2’s split-contamination row, committed by a step nobody thinks of as a model. The core classifier therefore uses the behavioral variables the segments were built from, all of which the Chapter 8 feature builder already produces at the snapshot: recency, frequency, tenure, average order value, orders and revenue per month, discount share, store share, occasionwear share, and the two engagement flags. Segment membership returns in Section 9.12’s fairness appendix as an external profiling variable — a lens for reading who the model is treating — which is the role it can play without contaminating anything.

One practical note belongs here before the reassurances. The coefficients this section teaches you to read are coefficients in customer units — days, orders, shares — and the predictive candidates of Section 9.12 will not supply them, because they standardize their inputs and carry the software’s default regularization, which is right for prediction and wrong for reading. Code 9.6 therefore fits a small, separate model whose only job is interpretation, labeled in the lab as ineligible for the race. Keeping the reading model and the scoring model apart is the explanation-versus-prediction boundary of Section 8.2, drawn inside a single chapter.

One reassurance and one caution close the section. The reassurance: logistic regression is this chapter’s primary instrument for the same reason linear regression anchored Chapter 7 — it is transparent, stable, fast, and its coefficients can be audited against known structure, which is what Section 9.11’s audit will do with the dataset’s designed churn drivers (long recency, single-purchase history, discount-only buying, and store-only suburban profiles all raise churn odds by design). The caution: everything the model produces is a probability estimate conditioned on the feature-window snapshot — a statement about customers who resembled this one, in exactly the sense Section 8.13 established for scores, and nothing in the sharpened vocabulary of odds changes what Section 9.13 will do to the sentence “the model says she is leaving.”

9.5 Trees and Neighbors: Two More Ways to Score

Logistic regression reaches a probability through an equation; this section adds two scorers that reach one through entirely different structures, and the point of the addition is only partly the methods themselves. Chapter 8’s comparison discipline — same folds, same metric, same baselines, one leaderboard fixed before the race — was built for exactly this moment: multiple candidate models, structurally dissimilar, competing for one budget line. The two candidates are chosen for their instructional contrast: one scores by asking questions, the other by finding resemblances, and each fails in a way that reviews a Chapter 8 lesson from a new angle.

A classification tree scores a customer by interrogation. The model is a sequence of yes/no questions about features — is recency_days greater than 120? is frequency equal to 1? is discount_share above 0.6? — arranged so that each answer routes the customer down a branch, until she lands in a leaf: a terminal group of training customers who answered every question the same way. Her predicted churn probability is simply the churn rate among the training customers in her leaf. The tree is grown from the data by recursive splitting: at each step, the algorithm chooses the question that best separates churners from non-churners among the customers reaching that point, then repeats within each branch (Breiman et al., 1984). Two properties make trees permanently useful in marketing rooms. They are readable — a modest tree prints as a flowchart a CRM manager can walk through, and its top question is a finding (on StyleCraft’s designed data, expect it to be a recency split, the ninety-day rule rediscovering itself as the first branch). And they are natively interactive in Section 7.9’s sense — a tree that splits on discount_share only within the low-frequency branch has found, without being told, that the discount signal matters for some customers and not others. Their characteristic failure is the one Section 8.8 built the vocabulary for: an unconstrained tree keeps splitting until leaves hold a handful of customers each, memorizing training noise leaf by leaf — overfitting in its most visible form, with the telltale gap between training and cross-validated performance of Table 8.4, controlled in practice by limiting depth or leaf size. The lab manufactures exactly this failure on purpose, entirely inside the training customers, because a tree overfitting is the fastest visual demonstration of Chapter 8’s U-curve a student will ever run.

DEFINITION

Classification Tree

A classification tree predicts by routing each case through a learned sequence of yes/no feature questions to a leaf, where the predicted class probability is the share of positive-class training cases in that leaf. The tree is grown by recursive splitting — repeatedly choosing the feature question that best separates the classes among cases reaching each point — and its flexibility is governed by limits on depth or leaf size, without which it overfits in exactly the sense of Section 8.8.

Source: Adapted from Breiman et al. (1984) and Provost and Fawcett (2013).

k-nearest neighbors dispense with fitted structure altogether. To score a customer, find the k training customers most like her — nearest in feature space, by the distance machinery of Section 6.6 — and predict the churn rate among those k neighbors. With k = 25, a customer whose twenty-five nearest training neighbors include ten churners is scored 0.40. The method’s premise is pure resemblance: customers like you did what you will do (Cover & Hart, 1967). Its virtues are conceptual honesty — it is the “conditional average over similar customers” reading of Section 8.13 made literal — and freedom from any assumed equation shape. Its obligations and failures are equally instructive. Because it runs on distances, it inherits wholesale the scaling discipline of Section 6.7: unscaled, recency_days’ hundreds dominate every 0/1 flag, and the neighbors are neighbors in recency only — the tyranny of units, third appearance, and the reason the lab’s k-NN pipeline standardizes inside each training fold per the split-hygiene rule of Table 8.2. The choice of k is the bias–variance dial of Section 8.8 with a new handle: k = 1 memorizes (every training customer is her own nearest neighbor — training accuracy perfect, held-out accuracy poor), while enormous k averages everyone toward the base rate (the majority-class baseline rediscovered as a limiting case), and the lab therefore selects k by cross-validation inside the training data rather than by declaration. And k-NN produces no coefficients at all — nothing to read, nothing to audit against designed structure — which makes it this guide’s standing example of the trade Table 7.4 named: predictive flexibility purchased with explanatory silence.

DEFINITION

k-Nearest Neighbors (k-NN)

k-nearest neighbors predict a case’s class probability as the positive-class share among the k training cases closest to it in feature space, under a stated distance measure computed on features scaled per Section 6.7. It fits no equation and learns no coefficients; its flexibility is governed by k, with small k overfitting and large k defaulting toward the base rate.

Source: Adapted from Cover and Hart (1967) and James et al. (2021).

Table 9.2 collects the comparison the leaderboard will run. Its last row states a designed expectation rather than a promise: on StyleCraft’s data, whose churn structure was planted as a logistic relationship with a realistic noise ceiling, the three methods should land in a similar band with the logistic model competitive or ahead. That expectation is about this dataset, and it should not be generalized into a claim that method choice does not matter. The industry’s own churn tournament found the opposite: across forty-five submissions on common validation data, differences in predictive accuracy between modeling approaches were large enough to change the profitability of a churn campaign by hundreds of thousands of dollars, with logit and tree approaches outperforming several alternatives (Neslin et al., 2006). The lesson the tournament teaches is the one this chapter’s leaderboard enforces: approaches differ materially, so they must be compared on common validation data under a common campaign objective — which is a stronger argument for the leaderboard than the folk claim that any method will do.

Table 9.2
Three scorers compared

9.6 The Confusion Matrix: Four Outcomes with Business Names

The scorers produce probabilities; suppose for the next three sections that a threshold has been chosen — the software’s 0.5 will do for now, under protest that Section 9.9 will formalize — so that every customer carries a verdict. The instrument that grades verdicts is a two-by-two table so central to this chapter that its arithmetic is the chapter’s verification theme: every count in it will be checked by hand in the labs, because it is the exhibit most often pasted into decks unexamined and most easily garbled in transit. Cross the truth against the prediction and every customer lands in exactly one of four cells. The model predicted churn and the customer churned: a true positive. Predicted churn and she stayed: a false positive. Predicted stay and he churned: a false negative. Predicted stay and she stayed: a true negative. The table of counts is the confusion matrix, and this guide’s insistence — the reason Table 9.3 exists — is that the four cells are read in business vocabulary before any metric is computed from them, because each cell is a different event in the Comeback Edit program with a different line on the P&L.

DEFINITION

Confusion Matrix

A confusion matrix cross-tabulates actual against predicted classes at a stated threshold, partitioning all evaluated cases into true positives (predicted and actual positive), false positives (predicted positive, negative), false negatives (predicted negative, positive), and true negatives (predicted and actual negative). Every threshold-dependent classification metric — accuracy, precision, recall, and their relatives — is an arithmetic summary of these four counts, and changing the threshold changes the matrix.

Source: Adapted from Provost and Fawcett (2013) and Fawcett (2006).

Table 9.3 states the four cells in the program’s own vocabulary, and its cost column is written to match the ledger Section 9.9 will derive rather than to sound expensive. Two of the four rows repay care, because the intuitive phrasing overcounts. A false negative does not cost the lost customer’s entire future margin: the program could never have captured that margin with certainty, because only a fraction of treated would-churners are expected to be saved. What a false negative cost is the missed expected incremental save — the save rate times the retained customer’s margin, which Section 9.9 will price at roughly forty dollars. A false positive does not cost the treatment plus margin given away, either: the twelve-dollar treatment cost is already a blended figure with redemption and margin give-up inside it, so adding margin a second time double-counts. What a false positive cost is the blended twelve dollars, spent on a customer who was staying.

Table 9.3
The confusion matrix in Comeback Edit vocabulary

From the four counts, the first summary metric is the obvious one: accuracy, the share of all verdicts that were correct — true positives plus true negatives, over everything. Accuracy is legitimate arithmetic, this chapter’s opening vocabulary note is hereby discharged (the word that was misapplied to a numeric target in Section 8.11’s vendor deck belongs here, to verdicts), and accuracy is also the most reliably misleading number in applied classification, for a reason the fourth row of Table 9.3 telegraphs: the true-negative cell is usually enormous. Churn, response, conversion, fraud — marketing’s positive classes are minorities, a situation called class imbalance, and under imbalance accuracy is dominated by the model’s performance on the class nobody is asking about. The arithmetic is brutal and clarifying. If thirty percent of StyleCraft’s eligible customers churn, the rule “predict that nobody churns” is seventy percent accurate while identifying not one customer to save. The platform’s 89 percent begins to look different against that floor — not necessarily worse, but no longer self-explanatory — and the floor has a name the reader already knows it is the majority-class baseline, defined in Section 8.7, held at concept level through the numeric chapter with a promise that it would do serious work here. This is the work. Every accuracy figure in this chapter’s deliverables is reported beside the majority-class baseline and the base rate, and an accuracy whose margin over the do-nothing rule is never computed is Chapter 8’s baseline discipline waiting to deliver its verdict.

DEFINITION

Accuracy and Class Imbalance

Accuracy is the share of evaluated cases whose predicted class matches the actual class. Class imbalance is the condition — usual in marketing — in which one class heavily outnumbers the other, and under it accuracy is dominated by the majority class: the majority-class baseline of Section 8.7 achieves accuracy equal to the majority share while identifying no positive cases at all. Accuracy is therefore reported only beside the base rate and the majority-class baseline, and never as a classifier’s headline under imbalance.

Source: Adapted from Provost and Fawcett (2013) and He and Garcia (2009).

Imbalance also raises a modeling question students meet the moment they search beyond this guide: an internet’s worth of advice recommends rebalancing the training data — resampling the minority class upward, the majority down, or synthesizing minority examples — so the model “sees” more churners. This guide’s treatment is deliberately brief and at concept level. Rebalancing exists, sometimes helps ranking, and always distorts the model’s probability scale (a model trained on a rebalanced world overestimates churn in the real one), which matters enormously in a chapter whose destination is a threshold denominated in real-world probabilities and whose Section 9.8 makes calibration an explicit deliverable. The disciplined default for problems at StyleCraft’s imbalance — a minority class near a third, not a thousandth — is to change nothing about the data and everything about the evaluation: grade with the base-rate-aware instruments this chapter builds, keep every split and every fold stratified so both sides of the wall carry the true base rate, and reserve rebalancing for the rare-event regimes where it earns its distortions. The one-sentence version students should carry: imbalance is usually an evaluation problem wearing a data-problem costume.

9.7 Precision, Recall, and F1 in Campaign Terms

Accuracy failed because it pooled the four cells into one number; the repair is to ask the two questions the program cares about, each a different ratio of the same counts. The first question belongs to the CFO, who is paying for the treated list: of the customers we sent the offer, how many were leaving? That is precision — true positives over all predicted positives, the purity of the treated list. Its complement is the waste rate: precision of 0.5 means half the Comeback Edit budget landed on customers who were staying anyway. The second question belongs to the CRM manager, who owns the churn number: of the customers who were leaving, how many did we reach? That is recall — true positives over all actual positives, the coverage of the at-risk population. Its complement is the leak: recall of 0.67 means a third of the departing customers walked out without anyone extending a hand.

DEFINITION

Precision and Recall

Precision is the share of predicted positives that are positive — of those treated, how many were the real thing; the purity of the target list. Recall is the share of actual positives that were predicted positive — of the real thing, how many were caught; the coverage of the at-risk class. The two are computed from the same confusion matrix, respond to the threshold in opposite directions, and answer different stakeholders’ questions; neither is “the” performance number. Precision is undefined when nothing is treated, which is why a policy table that includes a do-nothing rule must handle the empty denominator explicitly rather than divide by zero.

Source: Adapted from Provost and Fawcett (2013) and Fawcett (2006).

The two metrics are structurally in tension, and the tension is the threshold made visible. Lower the threshold and the model calls churn more freely: the treated list grows, recall rises (fewer departing customers missed), and precision falls (more loyal customers swept in). Raise the threshold and the reverse. Neither direction is progress in itself — the movement just relocates cost from one cell of Table 9.3 to the other — and this is the precise sense in which Section 9.2 insisted the score be separated from the verdict: the model fixes how good the ordering is, and the threshold chooses where along the ordering the program stops treating, which is a question about the two cells’ prices, not about the model. Students should therefore hear alarm bells at any unqualified claim that a classifier “has” a precision or a recall: it has one at a threshold and quoting either without the threshold — or quoting the pair achieved at different thresholds, a vendor-deck classic — is the metric-shuffling of Section 8.9 in its classification uniform.

When a single number is genuinely required — a leaderboard column, a tuning target — the convention is F1, the harmonic mean of precision and recall. The harmonic mean is the right kind of average for the job because it is dragged toward the smaller of the two: a model with precision 0.9 and recall 0.1 posts an F1 near 0.18, not the flattering 0.5 an arithmetic mean would print, so gaming one metric at the other’s expense stops paying. F1’s limit is stated with equal clarity: it weights the two errors equally, and this entire chapter exists because the two errors are not equally priced. F1 is a summary for contexts that have not yet priced their errors; Section 9.9 is for contexts that have, and StyleCraft, with twelve dollars against forty, is about to become one. Table 9.4 collects this section’s vocabulary as working questions — the design rules the deliverable’s evaluation exhibits follow, with each metric assigned to the stakeholder whose question it answers.

Table 9.4
The working questions of classification evaluation

9.8 Ranking Skill and Probability Quality: ROC, AUC, and Calibration

Precision and recall grade the model after a threshold has committed it to verdicts; this section grades the two properties of the scores themselves — how well they order customers, and whether the numbers mean what they say — before any threshold intervenes. The first instrument sweeps the threshold instead of choosing it. Start at a setting so high no one is called a churner, then lower it continuously; at each setting, record two rates: the true positive rate (recall — the share of actual churners correctly flagged) and the false positive rate (the share of actual non-churners incorrectly flagged). Plotted against each other across the full sweep, the pairs trace the ROC curve. A model with no ranking skill traces the diagonal — at every threshold it flags churners and loyalists in equal proportion, exactly what random guessing achieves. A model with perfect ranking hugs the top-left corner: all the churners flagged before any loyalist. Real models bow between the two, and the area under the curve — AUC — summarizes the bow in a single number, with an interpretation compact enough to survive a meeting: AUC is the probability that a randomly chosen actual churner receives a higher score than a randomly chosen actual non-churner, with tied scores contributing half credit (Fawcett, 2006).

DEFINITION

ROC Curve and AUC

The ROC curve plots a scorer’s true positive rate against its false positive rate across all possible thresholds, tracing the trade-off between catching positives and falsely flagging negatives. AUC — the area under the ROC curve — summarizes threshold-free ranking skill and ranges from 0 to 1. An AUC of 0.5 indicates chance-level ordering; values above 0.5 indicate useful ordering in the stated direction; values below 0.5 indicate that the scoring direction is systematically reversed or worse than chance. AUC equals the probability that a randomly chosen positive case outranks a randomly chosen negative one, with ties counted as half. It is insensitive to the base rate and to any threshold: it grades the ordering, and neither the calibration nor the costs.

Source: Adapted from Fawcett (2006) and Google for Developers (2026).

Three properties define AUC’s proper use in this guide, each earning a paragraph of discipline. First, AUC is a good general ranking diagnostic and a poor sole criterion. It cannot be flattered by imbalance and does not depend on a threshold nobody has justified yet, which is exactly why it is the right column for a first-pass comparison of structurally different scorers. But it grades ordering across the entire false-positive-rate range, and the Comeback Edit decision does not live across the entire range: it lives near a particular economic threshold, inside a budgeted upper portion of the file, using probability values to compute expected economics. A model with the highest overall AUC can be inferior in the top two deciles, inferior near the 0.30 cut, worse calibrated, and worth less money. The frame therefore declares a two-stage rule rather than a single number, and the two stages answer different questions. AUC, cross-validated and compared fold by fold against a stated baseline, is the model-family selection metric: it chooses which kind of scorer orders customers best, and it is the right instrument for that job precisely because it is threshold-free. Expected net value at the derived threshold, computed from calibrated out-of-fold probabilities, is the policy-validation gate: the selected model must clear it, against zero and against the incumbent rule, before any money is committed. Calibration is the condition that makes the gate computable at all, and precision, recall, and treated-list size at the stated threshold are the policy diagnostics reported beside it.

The ordering of those two stages is not cosmetic. Ranking evidence is available from raw scores; economic evidence is not, because a dollar figure computed from uncalibrated probabilities is denominated in a currency the model does not print. So, the family is chosen first on ordering, the chosen model is calibrated inside the training data, and only then does the arithmetic of Section 9.9 run. A leaderboard that carries a net-value column beside raw scores has quietly inverted that order, and the labs of Section 9.12 are built so that it cannot. This is Chapter 8’s principle — grade the model on the artifact the meeting is choosing — carried into classification with one added clause: grade it in dollars only once the probabilities have earned the right to be multiplied by them.

Second, AUC has calibrated expectations, and the calibration is Chapter 8's too-good logic wearing this chapter's units. Individual churn behavior carries irreducible noise — the outcome window contains job changes, moves, weddings, and whims that no snapshot feature can see — and the StyleCraft data was designed with a realistic ceiling in mind, which the labs will document empirically for this population rather than assert from a general industry figure. That distinction matters: published churn AUCs vary substantially with the label, the horizon, the population, and the data available, so a number quoted as the industry's honest range certifies nothing about StyleCraft's. What transfers digit for digit is the reflex Table 8.4 installed. An AUC far above what the base's volatility, the honest baselines, and the label definition would support is not a triumph; it is a leak until proven otherwise — the label's ingredients have almost certainly contaminated the features, with the outcome-window recency of Table 8.2's third route as the usual suspect — and the auditor's first move is the feature trace, not the celebration. The labs manufacture this failure deliberately, as Chapter 8's did, because a leak a student has built is a leak she will recognize in a vendor deck.

Third, AUC is silent about the property the next section’s arithmetic consumes. AUC grades ordering; the threshold derivation grades levels. A model can rank customers beautifully while producing probabilities that are systematically too high or too low — multiply every score by 0.6 and the ordering, and therefore the AUC, does not change at all, while every economic calculation built on those scores does. The property that closes the gap is calibration: a scorer is well calibrated when, among customers scored near 0.30, roughly thirty percent churn. Calibration is checked with a reliability exhibit — bin the scores and compare each bin’s mean predicted probability against its observed churn rate and its customer count — and, where the comparison is materially off, corrected with a cross-validated calibration procedure fitted inside the training data (scikit-learn developers, 2026a). The discipline that governs the correction is Chapter 8’s: calibration is evaluated and, if necessary, repaired using training-only, out-of-fold evidence, never by adjusting a model repeatedly against the test set until the reliability plot looks agreeable.

DEFINITION

Calibration

A probability scorer is calibrated when its predicted probabilities match observed event frequencies: among cases scored near p, about a share p are positive. Calibration is distinct from discrimination — the ranking quality AUC measures — and neither implies the other. It is assessed with a reliability exhibit comparing mean predicted probability against observed positive rate within score bins, alongside each bin’s count, and it is the property that entitles a threshold derived from expected costs to be applied to a model’s raw scores.

Source: Adapted from scikit-learn developers (2026a).

9.9 The Threshold Decision: Asymmetric Costs, Operationalized

Section 2.7 introduced the asymmetric cost of being wrong as interpretive discipline: before any analysis, name which direction of error is dearer, because the answer should shape what the analysis treats carefully. Chapter 8 previewed the asymmetry twice — in the Backstage program’s two error types and in the choice between MAE and RMSE — and promised that this chapter would make it the whole instrument. This section keeps the promise, and the registry entry deserves to be honored in the text as it was written this is the operationalization of Section 2.7. The asymmetric-cost idea stops being a lens for reading results and becomes the arithmetic that produces one — the decision threshold, the number that converts every score into a treatment and every error into a bill.

Begin with what the default conceals. Classification software, asked for verdicts, cuts at a probability of 0.5, and the choice feels so natural it is rarely recognized as one: call churn when churn is more likely than not (scikit-learn developers, 2026b). But 0.5 is the right threshold only under an assumption nobody at StyleCraft believes — that the two errors cost the same. Cutting at 0.5 says a wasted twelve-dollar voucher and a missed save are equivalent misfortunes, and that a customer with a 0.45 churn probability — nearly a coin flip on losing her entire future relationship — should be left alone because the coin leans slightly toward staying. The default is not a neutral setting; it is a cost assumption wearing a factory setting’s camouflage, and the first act of threshold discipline is simply to see it. Two further conditions ride inside the equal-cost reading and should be stated with it: it assumes the scores are calibrated probabilities in the sense of Section 9.8, and it assumes the stated two-error ledger contains the relevant program economics — no additional benefit or cost attached to the true positives and true negatives, no class weighting, and no transformed probability scale quietly in play.

DEFINITION

Decision Threshold

A decision threshold is the probability above which a scored case is treated as positive — here, the churn probability above which a customer receives the retention treatment. The threshold is not a property of the model: it is a policy choice that allocates the model’s inevitable errors between false positives and false negatives, and it is derived from the relative costs of the two errors, not from statistical convention. The software default of 0.5 embodies the assumption that the two errors cost the same and inherits its authority from nothing else.

Source: Adapted from Provost and Fawcett (2013) and scikit-learn developers (2026b).

The derivation replaces the default with the program’s own numbers, using the expected-value framing Section 8.10 introduced at concept level. Treating a customer costs money with certainty and produces benefit only probabilistically, so the treatment is worth extending exactly when the expected benefit clears the cost. Assemble the Comeback Edit’s ledger. The cost side is finance’s blended figure: twelve dollars per treated customer, redemption and margin give-up included. The benefit side requires two estimates that the analyst must obtain, document, and flag as estimates: the probability that a treated would-churner is actually saved — the save rate, which finance and CRM place at roughly one in four based on past win-back performance — and the value of a save, the retained customer’s expected future margin, which finance carries at roughly $160 over the planning horizon. A treated true churner is therefore worth about 0.25 × $160 = $40 in expectation; a treated loyalist returns nothing the program can claim (she was staying) and costs the full twelve dollars. The treatment rule follows in one line: treat a customer whose churn probability is p when the expected benefit p × $40 exceeds the certain cost of $12 — that is, when p exceeds 12 ÷ 40 = 0.30. The threshold is 0.30, not 0.5, and the difference is not a technical adjustment: at 0.5 the program treats only the customers the model is sure about, while at 0.30 it deliberately accepts more wasted vouchers because the error the vouchers prevent is more than three times as expensive as the vouchers themselves. In other words — and this is the sentence the deliverable prints under the derivation — the threshold is where the program chooses its mix of mistakes, and the choice belongs to the people who pay for the mistakes, made once, in writing, in arithmetic a CFO can re-run on a napkin.

The derivation’s honesty depends on four caveats, each a standing obligation. The first is calibration, and it is load-bearing rather than technical: the analytically derived threshold of 0.30 is valid only when the probabilities are sufficiently calibrated for the deployment population, because 12 ÷ 40 answers the question “at what genuine risk does the expected benefit clear the cost,” and a score that is not a genuine risk cannot answer it. The deliverable therefore carries the training-only reliability exhibit beside the derivation, and where the exhibit shows material miscalibration, a training-only calibration procedure is applied before the threshold is used — never a correction chosen by watching the test set. The second is estimate uncertainty: the threshold inherits the uncertainty of its inputs, so the deliverable shows the sensitivity (at a $30 benefit the cut moves to 0.40; at $60, to 0.20), because a threshold that swings widely under plausible input changes is telling the program that its economics, not its statistics, are the binding uncertainty. The third is causal: the save rate smuggles in a claim that the offer changes some would-churners’ behavior, which this chapter’s observational machinery cannot certify, per the discipline of Section 7.2; the deliverable names the randomized holdout as the purchase that would certify it. The fourth is heterogeneity: the arithmetic assumed every customer share one benefit figure, when Chapter 5’s concentration findings guarantee they do not — a refinement (customer-specific benefit, hence customer-specific thresholds) the exercises explore and real programs eventually adopt.

One tension remains between this section and the budget, and resolving it is the deliverable’s final piece of threshold work. The economic threshold answers “whom is it worth treating”; the budget answers “how many treatments we can afford,” and nothing forces the two answers to agree. If the 0.30 cut marks 2,600 customers as worth treating and the November budget covers 1,600, the analyst faces a fork with two honest tines: cut by rank — treat the 1,600 highest scores, the capacity logic of Section 8.10’s rank-and-cut, accepting that economically worthwhile treatments go unfunded — or take the gap to the meeting as a priced argument, because the lift machinery of the next section can state, in dollars, what the unfunded thousand treatments are expected to return. That argument — the model as a budget case, not just a list — is where this chapter’s analyst does the most valuable work, and it is built from the exhibits the next section constructs.

CONCEPT

The Role of the Threshold

Every other number in the pipeline belongs to the analyst: the label, the split, the features, the leaderboard. The threshold belongs to the business, because it is nothing but a statement of relative prices — how many wasted treatments one missed save is worth — translated into a probability. The analyst’s job is not to choose it; it is to surface it: to drag the cost assumption out of the software default, price the two errors with the people who own their budgets, derive the cut as arithmetic, show its sensitivity, evidence the calibration that entitles it to be applied, and record the decision.

An analyst who does this once changes how the room thinks about every scored program it funds afterward, because the question “what threshold, and from what costs?” — once heard — is impossible to unhear. This is the chapter’s verification theme in its managerial form: any threshold you are handed, including your own software’s, must be re-derivable from stated costs, and a threshold that cannot be is a cost assumption nobody agreed to.

Source: Course concept developed for this guide, informed by Provost and Fawcett (2013).

9.10 Propensity, Deciles, and the Lift and Gains Exhibits

The threshold converts scores to verdicts one customer at a time; this section builds the exhibits that convert the whole scored file into the meeting’s language, and it starts by giving the score its industry name and disambiguating it. A behavioral propensity score is a model-estimated probability that a customer will exhibit a behavior — churn, respond, convert, upgrade — used to rank customers for differential treatment. The term is worth owning because it is the form in which this chapter’s machinery is packaged everywhere in the marketing stack: every CDP “likelihood to buy,” every CRM “response score,” every retention flag is a propensity score with a wardrobe, and the questions this chapter has equipped — what label, what window, what leakage audit, what calibration, what threshold, against what baseline — are the questions that audit them all.

The disambiguation matters because the phrase has a second, established meaning that Chapter 11 will need. In causal inference, a propensity score is the estimated probability of receiving a treatment conditional on covariates, used to adjust comparisons between treated and untreated groups. This chapter uses behavioral propensity score throughout to mean a predicted probability of a marketing behavior. It is not the treatment-assignment propensity score of causal inference, and the two should never be substituted for one another in a sentence: one ranks customers for an offer, and the other tries to make an observational comparison behave more like an experiment.

DEFINITION

Behavioral Propensity Score and Ranked Targeting

A behavioral propensity score is a model-estimated probability of a defined customer behavior, used to rank customers for treatment. Ranked targeting treats customers in descending score order — to a threshold, a budget, or a capacity — so that the program’s yield depends on the ordering’s quality rather than on any individual score being exactly right. Its standard evaluation exhibits are the decile table, the lift chart, and the gains chart, all built by sorting the evaluated file by score and comparing capture against the base rate. The term is distinct from the treatment-assignment propensity score of causal inference, which estimates the probability of receiving a treatment and is introduced in Chapter 11.

Source: Adapted from Provost and Fawcett (2013) and Neslin et al. (2006).

The exhibits are built by one mechanical act — sort the evaluated customers by score, highest first, and slice the sorted file into ten equal deciles — followed by bookkeeping. For each decile, count the actual churners it contains and compute two quantities. Lift is the decile’s churn concentration relative to the base rate: if the top decile’s customers churn at 2.4 times the overall rate, the model’s best ten percent is 2.4 times richer in the target class than a randomly chosen ten percent — a number with a built-in baseline, which is why it survives in industry vocabulary. Cumulative gains is the running capture: treating the top decile reaches what share of all churners; the top two deciles, what share; and so on to the whole file, where capture reaches one hundred percent. Plotted, the gains curve rises steeply while the model’s ordering is finding churners and flattens as the remaining file thins out, against the diagonal that random targeting would trace — the visual sibling of the ROC curve, redrawn in the units a budget meeting thinks in how much of the problem do we reach for how much of the file.

The exhibits earn their central place because they price program designs directly, and the reader has seen this move before it is the capture head-to-head of Section 8.10, matured into a full curve. If the top two deciles of StyleCraft’s scored file capture on the order of half of all churners — the designed expectation for a model near the data’s quality ceiling, which the lab will test — then the “treat the top two deciles” program reaches roughly half the at-risk margin for a fifth of the blanket program’s cost, and the comparison against the alternatives is finally on one page in one currency. The ninety-day rule occupies a fixed point on the same axes (its treated list is one particular slice of the file, with its own capture), the blanket occupies the far corner (all cost, all capture), and the marginal logic of where to stop is read straight off the curve’s slope: keep funding deciles while the incremental capture in a decile is worth more than the decile costs to treat — the threshold economics of Section 9.9 re-expressed as a stopping rule on a chart, which is how it most often survives contact with a budget meeting. The decile table, the lift chart, and the gains chart are, for ranked-treatment programs, what the leaderboard was for model choice in Chapter 8: the fixed exhibit family within which every candidate — model, rule, and blanket — competes on the same field, and this chapter’s deliverable closes on them.

One reading discipline attaches to the decile table, because the draft version of this sentence is easy to overstate. Lift should generally be strongest in the early deciles and weaken as the list deepens; that is what a model with real ordering skill does. Perfect monotonic decline is not guaranteed in a finite evaluation sample, and a non-monotone middle is a finding about where the ordering blurs rather than an error to hunt down. What would be an error is a top decile whose lift sits near 1.0, because that says the model’s best ten percent is no richer in churners than a random ten percent — the ranking has failed at exactly the end of the file the program funds.

9.11 AI as a Classification Assistant

The division of labor this guide has developed across three chapters — the assistant drafts mechanics, the analyst audits meaning — reaches its most consequential form here, because classification is where an assistant’s unexamined defaults stop being statistical choices and start being business policy. A current assistant, handed the feature table and “build a churn model,” will construct the label, fit three model families, tune them, print a confusion matrix, and report accuracy — and in that single fluent response it will have chosen a churn definition (undisclosed), a threshold (0.5, unmentioned), and a headline metric (the one this chapter spent Section 9.6 dethroning), while narrating the result with congratulations. None of these is a bug; each is a default standing in for a decision nobody was asked to make. The failure modes deserve their names, because the audit targets them.

The accuracy celebration: the assistant reports “91 percent accuracy” without the base rate or the majority-class baseline, and under imbalance the margin over the do-nothing rule is left un-computed — the narration inherits the internet’s enthusiasm for accuracy exactly as Chapter 8’s assistants inherited its enthusiasm for R². The silent threshold: the confusion matrix arrives cut at 0.5, no costs discussed, and every downstream number — precision, recall, the treated-list size — quietly inherits the unexamined default; asked for “the model’s precision,” the assistant answers as if the threshold were a fact of nature. The label improvisation: asked for a churn model without a definition, the assistant invents one — often “no purchase in the last N days” computed at the analysis date, which builds the label out of the feature window and manufactures the outcome-window contamination of Table 8.2 in a single step; the resulting AUC is spectacular, and spectacularly meaningless. The test-set race: three models fitted and compared on test AUC, then a depth or a k chosen by watching the same test numbers move — the sealed set spent as a development instrument, which is Section 8.5’s violation in its classification uniform. The rebalancing reflex: prompted with “imbalanced data,” many assistants reach for resampling by default, distorting the probability scale that Section 9.9’s threshold arithmetic consumes — a repair applied to a problem the evaluation should have absorbed. The uncalibrated economics: expected-value tables computed from raw scores with no reliability exhibit anywhere, so the dollars are denominated in probabilities nobody checked. And the verdict language: scores narrated as facts — “the model identified 412 customers who will churn” — the probability-to-verdict collapse of Section 9.2, performed by a sentence.

Where the assistant genuinely helps, use it deliberately, and the list is long because the mechanics are real work: the label-construction code from a written definition (the definition supplied, never delegated); the split, the stratified fold generator, the scalers inside the training folds, and the three-model scaffold from the declared frame; the confusion-matrix, reliability, decile, lift, and gains boilerplate, which is tedious and perfectly specifiable; error analysis — “which actual churners did the model score lowest, and what do they share?” surfaces the lumpy-cadence miss faster than any manual pass; and threshold sensitivity tables from stated cost ranges, which is exactly the well-specified arithmetic assistants execute reliably. The governing instrument extends Chapter 8’s audit rather than replacing it: the five points of Table 8.5 run unchanged — frame, leakage, split hygiene, baseline, error in decision units — and Table 9.5 adds the four checks that classification makes necessary. Together they are this chapter’s verification theme made procedural: the confusion matrix is re-added by hand, and the threshold is re-derived from stated costs, before any result is believed or repeated.

Table 9.5
The classification supplement to the five-point audit (run with Table 8.5)

AI IN PRACTICE

The Classifier That Must Survive Its Audit

The two-prompt pattern of Sections 7.11 and 8.11, with the supplement between. Prompt one, the frame as specification: paste the churn definition of Section 9.3 verbatim — snapshot, windows, activity rule, both eligibility conditions — plus the feature list with derivation rules and the snapshot-only instruction, the split protocol with seed and stratification, the declared selection protocol (stratified five-fold cross-validation inside the training customers), and the baselines named (a constant-score floor, a recency-only logistic baseline, and the ninety-day rule evaluated as a fixed policy). Then give the instruction that ends the improvisation: “Build exactly this pipeline. Select the model family and every setting using stratified five-fold cross-validation on the training customers only. Report one leaderboard graded on cross-validated AUC alone — no dollar figures beside raw scores. Then calibrate only the selected model, inside the training data, and report its out-of-fold reliability table and its cross-validated expected net value at the stated threshold. Do not touch the test set. Output scores, not verdicts. Choose no threshold yourself. Narrate nothing yet.”

Then run Table 8.5’s five points, followed by Table 9.5’s four: confirm the label came back as defined and was built from purchase activity rather than net revenue; trace three features across the snapshot and run the recency range check of Section 8.12; confirm the scalers lived inside the folds and no test customer was consulted; read the cross-validated AUCs against the base’s documented volatility, treating anything implausibly high as a leak to trace rather than a result to enjoy; check that the reliability exhibit describes the model that was actually selected and not some other candidate, and that every dollar figure in the reply descends from those calibrated probabilities; then — and only then — open the test set once, apply the threshold you derived in the Section 9.9 arithmetic, produce the confusion matrix, and re-add its four cells by hand against the test-set count. Prompt two, after survival: “Draft the recommendation paragraph for a marketing VP: the model’s capture in the top two deciles against the ninety-day rule’s, the treated-list size at the 0.30 threshold with the cost assumptions stated, and no sentence that asserts an individual customer will churn.” Audit the draft’s verbs per Section 7.2 — propensity language earns “is likely to,” never “will” — and sign the scored list as the analyst of record, threshold and all.

Source: Course concept developed for this guide, informed by scikit-learn developers (2026c) and Provost and Fawcett (2013).

9.12 Marketing Interpretation and Managerial Insight

The lab’s outputs are matrices, curves, and a derivation; the twenty-eight-day deadline runs on sentences. This section translates, and — per this guide’s standing practice — it does so partly by exhibiting the wrong managerial readings and correcting them, because classification mints misreadings faster than any machinery yet: its numbers sound like everyday words (accurate, precise), its scores sound like verdicts, and both will be spoken in the targeting meeting by people whose only error is trusting the vocabulary.

The first wrong reading arrives holding the platform’s documentation: “Their churn flag is 89 percent accurate. Our model’s accuracy at the threshold you chose is about the same — and theirs is one click. Why did we build anything?” The correction is not that 89 percent is bad; it is that 89 percent is not yet a number anyone can read. At a base rate near thirty percent, seventy percent accuracy is available for free by treating no one, so the claim’s real content is a margin over that floor — a margin that cannot be evaluated without the confusion matrix behind it, the threshold that produced it, and the cost structure that prices the two errors it distributes. And the claim is unauditable besides, because 89 percent accurate at what definition of churn, at what threshold, on whose customers are questions the panel’s documentation does not answer. Then the reframe the meeting needs accuracy was never the currency. The deliverable’s currency is capture against cost — the model’s top two deciles reach a stated share of the departing customers at a stated program cost, the platform flag’s list (once exported and graded on the same test months) reaches its own share at its own cost, and the two can compete on one gains chart the moment the vendor discloses enough to be graded. The platform verdict is Chapter 8’s vendor verdict with this chapter’s supplement: the five questions of Table 8.5, plus Table 9.5’s four — and the one that ends most such meetings is S3, because a flag that cannot state its threshold’s cost assumption is a business policy no one at StyleCraft agreed to.

The second wrong reading is quieter and does more damage per occurrence: “The model says C003914 has a 0.62 churn probability — she’s leaving. Why is she still getting the full-price catalog? And C005108 is at 0.08, so we can stop worrying about him.” Both sentences collapse a probability into a verdict, and Section 8.13’s discipline transfers with the units changed: 0.62 means that among snapshot-similar customers, roughly six in ten went dark — and four in ten did not. The score is not a prophecy about her; it is a rank and a rate, and the program built on it is justified by the portfolio — treat enough 0.62s and the arithmetic of Section 9.9 pays reliably — not by any individual call being right. The corrected sentences change the object of confidence exactly as Chapter 8’s did: not “she is leaving” but “she belongs to the group most worth treating”; not “he is safe” but “treating his group costs more than it returns.” One implication deserves its own sentence in the deliverable, because it defuses the meeting’s most predictable ambush: under a 0.30 threshold, the majority of treated customers will not have churned even if the model is excellent — precision below one half means most vouchers land on stayers by design, because the program deliberately overspends on the cheap error to avoid the dear one. An executive who has not been told this in advance will read the first post-campaign report as proof the model failed; an executive who has been told will read the same report as the cost structure working as priced. Setting that expectation before launch is not spin — it is the threshold decision’s second half and forgetting it has ended more scoring programs than bad models have.

A third misreading is newer to this chapter and specific to its arithmetic: “The model says 0.30 is the cut, so let’s use 0.30.” That sentence is correct only if the scores near 0.30 mean thirty percent, which is a property the reliability exhibit of Code 9.12 either establishes or refutes. Where it refutes, the threshold is not wrong — the ledger is still 12 against 40 — but the scores are the wrong ruler for it, and the repair belongs in the training data rather than in the meeting. This is the clearest case in the chapter where a number that survives every ranking diagnostic can still price a decision incorrectly, and it is the reason the deliverable carries the calibration exhibit rather than merely mentioning calibration in a footnote.

The deliverable that survives all three misreadings has a fixed anatomy, assembled entirely from lab outputs. Page one is the churn definition per Section 9.3 — the label, its activity basis, its two eligibility conditions, its rejected alternatives, and who each alternative would have added or removed — because every later number is downstream of it. The training-only leaderboard follows, per Section 8.9’s fixed rules: three models against the constant-score floor and the recency-only logistic baseline, graded on cross-validated AUC alone, with the base rate printed in the header, the paired fold margins shown, and no dollar figure anywhere on it. Then the selection rule, executed unedited, and the calibration of the selected model inside the training data. Then the threshold derivation as arithmetic — the $12, the $40, the 0.30, the sensitivity at $30 and $60 — stated so the CFO can re-run it, with the out-of-fold reliability exhibit and the economic gate beside it: the calibrated model’s cross-validated net value per thousand customers against zero and against the incumbent rule, computed before the test set was opened. A threshold the CFO has re-derived is a threshold the CFO defends thereafter; a threshold applied to unchecked probabilities is a threshold nobody should defend; and a program that cannot clear its own gate on training evidence is a program the deliverable should decline to fund rather than test its way into. Then the single test-set grade, opened once, on the frozen estimator-plus-calibrator pair. Then the five-policy table at the derived threshold, hand-reconciled, with the precision expectation set in plain words. Then the gains chart with the three policies on one page — model deciles, ninety-day rule, blanket — and the budget sentence the stopping rule produces. The incremental caveat, flagged per Section 7.2 and priced per Section 9.9: the save rate is an assumption this design cannot certify, the experiment that would certify it is named, and the deliverable recommends the program launch with a randomly held-out control group, because the cheapest moment to buy causal evidence is before the rollout, not after the doubt. The monitoring plan per Section 8.10, with two classification-specific lines: the base rate itself is monitored, because a drifting base rate silently re-prices the threshold even when the ordering holds; and the calibration is re-checked on each matured cohort, because probabilities drift out of true before rankings do. The permitted-use note per Section 8.15. And the fairness appendix from Lab 9.2 Part G — the per-group error rates, the new-store exhibit, and its tenure-restricted comparison with the residual tenure imbalance reported, framed as a minimum diagnostic screen — which Section 9.15 now takes up on its own terms.

9.13 Business Analytics in Practice

This section turns from the fictional case to how churn scoring, retention economics, and classification governance operate in industry — where churn models are among the oldest production analytics in marketing, where the lift chart is a budget instrument, and where the hardest lessons are about what the model made people do. The vignettes below are drawn from published research and from recurring professional patterns rather than from any single named organization.

9.13.1 The Telecom Churn Playbook and Two Different Warnings

The first vignette is the telecom churn playbook, because telecommunications are where churn modeling grew up and where its most instructive surprises were documented. Contractual carriers have modeled churn for three decades: rich behavioral data, a legally crisp label (the cancellation event this chapter’s non-contractual setting had to construct by hand), dedicated save desks, and retention offers whose economics are exactly Section 9.9’s ledger at industrial scale. The playbook’s mature form looks like this chapter’s deliverable — scored base, threshold from offer economics, treatment tiers by decile — and it comes with two distinct pieces of scar tissue that students routinely fuse into one and should not.

The first warning is about targeting. Ascarza (2018), combining two field experiments with machine-learning targeting rules, found that the customers a churn model ranks riskiest are not necessarily the customers most responsive to a retention intervention, so programs aimed by risk rather than by sensitivity to the treatment can misallocate their budgets — and that targeting on estimated response heterogeneity outperformed the standard practice of targeting the highest-risk customers. In this guide’s vocabulary: risk is who is leaving; responsiveness is whom the treatment changes; they are different quantities, and this chapter’s observational machinery estimates only the first.

The second warning is stronger and comes from a different study, which is why the two should not be cited interchangeably. Ascarza et al. (2016), in a large-scale field experiment in which some customers received proactive pricing-plan recommendations and some did not, found that the intervention increased churn: 10 percent of the treated group churned in the three months following the campaign against 6 percent of the control group, with the authors attributing the effect to lowered inertia about switching and to raised salience of past usage. That is not a modeling error at all — the churn model predicted honestly; the program assumed that predicting and preventing are the same problem. Only a held-out control inside the campaign can measure the second, which is Chapter 11’s machinery. The practice lesson, planted in Section 9.13’s deliverable and repeated here because industry paid heavily to learn it: launch scored programs with controls from day one, and treat “intervention is not prediction” as a design constraint rather than a footnote.

9.13.2 The Lift Chart as a Budget Instrument

The second vignette is the lift chart as a budget instrument, because the exhibit students meet as a model-evaluation chart functions in industry as a finance document. At subscription and retail firms with mature retention programs, the annual retention budget is negotiated off the gains curve: the analytics team presents cumulative capture by decile, finance attaches the treatment cost and save economics — the same ledger as Section 9.9, at line-item scale — and the budget conversation becomes a stopping-rule conversation: fund deciles while the marginal decile’s expected saved margin clears its treatment cost, stop where the curve’s slope says to stop. Teams that operate this way report the cultural shift this chapter’s Concept box predicted: the argument stops being “is the model good” — a question finance cannot engage — and becomes “where does the curve stop paying,” a question finance is better at than the analysts.

The recurring failure mode is equally instructive: programs that set the treated-list size first (a round number, a platform tier limit, last year’s list) and never reconcile it against the curve, so that the budget quietly funds deciles the arithmetic abandoned — the capacity-versus-threshold tension of Section 9.9, resolved by inertia instead of by the priced argument. A second failure is subtler and belongs to this chapter’s calibration thread: teams that read the gains curve correctly and then compute the program’s expected return from raw model scores that nobody checked for calibration, producing a budget denominated in probabilities the model does not actually print. The lesson is the closing thread’s preview: the decile table is where analytics and finance speak the same language, and the analyst who brings the curve — and the reliability exhibit that entitles its dollar figures — to the budget meeting sets the agenda.

9.13.3 When a Marketing Score Crosses a Regulatory Line

The third vignette is the regulatory boundary, because propensity machinery does not stay in marketing. The same scored file that ranks customers for retention vouchers can rank them for credit-like treatments — preapproved financing offers, deposit requirements, payment-plan eligibility, service-tier routing — and the moment it does, it leaves the domain of marketing judgment and enters one governed by statute, supervision, and litigation.

The practice lesson here is a boundary discipline rather than a legal conclusion, and the reason is that the legal ground genuinely moves. In the United States, the federal treatment of disparate impact in credit was substantially revised in 2026: the Consumer Financial Protection Bureau published a final rule amending Regulation B, effective July 21, 2026, stating that the Equal Credit Opportunity Act does not authorize disparate-impact liability and removing the effects test from the regulation (Consumer Financial Protection Bureau, 2026). Commentators note that disparate-impact theories may still operate under other federal statutes and under state fair-lending law, and litigation over the rule’s scope was active at the time of writing. This chapter therefore declines to state a general rule and states an obligation instead: credit-adjacent uses of customer scores can trigger federal and state fair-lending, discrimination, adverse-action, explanation, and consumer-protection requirements; the governing rules are jurisdiction- and use-specific and change; and they should be reviewed with legal and compliance specialists before deployment rather than reasoned out from a textbook paragraph.

What does not change is the mechanism the analyst is uniquely positioned to see, and it is the one Barocas and Selbst (2016) mapped and Section 6.16 previewed for segmentation: neutral-looking features such as geography, tenure, or shopping channel can carry protected characteristics by correlation, so a model that never sees a protected attribute can still distribute its errors unequally across groups defined by one. Mature organizations institutionalize the boundary exactly the way this guide’s permitted-use note does: the score’s approved uses are written down, credit-adjacent repurposing triggers legal review and formal fairness testing across groups, and the analyst of record is the person expected to notice when a marketing score is drifting toward a regulated decision. The lesson: the fairness screen of Lab 9.2 Part G is not an academic garnish — it is the junior version of an audit that, in adjacent industries, is performed under subpoena.

9.13.4 In Your First Analyst Job

In your first analyst job, these vignettes compress into one expectation, and it is this chapter’s closing thread: threshold-setting is where the analyst meets the CFO. The model will be a commodity — every platform ships one, every assistant drafts one — and the meetings that matter will be about the numbers around it: what the label means, what the two errors cost, whether the probabilities mean what they say, where the curve stops paying, what the program may and may not be used for, and whether anyone built the control group that can prove the treatment works. Every one of those is a conversation this chapter equips, and none of them is a modeling conversation. The analysts who advance are the ones who can hold the room through the cost arithmetic — because the threshold is the one number in the pipeline that the CFO, once shown, will insist on owning, and the analyst who taught the CFO to own it becomes the person the CFO calls before every scored program thereafter.

9.14 Ethics, Fairness Across Segments and the Loops Models Close

The Business Analytics in Practice section ended at a legal boundary; this section examines the ethical ground on both sides of it, extending the guide’s running discussions — data use (Section 1.13), problem framing (Section 2.12), measurement design (Section 3.13), cleaning as editorial power (Section 4.14), honest summarization (Section 5.14), differential treatment of segments (Section 6.16), causal language (Section 7.15), and acting on predictions about people (Section 8.15) — to what classification adds: verdicts about individuals, sorted into groups, at scale, by machinery whose errors are invisible one customer at a time. Two obligations are new here, and the dataset was designed to make the first one unmissable.

The first obligation is fairness across segments, and its instrument is one this chapter already built: the confusion matrix, computed by group. A model’s aggregate metrics — the leaderboard AUC, the overall precision and recall — are averages across the file, and averages hide people; that was Chapter 5’s ethics, and it returns here with teeth, because a classifier can post excellent aggregate numbers while distributing its errors unequally across identifiable groups: flagging one group’s loyal customers as churners at twice the rate it does another’s or missing one group’s departing customers systematically. The new-store exhibit of Lab 9.2 Part G is the designed demonstration — the second strike of the dataset’s ethics trap, and the sharper one. The first strike, in Chapter 6, was about deliberate differential treatment of known segments; this strike is about accidental differential judgment of a group nobody defined: customers of the newest stores, whose short tenure and thin history are artifacts of their stores’ opening dates and StyleCraft’s own under-marketing during ramp-up, and whom a naive churn model therefore scores as high-risk for reasons that describe the company’s history, not the customers’ intentions. A retention program acting on the raw scores would flood the newest markets with discount vouchers — training precisely the wrong price expectations in the expansion’s youngest relationships — and, worse, the same scores repurposed upstream (“the new markets show terrible, predicted retention”) would argue for abandoning stores whose only sin is being new. The dataset’s designers built the trap so that the correction is achievable with this guide’s own tools: the tenure-restricted comparison of Code 9.18, the store-cohort controls the exercises rehearse, and the standing rule that the deliverable’s fairness appendix reports error rates by segment, by metro, and by store cohort before any scored program launches. The disparate-impact frame of Barocas and Selbst (2016), cited since Section 1.13, names the general mechanism: models trained on data shaped by past decisions re-encode those decisions as predictions, and the analyst who does not look for the re-encoding has, by default, endorsed it.

The second obligation has no descriptive-analytics ancestor, because it belongs to models that act: the feedback loop, in which the model helps cause the outcome it predicts, and its own future training data records the episode as vindication. The mechanisms are concrete and none requires malice. A customer flagged as likely to churn is routed to reduced marketing investment — why spend on the leaving? — and, starved of contact, drifts away: the prediction manufactured its own confirmation, and next year’s training data files her under “correctly predicted churner,” making next year’s model more confident in the judgment that caused the loss (O’Neil, 2016). The retention offer itself loops in the other direction: treated customers who stay are recorded as stayers, so the treatment contaminates the label, and a model retrained naively on post-program data learns that high-risk profiles stay — because the program kept saving them — degrading exactly where the program worked; the repair, at concept level, is to train on untreated customers or on the control group the Section 9.13 deliverable insisted on, and the full machinery is one more argument for randomized holdouts as standing infrastructure rather than occasional experiments. Even the churn-increasing intervention of Section 9.14 is a feedback loop in miniature: the model’s list determined who received the contact that changed the outcome the model was predicting. The general statement deserves the chapter’s last box, because it is the deepest way this chapter’s subject differs from every one before it: a descriptive summary leaves the world as it found it; a deployed classifier is an intervention wearing an observation’s clothes, and the analyst who deploys one inherits a monitoring obligation — not just “is the model still accurate” (Section 8.10’s decay) but “is the model changing the population it scores,” which no accuracy metric detects.

9.15 Chapter Summary

This chapter completed the predictive machinery Part II has been assembling, by carrying it across the line from how much to whether. The frame, the label wall, the leakage audit, the training-only selection protocol, the sealed test set, and the fixed leaderboard all transferred from Chapter 8 intact; what changed was the geometry of being wrong. A binary decision fails in two directions with two prices, and the chapter’s whole architecture followed from refusing to average those prices: the confusion matrix kept the four outcomes separate and named them in program vocabulary, with its cost column priced against the program’s own ledger rather than against the full margin of a lost customer; precision and recall gave the two stakeholders their two questions; accuracy was demoted to a number reportable only beside its base rate and the do-nothing floor, with the margin between them computed rather than implied; AUC graded the ordering the scorers produce, on its full zero-to-one range, with ties counted as half and with its silences stated — and was assigned the job it can do, choosing a model family, rather than the job it cannot, pricing a program; and calibration was promoted from a footnote to a gate, because the chapter’s destination multiplies probabilities by dollars and a well-ordered score can still be the wrong ruler.

The chapter’s destination redeemed a seven-chapter-old promise: the asymmetric cost of being wrong, introduced in Chapter 2 as interpretive discipline and previewed in Chapter 8 as expected-value framing, became the decision threshold — twelve dollars against forty, a cut at 0.30 instead of the default’s 0.5, derived as arithmetic a CFO can re-run, shown with the sensitivity that reveals whether the model or the economics is the binding uncertainty, and applied only after a reliability exhibit established that the probabilities could bear it. Around that center, the chapter built the working instruments of scored marketing: the churn label as a measurement decision with a policy inside it, defined from purchase activity rather than from net revenue and carrying both of Chapter 8’s eligibility conditions; logistic regression reading risk in odds, with trees and neighbors as structurally different second opinions whose flexibility was chosen inside the training folds rather than declared; and the behavioral propensity exhibits — deciles, lift, gains — that turn a scored file into a budget argument with a stopping rule, carefully distinguished from the treatment-assignment propensity score Chapter 11 will need.

The AI section extended the five-point audit with the classification supplement — base rate and floor, matrix arithmetic by hand, threshold provenance, calibration and selection hygiene — and the labs ran the whole discipline twice: once at hand scale on ten customers where three policies tied at exactly 0.70 accuracy while netting nothing, thirty-two dollars, and forty-eight dollars — and where the incumbent rule, tied with none of them, lost eight — and once at full scale across eighteen numbered cells, where flexibility was chosen by a one-standard-error rule, the family was chosen on ranking evidence alone, the selected model was calibrated inside the training data and made to clear an economic gate before any money was discussed, and the deliberate failures were manufactured and caught without ever spending the test set on a diagnostic. The interpretation, practice, and ethics sections carried the scores into the rooms that matter: the platform flag met the supplement’s questions, the probability was defended from becoming a verdict, the threshold was defended from being applied to unchecked scores, industry’s own scar tissue was read at market prices — with the churn tournament’s actual finding restored, the two Ascarza results separated, and the 2026 fair-lending change treated as a moving legal boundary rather than a settled rule — and the deliverable acquired its two standing appendices: the fairness screen that catches a model judging customers for their stores’ age, and the control group that catches a model causing what it predicts.

Looking ahead, the machinery now predicts individuals — their spend, their departure — from a snapshot of their history. The next family of marketing questions abandons the individual for the aggregate and the snapshot for the flow of time itself: what will revenue be in December, how will the new stores’ ramp reshape the season, what should the buy plan assume about demand that has not happened yet? Time changes the rules more deeply than the target type did. Observations are no longer exchangeable customers but ordered days, the random splits this Part has relied on would let the future leak into the past wholesale, and the baselines with teeth stop being spreadsheets and become the calendar’s own rhymes — tomorrow resembles today, December resembles last December. The next chapter takes up forecasting demand, sales, and campaign performance: trend, seasonality, and the discipline of predicting a future the model must be graded against honestly — including the one event StyleCraft’s own history planted in the series, the expansion’s inflection, which any forecast worth funding has to see coming.

9.16 Exercises for Practice

The following exercises practice the chapter’s main habits: define the label in writing before touching data, carry the base rate into every exhibit, verify confusion-matrix arithmetic by hand, choose model settings inside the training folds, check calibration before computing dollars, derive thresholds from stated costs rather than inheriting them, grade rankings with lift and gains against real incumbents, audit AI-built classifiers with the supplement, and check every scored program’s behavior across customer groups.

9.16.1 Core Chapter PracticeExercise 9.1 Concept Check

Answer each in two or three sentences, in your own words.

• Distinguish classification from class-probability estimation and explain why this guide insists the threshold that converts one into the other is not a model property.
• Explain why churn must be defined rather than observed in a non-contractual business and name the three definitional levers of Table 9.1 with one consequence each.
• The chapter builds the churn label from counted purchase orders rather than from zero net revenue. Give two concrete ways the two definitions can disagree, and state which one the deliverable should report.
• State both conditions of this chapter’s eligibility rule and explain what breaks in the feature table if the second condition is dropped.
• A logistic coefficient on discount_share is 1.1, exponentiating to about 3.0. Write the honest plain-language sentence, including the ceteris paribus clause and a verb that survives Section 7.2.
• Why does k-NN require the scaling discipline of Section 6.7 when logistic regression does not require it for validity?
• A churn model posts 88 percent accuracy on a file whose base rate is 9 percent. Compute the majority-class baseline, state the margin, and state what the comparison does and does not establish.
• State precision’s question and recall’s question in Comeback Edit vocabulary, name the stakeholder who owns each, and explain why the threshold moves them in opposite directions.
• AUC is 0.77. Write the one-sentence probabilistic interpretation including the treatment of ties, state the full range AUC can occupy and what a value below 0.5 would mean, and name two things this number does not certify.
• Define calibration, explain how it differs from discrimination, and state precisely which step of the Section 9.9 derivation fails when scores are well ordered but systematically too high.
• An analyst inherits a confusion matrix cut at 0.5 and is told “that’s just the standard.” State the cost assumption the default embodies, name the two further conditions that ride inside the equal-cost reading, and show with this chapter’s ledger what threshold the program’s own economics imply.
• Distinguish the behavioral propensity score of this chapter from the treatment-assignment propensity score of causal inference and give one sentence on why confusing them would matter in Chapter 11.

Exercise 9.2 Confusion-Matrix and Policy Arithmetic

Using only the ten-customer miniature of Lab 9.1 Part A (no code): rebuild both confusion matrices — thresholds 0.50 and 0.30 — customer by customer, both accuracies, both precisions, both recalls, and the F1 at each threshold (approximately 0.57 and 0.67; show the harmonic-mean arithmetic). Then rebuild the five-policy economics table — treat nobody, the 0.50 cut, the 0.30 cut, the ninety-day rule, and the blanket — and verify the nets of $0, +$32, +$48, −$8, and $0. State explicitly what your table does with precision in the do-nothing row and why that is not zero. Identify the single customer whose reclassification between the two thresholds contributes most to the net improvement and write one sentence on what her profile says about which customers live near a threshold.

Exercise 9.3 Derive the Threshold

For each program below, derive the economic threshold from the stated costs, state the treatment rule in one sentence, and note anything the arithmetic reveals about the program’s design. Close each with one sentence on what calibration evidence you would require before applying your threshold to a model’s raw scores.

A win-back email costs $3 per treated customer once redemptions and margin give-up are blended, against a $30 expected benefit per treated true churner. Derive the threshold and explain what a very low threshold implies about who should receive the email — and what Section 9.14’s first vignette warns even “nearly free” treatments can cost.

A high-touch save program — a personal stylist calls plus a $40 credit — costs $55 per treated customer, against an expected benefit of $110 per treated true churner. Derive the threshold and state, using the gains-chart logic of Section 9.10, why this program and the email program should not share a treated list.

A program’s benefit estimate is uncertain: finance brackets it between $24 and $72, against a $12 treatment cost. Derive the threshold at both ends and write two sentences on what the range means for the deliverable, per Section 9.9’s sensitivity discipline.

Exercise 9.4 Spot the Failure

Each scenario below contains at least one failure from this chapter (or none). Name it, cite the section or table row, and state the repair.

A churn model’s features include days_since_last_purchase computed at the June 30, 2026, analysis date, for a label whose outcome window began January 1, 2026. Its test AUC is 0.96.

A team fits logistic regression, a tree, and a k-NN, compares all three on test AUC, then tries three tree depths and keeps the one with the best test AUC. Everything else in the pipeline is clean.

An assistant, told the classes are imbalanced, oversamples churners until the training data is 50/50, then computes an expected-value table at the 0.30 threshold from the resulting scores.

A deck reports precision of 0.81 and recall of 0.74 for the same model. A footnote reveals the two figures were achieved at different thresholds.

A team evaluates its churn model with a stratified 80/20 split, selects model family and settings by stratified cross-validation inside the training customers, reports test AUC beside the majority-class baseline and the incumbent rule, and prints an out-of-fold reliability table beside its threshold derivation.

A churn model’s AUC is 0.79 and its scores near the economic threshold run about fifteen percentage points above the observed churn rate in those bins. The team applies the 0.30 cut anyway on the grounds that AUC is strong.

A retention program treats the top decile, and the following year’s retrained model shows the old top decile’s profiles now churn at nearly the base rate. The team concludes the original model was wrong.

A model’s aggregate recall is 0.72, but recall computed within the resort-metro cohort is 0.31. The deck reports only the aggregate.

Exercise 9.5 The Full Churn Deliverable

Complete Labs 9.1 and 9.2 on the certified files and assemble the targeting deliverable per Section 9.13: the label definition page with its activity basis, both eligibility conditions, and the rejected alternatives; the training-only leaderboard with three models, the constant-score floor, and the continuous-recency baseline on cross-validated AUC and expected net value, with the base rate stated and the paired fold margins shown; the declared selection rule and the single sealed-test grade; the threshold derivation with its sensitivity table and its out-of-fold reliability exhibit; the hand-reconciled five-policy table at the derived threshold, including the ninety-day rule and the blanket, with the precision-expectation paragraph for the executive reader; the ROC, lift, and gains figures with the incumbent marked as an operating point; the budget sentence the stopping rule produces; the incremental caveat with the control-group recommendation; the monitoring plan including base-rate and calibration monitoring; the permitted-use note; and the fairness appendix with per-group error rates, the new-store exhibit, its tenure-restricted comparison with the residual tenure imbalance reported, and the sentence framing it as a minimum diagnostic screen rather than a certification.

Exercise 9.6 AI Classification Audit

Give an AI assistant the raw StyleCraft tables and this deliberately loose prompt: “Build me the best churn model you can — maximize accuracy.” Audit the response in writing with Table 8.5’s five points followed by Table 9.5’s four: reconstruct the churn definition the assistant invented and compare it against Section 9.3’s, naming which levers of Table 9.1 it moved and whether it built the label from activity or from revenue; trace every feature across the snapshot, running the recency range check; identify the threshold it applied and whether it disclosed applying one; compute the base rate and majority-class baseline it omitted and state the margin; determine whether any model family or setting was chosen by consulting a test split; note whether any calibration evidence accompanies any dollar figure; and re-derive its headline metrics from its own confusion matrix, reconciling the cell counts by hand. Then re-prompt with the full frame per the AI in Practice box and compare the two responses’ leaderboards. Conclude with two sentences on which supplement point caught the most serious problem.

9.17 Glossary of Terms

This glossary includes only the terms introduced in this chapter. Each definition is tied to the sources used in the chapter rather than added for decoration.

Accuracy. The share of evaluated cases whose predicted class matches the actual class; under class imbalance, dominated by the majority class and reportable only beside the base rate and the majority-class baseline, with the margin between them computed (adapted from Provost & Fawcett, 2013).

AUC (area under the ROC curve). A threshold-free summary of ranking skill ranging from 0 to 1, equal to the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative one, with ties counted as half. A value of 0.5 indicates chance-level ordering and values below 0.5 indicate a systematically reversed or worse-than-chance ordering; the measure is silent about calibration and costs (adapted from Fawcett, 2006; Google for Developers, 2026).

Base rate. The share of the positive class in the evaluated population; the context number without which no classification metric is legible, and itself a constructed quantity under a constructed label (adapted from Provost & Fawcett, 2013).

Behavioral propensity score. A model-estimated probability of a defined customer behavior, used to rank customers for differential treatment; the packaged form in which this chapter’s machinery appears throughout the marketing stack, and distinct from the treatment-assignment propensity score of causal inference (adapted from Provost & Fawcett, 2013).

Calibration. The property of a probability scorer whose predicted probabilities match observed event frequencies, so that among cases scored near p, about a share p are positive; distinct from discrimination, assessed with a reliability exhibit of mean predicted probability against observed rate by score bin, and required before an economically derived threshold may be applied to raw scores (adapted from scikit-learn developers, 2026a).

Churn label. The operationalized record of churn for a historical case: a defined activity — here, at least one completed purchase order — failing to occur within a defined outcome window after the snapshot, under stated eligibility rules; a measurement decision whose definitional levers change the model and the treated population (adapted from Neslin et al., 2006; Provost & Fawcett, 2013).

Class imbalance. The condition in which one class heavily outnumbers the other, under which accuracy is dominated by the majority class and rank-based or cost-based evaluation is required (adapted from He & Garcia, 2009).

Class-probability estimation. The prediction, for each case, of the probability of the positive class — a score that ranks cases and defers verdicts to a separately chosen threshold (adapted from Provost & Fawcett, 2013).

Classification. The prediction of a categorical target by assigning each case to a class; in this guide, produced by thresholding estimated probabilities rather than emitted directly by the model (adapted from Provost & Fawcett, 2013).

Classification tree. A model that predicts by routing cases through a learned sequence of yes/no feature questions to leaves, where predicted probabilities are the leaves’ training-class shares; grown by recursive splitting and governed by depth or leaf-size limits chosen inside the training data (adapted from Breiman et al., 1984).

Confusion matrix. The cross-tabulation of actual against predicted classes at a stated threshold, partitioning cases into true positives, false positives, false negatives, and true negatives; the arithmetic source of every threshold-dependent metric (adapted from Provost & Fawcett, 2013; Fawcett, 2006).

Decision threshold. The probability above which a scored case is treated as positive; a policy choice allocating errors between false positives and false negatives, derived from the two errors’ relative costs rather than from the software default of 0.5, and valid only where the scores are sufficiently calibrated (adapted from Provost & Fawcett, 2013; scikit-learn developers, 2026b).

F1 score. The harmonic mean of precision and recall, dragged toward the smaller of the two; a single-number summary that weights the two errors equally and is therefore provisional wherever costs are asymmetric (adapted from Provost & Fawcett, 2013).

False negative. A positive case predicted negative — here, a departing customer no one tried to save; priced as the missed expected incremental save rather than as the customer’s full future margin (adapted from Provost & Fawcett, 2013).

False positive. A negative case predicted positive — here, a retention offer spent on a customer who was staying; priced as the blended treatment cost and bought deliberately by low thresholds (adapted from Provost & Fawcett, 2013).

Gains chart. The cumulative share of all positive cases captured as treatment extends down the score-ranked file, plotted against the share of the file treated; read against the random-targeting diagonal and used with treatment economics as a budget stopping rule (adapted from Provost & Fawcett, 2013; Neslin et al., 2006).

k-nearest neighbors (k-NN). A model that predicts a case’s class probability as the positive share among its k closest training cases in scaled feature space; coefficient-free, with flexibility governed by k and k chosen by cross-validation inside the training data (adapted from Cover & Hart, 1967).

Lift. The ratio of a targeted group’s positive rate to the base rate — how many times richer in the target class a model-selected slice is than a random one; the ranked-targeting metric with its baseline built in (adapted from Provost & Fawcett, 2013).

Log-odds. The natural logarithm of the odds; the unbounded, symmetric rescaling of probability on which logistic regression’s linear equation operates (adapted from Hosmer et al., 2013).

Logistic function. The S-shaped curve that converts any log-odds value into a probability between 0 and 1 — flat near the extremes, steepest at even odds (adapted from Hosmer et al., 2013).

Logistic regression. The model that expresses the log-odds of a binary target as a linear function of features and recovers probabilities through the logistic function; its exponentiated coefficients are read as multiplicative effects on odds, ceteris paribus (adapted from Hosmer et al., 2013; James et al., 2021).

Odds. The ratio of an event’s probability to its complement’s — p ÷ (1 − p); the currency in which logistic coefficients speak after exponentiation (adapted from Hosmer et al., 2013).

Precision. The share of predicted positives that are positive — the purity of the treated list, and finance’s question; undefined rather than zero when nothing is treated (adapted from Provost & Fawcett, 2013).

Recall (true positive rate). The share of actual positives predicted positive — the coverage of the at-risk class, and the churn-number owner’s question (adapted from Provost & Fawcett, 2013; Fawcett, 2006).

ROC curve. The plot of true positive rate against false positive rate across all thresholds, tracing a scorer’s ranking trade-off independent of any single cut (adapted from Fawcett, 2006).

True negative. A negative case predicted negative — the quiet majority correctly left untreated, and the cell that inflates accuracy under imbalance (adapted from Provost & Fawcett, 2013).

True positive. A positive case predicted positive — here, a save opportunity reached (adapted from Provost & Fawcett, 2013).

9.18 References

Ascarza, E. (2018). Retention futility: Targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1), 80–98. https://doi.org/10.1509/jmr.16.0163

Ascarza, E., Iyengar, R., & Schleicher, M. (2016). The perils of proactive churn prevention using plan recommendations: Evidence from a field experiment. Journal of Marketing Research, 53(1), 46–60. https://doi.org/10.1509/jmr.13.0483

Barocas, S., & Selbst, A. D. (2016). Big data’s disparate impact. California Law Review, 104(3), 671–732. https://doi.org/10.15779/Z38BG31

Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees. Wadsworth.

Consumer Financial Protection Bureau. (2026). Equal Credit Opportunity Act (Regulation B), 12 C.F.R. pt. 1002. https://www.consumerfinance.gov/rules-policy/regulations/1002/

Cover, T. M., & Hart, P. E. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. https://doi.org/10.1109/TIT.1967.1053964

Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010

Google for Developers. (2026). Classification: ROC and AUC [Machine Learning Crash Course]. https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc

He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263–1284. https://doi.org/10.1109/TKDE.2008.239

Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (3rd ed.). Wiley. https://doi.org/10.1002/9781118548387

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning: With applications in R (2nd ed.). Springer. https://doi.org/10.1007/978-1-0716-1418-1

Neslin, S. A., Gupta, S., Kamakura, W., Lu, J., & Mason, C. H. (2006). Defection detection: Measuring and understanding the predictive accuracy of customer churn models. Journal of Marketing Research, 43(2), 204–211. https://doi.org/10.1509/jmkr.43.2.204

O’Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Crown.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

Provost, F., & Fawcett, T. (2013). Data science for business: What you need to know about data mining and data-analytic thinking. O’Reilly Media.

scikit-learn developers. (2026a). Probability calibration [Software documentation]. https://scikit-learn.org/stable/modules/calibration.html

scikit-learn developers. (2026b). Post-hoc tuning the cut-off point of decision function [Software documentation]. https://scikit-learn.org/stable/auto_examples/model_selection/plot_tuned_decision_threshold.html

scikit-learn developers. (2026c). Cross-validation: Evaluating estimator performance [Software documentation]. https://scikit-learn.org/stable/modules/cross_validation.html