Machine Learning Foundations
Executive view
Technical view
Why this matters
Enterprise AI still spends most production inference cycles on classical ML: credit scoring, fraud, churn, demand forecasting, routing, propensity, anomaly detection, and ranking. GenAI attention does not remove the need to know when a gradient-boosted tree on tabular features outperforms a prompt—and at one-thousandth the cost. Misclassified problem types produce wrong architecture: using LLMs for stable structured prediction burns latency and money; using brittle rules where ML generalises loses revenue.
Metric misuse destroys credibility. Accuracy on imbalanced fraud data is actively misleading. Optimising RMSE on revenue forecasting when finance cares about bias at peak seasons misaligns teams. Leakage—training on information unavailable at decision time—creates models that cannot be reproduced in production and triggers model risk management incidents in regulated sectors.
For an AI Solution Engineer, ML literacy is quality assurance on feasibility claims. You read eval plans, challenge holdout design, insist on baselines (rules, heuristics, prior model), and translate precision/recall trade-offs into £ and customer experience. This topic connects to topic 09 (features and pipelines), topic 11 (when LLM adds value), topic 21 (evaluation), and AI Patterns for pattern selection.
Learn
Problem types: supervised, unsupervised, and reinforcement learning
Definition. Supervised learning learns from labelled examples—input → known output. Unsupervised learning finds structure without labels (clusters, embeddings, anomalies). Reinforcement learning (RL) learns policies via reward signals from actions in an environment—common in robotics and some ad bidding; rare in typical enterprise business apps today.
Engagement use. First question on any use case: do we have reliable labels at decision time? If yes, supervised paths dominate. If labels are expensive, consider weak supervision, active learning, or proxy labels—with leakage review. Unsupervised fits exploration: customer segmentation, transaction clustering for AML alerts, embedding users for similarity. Do not call a chatbot "RL" because it gets thumbs up—most enterprise feedback loops are offline retraining, not true RL.
Pitfalls.
- Treating LLM classification as supervised without labelled eval set.
- Clustering presented as "AI segments" without stability checks across months.
- RL proposed where bandits or simple rules suffice.
- Mixing problem types in one metric—forecast accuracy on classification task.
Worked example. Telco churn: supervised binary classification—label churned_within_90d from billing data. Marketing also wants "high-value segment discovery"—unsupervised clustering on usage features, validated by whether segments differ in churn rate (supervised check on clusters).
Classification, regression, ranking, forecasting, and anomaly detection
Definition. Classification predicts discrete classes (fraud/not, default/not). Regression predicts continuous values (loss amount, LTV). Ranking orders items (search results, lead priority). Forecasting predicts future series (demand, call volume). Anomaly detection flags unusual patterns—often unsupervised or semi-supervised.
Engagement use. Map business decision to formulation. "Approve loan" → classification with calibrated probabilities. "How much reserve" → regression. "Which claims first" → ranking with SLA weights. "Staffing next week" → forecasting with seasonality. "Unknown fraud pattern" → anomaly scores with human review queue.
Pitfalls.
- Multi-class problem treated as binary without hierarchy.
- Regression on heavy-tailed targets without log transform or quantile loss.
- Ranking optimised with pointwise loss when listwise metrics matter.
- Forecasting without holiday/event features in retail and banking.
Worked example. Insurance claims: triage ranking (urgency score), reserve regression (expected cost), fraud classification (refer to SIU), catastrophe forecasting (FNOL volume)—four models, four metrics, shared feature platform—not one LLM doing all.
Labels, ground truth, and class imbalance
Definition. Labels are the target variable for supervised learning. Ground truth must reflect the decision moment—not future outcomes unknowable then. Class imbalance occurs when one class is rare (fraud, default, disease).
Engagement use. Document label definition, source system, latency, and error rate in labelling. For imbalance, use stratified splits, appropriate metrics (PR-AUC, recall at precision), and cost-sensitive thresholds—not naive accuracy. Consider oversampling/undersampling in training only with careful validation design.
Pitfalls.
- Labels derived from human overrides that reflect model errors—feedback loop bias.
- Positive defined too broadly—unlearnable signal.
- Test set too small for rare class—unstable recall estimates.
- Ignoring label drift when policy changes.
Worked example. AML alerts: positive rate 0.8%; primary metric recall at 95% precision on holdout; secondary alert volume reduction at fixed investigator capacity. Threshold chosen to cap false positives at 200/day nationally.
Train, validation, test, and temporal splits
Definition. Training data fits model parameters; validation tunes hyperparameters and thresholds; test provides unbiased final estimate—each used once for that purpose. Temporal splits train on past, validate/test on future—mandatory for time-ordered data.
Engagement use. Default to time-based splits for anything operational (credit, fraud, churn). Random splits inflate scores when autocorrelation exists. Specify split dates explicitly in eval plan. Nested cross-validation for small data— but still respect time where needed.
Pitfalls.
- Random split on time series—leakage via future patterns.
- Tuning on test set by repeated peeking.
- Same customers in train and test—entity leakage.
- Validation set reused across many experiments without fresh holdout.
Worked example. Credit model: train through 2022-Q4, validate 2023-H1, test 2023-H2; no account appears in more than one split; features computed with point-in-time warehouse snapshots.
Bias, variance, and overfitting
Definition. Bias is systematic error from overly simple models; variance is sensitivity to training noise. Overfitting fits training noise—high train metric, poor generalisation. Underfitting misses signal—poor train and test.
Engagement use. Compare train vs validation curves. Regularisation, feature reduction, more data, and simpler models reduce overfitting. For high-capacity models (deep nets, LLMs), need more data and stronger regularisation. Ask: does performance improve on validation when we add data? If not, suspect label noise or leakage.
Pitfalls.
- Hundreds of features on thousands of rows—guaranteed overfit.
- Early stopping ignored in boosting runs.
- Evaluating on training distribution only—covariate shift in production.
- Complex ensemble for 2% gain not worth ops burden.
Worked example. Propensity model: train AUC 0.91, validation 0.84, test 0.83—acceptable gap. Experiment with 400 features → train 0.96, validation 0.81—rejected; deploy simpler 40-feature model with stable calibration.
Data leakage
Definition. Leakage introduces information into training that would not be available at prediction time, inflating offline metrics. Sources: future data, target-derived features, duplicate entities, preprocessing fit on full dataset, text containing outcome labels.
Engagement use. Run leakage audit checklist before trusting metrics: feature timestamp ≤ decision timestamp; no post-event fields; splits by entity; fit scalers/encoders on train only; review text fields for outcome mentions. Model risk teams in banking require documented leakage review.
Pitfalls.
days_since_last_paymentcomputed using payments after default date.- Including "fraud investigator notes" field populated only after fraud confirmed.
- Customer ID hash appearing in multiple rows across splits.
- Global mean imputation using test set statistics.
Worked example. Hospital readmission model failed audit: feature post_discharge_calls included calls after readmission window closed—removed; AUC dropped 0.89 → 0.76 but production matched 0.75—saved deployment of fraudulent confidence.
Metrics for classification
Definition. Confusion matrix drives precision, recall, F1, specificity. ROC-AUC measures rank quality across thresholds. PR-AUC emphasises positive class—better for imbalance. Calibration measures whether predicted 30% default rate matches actual 30%.
Engagement use. Choose primary metric from error costs. False negative on fraud → weight recall. False positive on credit decline → weight precision. Report confusion at chosen threshold, not only AUC. Executive summary: "At threshold T, we catch X% fraud at Y false alerts per day."
Pitfalls.
- AUC alone on imbalanced data—misleading comfort.
- Threshold picked on test set without business capacity constraint.
- Ignoring calibration when decisions use probability cuts.
- Comparing models at default 0.5 threshold when optimal differs.
Worked example. Fraud: optimise recall subject to precision ≥ 40%; at operating point recall 72%, 180 alerts/day, investigators clear 65%—within capacity. Credit: optimise Gini with reject rate cap 25%.
Metrics for regression and forecasting
Definition. MAE (mean absolute error), RMSE (penalises large errors), MAPE (percentage—unstable near zero), R², quantile loss for intervals. Forecasting adds sMAPE, MASE, and bias (systematic over/under).
Engagement use. Align with finance ops: inventory forecast may prefer MAE; peak staffing cares about under-forecast penalty (lost sales) vs over-forecast (overtime cost)—asymmetric loss functions or pinball loss for quantiles.
Pitfalls.
- RMSE on outliers drives wrong model when median matters.
- MAPE undefined for intermittent demand series.
- Forecast horizon mixed in one number without per-horizon breakdown.
- Ignoring confidence intervals for inventory safety stock.
Worked example. Energy demand forecast: report MAE and peak-day bias separately; SLA requires ≤8% error on top 10 peak days annually; model selected on weighted composite, not RMSE alone.
Business-weighted and custom metrics
Definition. Business metrics translate model errors to currency or KPI impact—expected loss, profit curve, cost-weighted misclassification. Often require simulation on holdout with business rules applied.
Engagement use. Build simple spreadsheet or simulator: at each threshold, compute fraud prevented £ minus investigation cost minus customer friction cost. Present Pareto frontier of operating points to steering committee—let them pick policy, not data science pick threshold in isolation.
Pitfalls.
- Fabricated cost weights without finance sign-off.
- Static weights when macro environment shifts.
- Optimising proxy metric never reported to executives.
- Ignoring second-order effects (customer attrition from false declines).
Worked example. Collections model: expected recovery £ = sum over accounts of P(pay|score) * balance * contact_cost; optimal segment differs from max AUC segment—finance approves weights.
Model lifecycle: experiment, deploy, monitor, retrain, retire
Definition. ML lifecycle spans problem definition → data → train → validate → deploy → monitor drift and performance → retrain on schedule or trigger → retire when superseded or harmful.
Engagement use. Define owners for each stage. Monitoring: feature drift, prediction drift, outcome capture lag, bias by segment. Retrain triggers: performance below threshold, policy change, new product launch. Retirement: archive model card, migrate consumers, delete non-compliant artifacts.
Pitfalls.
- Deploy without shadow period comparing to incumbent.
- No outcome feedback loop—cannot detect drift.
- Eternal retrain without champion/challenger discipline.
- Model cards missing intended use and prohibited use.
Worked example. Churn model: monthly drift dashboard; retrain quarterly; champion/challenger for 2 weeks; auto-rollback if precision drops >5pp; model card updated in central registry.
Classical ML vs LLM: decision framework
Definition. Classical ML (logistic regression, random forest, gradient boosting, small neural nets on tabular data) excels when inputs are structured, labels abundant, decisions repeat at scale, latency and cost sensitive, and explainability required. LLMs excel when task needs language understanding/generation, few-shot adaptation to varied phrasing, multimodal input, or weak structure—but cost more and vary in determinism.
Engagement use. Decision tree:
- Is input primarily tabular/ numeric with stable schema? → Start classical ML.
- Is task open-ended language (summarise, draft, converse)? → LLM.
- Is task classification on short text with large labelled set? → Compare fine-tuned encoder vs LLM; often encoder wins on cost/latency.
- Does regulation require feature-level explainability? → Classical or GAM/SHAP-friendly models first.
- Does solution need 100ms p99 at millions QPS? → Classical or small specialised models.
Pitfalls.
- LLM for credit scoring because "latest tech."
- Classical bag-of-words on long complex contracts without structure.
- Ignoring hybrid: classical route + LLM explain.
- No total cost of ownership—API fees at scale exceed ML ops headcount.
Worked example. Invoice field extraction: 500K labelled invoices → layout-aware classical CV + gradient boosting on fields beats GPT-4 on accuracy (94 vs 91) at 1/40th inference cost; LLM used only for exception explanation to clerks.
Baselines, champion/challenger, and AutoML realism
Definition. Baselines are simple comparators: business rules, heuristics, prior model, logistic regression. Champion/challenger runs new model parallel to production. AutoML automates feature search and model selection—useful for baselines, not excuse to skip problem framing.
Engagement use. No model proposal without beaten baseline documented. AutoML accelerates exploration on tabular data; still require leakage-safe splits and business metrics. Challengers need rollback and traffic caps.
Pitfalls.
- Beating "no model" only—strawman.
- AutoML leaderboard on leaked data.
- Challenger runs without same data pipeline as champion.
- Overfitting AutoML to validation through repeated runs without fresh data.
Worked example. Routing model: baseline = current rules engine 61% correct route; ML model 74%; AutoML 76% but less explainable—ship interpretable 74% model for phase one with challenger AutoML in shadow.
Frameworks and methods
Problem-type selection matrix
| Business ask | Formulation | Typical algorithms | Primary metrics |
|---|---|---|---|
| Approve/deny | Binary classification | Logistic, XGBoost | Precision/recall at capacity |
| Risk score | Probability calibration | GAM, boosting + calibration | Gini, Brier score |
| Next best action | Multiclass / bandit | Multinomial logit, contextual bandit | Uplift, conversion |
| Queue priority | Ranking | LambdaMART, learning to rank | NDCG@k, SLA hit rate |
| Demand next week | Forecasting | Prophet, ETS, TFT | Weighted MAE, peak bias |
| Unknown pattern | Anomaly | Isolation forest, autoencoder | Alert precision, volume |
CRISP-DM adapted for enterprise AI
Business understanding → data understanding (topic 09) → modelling → evaluation (topic 21) → deployment → monitoring. Align documentation to gates.
Model card essentials
Intended use, training data summary, metrics by segment, limitations, ethical considerations, contact owner, version, retrain date.
Fairness and segment analysis
Report metrics by protected or proxy segments where regulation applies; disparity triggers review—not only global AUC.
ML vs LLM quick reference
| Criterion | Favour classical ML | Favour LLM |
|---|---|---|
| Input | Structured tables | Unstructured text/images |
| Labels | Large, reliable | Scarce; instructions suffice |
| Latency | Sub-100ms needed | Seconds acceptable |
| Cost | High volume | Low/medium volume |
| Explainability | Required feature attribution | Narrative explanation enough |
| Determinism | High | Moderate; guardrails needed |
See AI Patterns for combined patterns.
8D alignment
Feasibility metrics in Design; training and eval in Develop; monitoring in Deliver. 8D Framework.
Explainability and model risk (regulated environments)
Banks, insurers, and health systems often require interpretable models or post-hoc explanations (SHAP, LIME, feature importance) for high-impact decisions. Gradient boosting with monotonic constraints or GAMs may beat black-box nets on approval velocity through model risk committees.
Engagement use: Document reason codes surfaced to humans—not raw SHAP dumps customers cannot parse. For LLM explanations of ML scores, ensure narrative does not contradict underlying score logic (conduct risk).
Pitfalls: Explainability after model chosen—committee rejects late. SHAP on correlated features misleads. LLM rationalisation of wrong score—worse than no explanation.
Sampling, active learning, and weak labels
When labels are expensive, active learning selects uncertain examples for human labelling. Weak labels from rules bootstrap training—must audit rule precision. Semi-supervised methods use small labelled + large unlabelled sets.
Engagement use: Phase pilot: 500 labels → model → active loop targeting 2,000 labels where uncertainty highest—cheaper than random 2,000 upfront. Document label budget £ and timeline in feasibility memo.
Ensemble and model stacking (when justified)
Ensembles combine models for marginal gains—justify ops cost. Stacking meta-learners on small data overfits easily. Prefer single well-tuned model until production pain proves otherwise.
Hyperparameter tuning without leakage
Tune on validation only; nested cross-validation for small datasets. Log all trials—repeated peeking at test set inflates confidence. For AutoML, cap trials and use fresh holdout quarterly.
Production monitoring checklist
| Signal | Detection | Action |
|---|---|---|
| Feature drift | PSI, KS test on distributions | Investigate pipeline; pause if severe |
| Prediction drift | Score distribution shift | Compare to outcomes when available |
| Concept drift | Performance drop on rolling window | Retrain or revert champion |
| Missing data spike | Null rate alerts | Upstream system incident |
| Latency | p99 inference time | Scale or simplify model |
Define thresholds in eval plan (topic 21)—not invented during incident.
Communicating ML trade-offs to executives
Translate metrics to decisions, not formulas:
- "At this threshold we approve 78% of good customers automatically vs 71% today; false declines fall by 12,000/year—worth ~£X retention."
- "We catch 68% of fraud cases in this segment; raising catch rate to 75% doubles false alerts and needs 40 new investigators—£Y."
Avoid AUC in steering decks unless audience is model risk; pair with operating point story.
Classical ML vs LLM extended decision cases
| Use case | Recommended path | Why |
|---|---|---|
| Churn on CRM features | XGBoost / logistic | Tabular, high volume, explainable |
| Sentiment on 500K labelled chats | Fine-tuned encoder or small LLM | Text but labels exist; compare cost |
| Contract clause classification | Layout LM + head | Structure matters; LLM costly |
| Ad-hoc executive Q&A on board packs | RAG + LLM | Open vocabulary, low volume |
| Real-time fraud scoring | Gradient boosting | Latency <50ms, features tabular |
| Product description generation | LLM | Generative language task |
Hybrid pattern: ML scores route; LLM explains to operator—common in collections and underwriting workstations.
Experiment tracking and reproducibility
Minimum experiment log: data snapshot ID, feature version, hyperparameters, seed, metric on val/test, git hash, author. Enables audit response: "Why did we deploy model v2.3?" Reproducibility required for model risk—not optional notebook on laptop.
Real-world scenarios
Scenario A: Fraud alert triage (retail banking)
Bank processes 42M transactions/month; fraud rate 0.12%. Vendor demo claims 99% accuracy. You recalculate: naive "all legit" baseline is 99.88% accurate but useless.
Reframe: binary classification with recall at precision floor. Investigators handle 350 cases/day capacity. Primary metric: recall at ≥35% precision; secondary: £ fraud prevented via simulator. Leakage audit removes chargeback_flag feature available only after investigation.
Result: selected model recall 68% at 38% precision vs rules 41%; £14M estimated annual fraud catch improvement; no accuracy headline in steering deck—executives see alerts/day and £ impact.
Scenario B: Credit document classification (commercial lending)
Lender wants LLM to classify 47 document types from submissions. 180K historical documents with labels; average 12 pages PDF.
Analysis: fine-tuned document classifier (layout LM + head) achieves 96.2% macro-F1 vs GPT-4 zero-shot 89%, fine-tuned GPT 93% at 12× cost. LLM valuable for low-confidence explanations to analysts.
Hybrid recommendation: classical model routes docs; LLM explains exceptions only. Latency p95 220ms vs 3.2s full LLM. Model risk accepts feature attributions on classical path.
Scenario C: Retail demand forecasting (stretch)
National retailer forecasts SKU-store weekly demand for 12K SKUs. Finance burned by RMSE-only model that under-forecast peak promotional weeks.
Fix: composite metric—0.6× MAE + 0.4× peak-week bias penalty; separate model for promotional flags from marketing calendar; temporal split by fiscal year. Holdout year 2024 promo weeks error cut 19%; inventory stockouts down 8% in pilot region (£2.3M benefit narrative).
Lesson: regression metric must match asymmetric business cost, not default loss function.
Scenario D: Manufacturing anomaly detection
Plant sensors stream 4K metrics; failures rare (14 serious events last year). Team proposes supervised classifier—but labels sparse.
Approach: unsupervised anomaly scores on windows + human review; semi-supervised refine with confirmed failures; precision at top-50 daily alerts as metric. Classical isolation forest + domain thresholds beats LLM on sensor logs—LLM used in separate runbook assistant (topic 11), not raw time-series.
Outcome: false alarm rate acceptable to maintenance leads; 3 pre-failure catches in 6-month pilot worth £1.1M avoided downtime.
Scenario E: Propensity model vs LLM marketing copy (retail media)
Retailer conflates two workstreams: (1) who to target with promotion; (2) what text to send. Data science proposes LLM for both. You separate:
Targeting: gradient boosting on 240 behavioural features; labels from campaign response; temporal split; primary metric uplift in holdout cell vs control. Copy: LLM generates variants; selection via bandit on small traffic—different eval.
Results: ML targeting + LLM copy beats LLM-only audience selection by 14% revenue per send at 1/25th inference cost. Executive dashboard shows ML for decision, LLM for language—becomes template for CRM programme.
Negative case avoided: LLM asked to "predict who will buy" from text profiles alone—no better than random on eval; would have shipped had ML vs LLM gate not existed.
Scenario F: Healthcare readmission prediction (fairness review)
Hospital system deploys readmission model; community advocates raise fairness concerns on postcode proxy features.
Response: Segment analysis on validation set shows AUC gap 0.04 between postcodes—not huge but politically salient. Remove proxy features; add calibration by ward; primary metric remains readmission capture at capacity, secondary max segment disparity on false negative rate.
Governance: Model risk committee requires annual fairness report; monitoring dashboard by segment; rollback if disparity exceeds threshold. LLM-generated patient education separate from score—no score in prompt to citizen-facing bot.
Outcome: Model approved with constraints; avoids headline "biased AI" while maintaining 78% recall at operating point; teaches that ML feasibility includes social licence, not only statistical metrics.
Reference tables for workshop facilitation
Classification metric cheat sheet (imbalanced binary):
| Stakeholder ask | Translate to | Watch out for |
|---|---|---|
| "Catch more fraud" | Recall ↑ | False alert capacity |
| "Stop blocking good customers" | Precision ↑ | Missed fraud £ |
| "Overall performance" | PR-AUC, not accuracy | Naive baseline |
| "Score ranks risk well" | ROC-AUC | Threshold still needed |
| "Probabilities trustworthy" | Calibration (Brier) | Overconfident scores |
When to escalate to model risk / compliance:
- Credit, insurance pricing, hiring, health triage, criminal justice adjacency
- Automated decision with material individual impact
- Use of protected attributes or close proxies
- Performance disparity across segments above agreed threshold
- Explainability cannot be produced for deployed architecture
Document escalation in feasibility memo—delays are cheaper than retroactive committee rejection.
Offline vs online evaluation (preview for topic 21)
Offline eval uses historical labelled data—fast iteration but risks leakage and distribution shift. Online eval uses A/B tests, shadow mode, or human review queues—slower but ground truth on production traffic. Feasibility plans should specify both: offline for model selection; online for scale gate.
Minimum offline set size: enough positive examples for stable recall estimate—rule of thumb ≥200 positives in test for rare events (adjust with confidence intervals). For GenAI overlap, topic 21 covers LLM eval; ML holds same discipline on labels and splits.
Pilot shadow mode: new model scores alongside production without driving decisions—compare disagreement rate before flip. Champion/challenger period typically 2–4 weeks in banking; shorter only with low-risk use cases and explicit sponsor acceptance.
Practice exercises
Primary exercise: Problem formulation memo (2–3 hours)
Choose: subscription churn, warranty claim fraud, or hospital no-show prediction.
Deliver:
- Problem type statement (supervised formulation, label definition, decision time).
- Primary and secondary metrics with business justification.
- Split strategy (temporal or entity)—with dates or rules.
- Three leakage risks and mitigations.
- ML vs LLM recommendation with five-bullet rationale.
- Baseline you must beat.
Acceptance criteria: Primary metric is not accuracy alone for imbalanced cases; leakage risks are specific fields or processes, not generic "data quality."
Stretch exercise: Threshold and cost simulator (2 hours)
For fraud or credit use case, build spreadsheet: columns threshold, TP, FP, FN, TN, investigator cost, fraud £, customer friction cost. Plot three operating points on precision-recall curve with steering recommendation.
Acceptance criteria: Finance-readable summary sentence at chosen threshold.
Reflection exercise: Metric post-mortem (30 minutes)
Scenario: model with 0.94 AUC deployed; complaints rise after 2 months. List five non-model causes (label drift, policy change, feature pipeline bug, threshold shift, population change). Write monitoring checks for each. File in pattern library.
Questions you should be able to answer
- Is this prediction, ranking, forecasting, or generation—and why?
- What is the label definition and when is it knowable relative to the decision?
- What is the class balance—and why is accuracy insufficient?
- What is the primary metric and how does it map to business cost?
- What baseline must a new model beat to be worth deploying?
- How are train, validation, and test split—and is time respected?
- What leakage paths did you audit in features and text?
- What is the expected precision/recall at the chosen operating threshold?
- Is the model calibrated for probability-based decisions?
- How will you monitor drift and performance in production?
- What triggers retrain or rollback?
- Why classical ML vs LLM for this specific input and volume?
- How do metrics differ across customer segments?
- What is prohibited use of the model per model card?
- What happens if the label definition changes next quarter?
Negative cases
Accuracy theatre on fraud. 99% accuracy, zero fraud caught. Fix: imbalance-aware metrics and baselines.
Random split on time series. Inflated forecast scores. Fix: temporal validation only.
Leakage luxury. Post-outcome features in training. Fix: point-in-time feature store audit.
Metric not owned by business. Data science optimises log loss; ops cares about SLA. Fix: joint metric workshop.
No baseline. Beating prior rules undocumented. Fix: champion baseline mandatory.
Threshold default 0.5. Wrong operating point. Fix: capacity-constrained optimisation.
LLM for tabular at scale. £2M API bill for churn. Fix: gradient boosting + LLM for explanations only.
AutoML without governance. Black box in credit. Fix: interpretability requirements per regulation.
Silent label drift. Product change breaks definition. Fix: label version in pipeline metadata.
Global metrics hide bias. Worst segment fails fairness review. Fix: segment dashboards.
Notebook ≠ production pipeline. Different imputation. Fix: shared feature code.
Forever champion. Challenger never promoted or killed. Fix: time-boxed experiments.
Integration with adjacent topics
| Topic 10 capability | Related resource |
|---|---|
| Features and pipelines | Topic 09, Data & Knowledge |
| When to add GenAI | Topic 11, AI Patterns |
| Eval design | Topic 21 |
| Model catalog context | Model landscape |
Crosswalk to GenAI topics
| ML foundation decision | Impacts |
|---|---|
| Problem is generative language | Topic 11 model selection |
| Ranking with text features | Hybrid: ML ranker + LLM summary |
| Labels scarce | Compare active learning vs LLM few-shot |
| Explainability required | May block LLM-only credit decisions |
| Monitoring metrics | Topic 21 unified eval dashboard |
Document handoff: when ML team passes scores to LLM layer, specify schema, latency SLA, and fallback if score service down—integration failure common in hybrid designs.
Study drill: metric selection under pressure
Drill (20 minutes): Stakeholder demands "95% accuracy" for imbalanced problem. Write three response bullets: (1) reframe metric in business language; (2) propose primary metric with operating point; (3) show baseline accuracy of naive classifier. Practice aloud—executive interruptions common in steering.
Practice checklist
- I can classify a use case without defaulting to GenAI
- Primary metric matches error costs—not generic accuracy
- I named three leakage risks for a structured prediction problem
- Temporal or entity split specified for operational data
- Baseline and beat criteria documented
- Practice memo filed in pattern library
Related playbook content
- AI Patterns — hybrid ML and GenAI solution shapes
- Data & Knowledge Guide — feature and label pipelines
- Model landscape — when specialised models beat generic LLMs
- 8D Framework — Design and Develop evaluation gates
- How to use this Learning Map — study method
- AI and data foundations — curriculum context
Supplemental scenario — telecom churn prediction vs retention copilot
Context: Mobile operator; two proposals—(A) classical churn model on 14M subscribers; (B) GenAI retention script copilot for call centre.
Formulation comparison:
| Aspect | Churn model (ML) | Retention copilot (LLM) |
|---|---|---|
| Labels | Churn within 90 days | None stable |
| Primary metric | Recall @ top 8% decile | Handle time (self-report weak) |
| Baseline | Logistic 0.71 AUC | Script library search 4.2 min |
| Leakage risk | Future usage in features | Prompt injection via CRM paste |
| Decision | Ship ML to outbound queue | Pilot with HITL; not replace model |
Metric economics: False negative on churn costs £840 lifetime value (Assumption); optimising accuracy alone would miss £12M/year—team chose recall-weighted threshold operating point 0.34 precision / 0.62 recall.
Lesson: Problem type discipline blocked "LLM for everything" narrative; steering funded both tracks with clear ML owns ranking; LLM owns language boundary.
Discussion
Comments
Share feedback or questions about this page. No account required.
Loading comments…