Final competency test
How to use this test
- Pick one live or realistic fictional case (named client, industry, use case).
- Write answers in a single document—two to four sentences per question minimum; more where risk or £ is high.
- Label every statement: Fact (with source), Assumption (with validation plan), or Hypothesis (with test).
- Score each answer Red / Amber / Green using the rubric at the end.
- Gap-map reds and ambers to Learning Map topics and schedule practice artefacts from Core deliverables.
Run VALUE on the full document before sending anything externally.
The twenty-two questions
Q1. What business problem are we solving?
Why it matters: Without a owned business problem, every downstream choice—data, model, integration, price—becomes arbitrary. Sponsors disengage when the conversation is "AI" instead of their KPI gap.
Strong answer includes: A decision-focused problem statement in plain language; named problem owner; link to a metric or risk the organisation already tracks; clear boundary of what is out of scope; distinction from adjacent problems.
Weak answer anti-patterns: "Digitise with AI"; feature list; problem copied from vendor deck; no owner; conflating symptoms ("too many emails") with root cause without validation.
Learning Map topics: Business Fundamentals, Consulting Problem Solving, Product Management, Business Briefing.
Q2. Why is AI appropriate?
Why it matters: AI carries cost, risk and change overhead. If rules, workflow redesign or search suffice, AI wastes money and erodes trust when it fails to beat simpler options.
Strong answer includes: Comparison against non-AI alternatives (process, policy, rules, analytics); explanation of why the problem needs learning, language, vision or prediction at scale; honesty about where human judgment remains essential; simplest effective pattern from the automation hierarchy.
Weak answer anti-patterns: "Because ChatGPT can do it"; no alternatives considered; AI as default for executive fashion; technical pattern named without linking to problem shape.
Learning Map topics: AI Opportunity Discovery, Machine Learning Foundations, Generative AI and LLM Fundamentals, Agentic AI and Workflow Automation.
Q3. Who receives the value?
Why it matters: Benefits realisation and adoption depend on knowing which roles, customers or functions gain time, revenue or risk reduction—and who may lose status, budget or control.
Strong answer includes: Primary and secondary beneficiary segments; "jobs to be done" or workflow role; stakeholders who experience pain today vs who capture P&L; potential losers and mitigation.
Weak answer anti-patterns: "The business" only; IT as beneficiary when P&L is elsewhere; ignoring frontline adoption; no distinction between user and payer.
Learning Map topics: Business Fundamentals, Stakeholder Management, UX and Human Factors, Change Management and Adoption.
Q4. How will value be measured?
Why it matters: Unmeasured value becomes unclaimed value. Finance and operations need baselines, targets, cadence and owners—not narrative after go-live.
Strong answer includes: Named KPIs with baseline source; target range and measurement method; timeframe (leading vs lagging); benefits owner; honest note on attribution difficulty.
Weak answer anti-patterns: "Efficiency"; no baseline; only technical metrics (latency, token cost); benefits claimed before defining instrument.
Learning Map topics: Commercial and Financial Modelling, Business Fundamentals, Performance and FinOps.
Q5. What data is required?
Why it matters: Data scope drives feasibility, timeline, cost, privacy and architecture. Vague data answers hide the hardest part of most AI programmes.
Strong answer includes: Dataset inventory (sources, formats, freshness, volume); training vs inference vs eval separation; label or feedback requirements; retention needs; gaps flagged with remediation owners.
Weak answer anti-patterns: "All data in the lake"; no inventory; conflating demo sample with production corpus; missing ground truth for supervised tasks.
Learning Map topics: Data Engineering and Architecture, Machine Learning Foundations, Retrieval-Augmented Generation.
Q6. Is that data usable and legally accessible?
Why it matters: Legal and ethical blockers stop programmes after spend. "We have the data" is not the same as lawful, consented, quality-assured access.
Strong answer includes: Legal basis or flag for legal review; classification (PII, special category, confidential); contract and licence constraints; cross-border transfer issues; quality blockers with remediation plan; DPIA or equivalent trigger noted.
Weak answer anti-patterns: "Legal will sort it"; assuming public internet training data fixes enterprise need; no DPO/security consultation plan; ignoring retention limits.
Learning Map topics: Privacy, Legal and Compliance, Data Engineering and Architecture, Responsible AI Governance.
Q7. Which AI pattern should be used?
Why it matters: Pattern choice determines team skills, eval approach, cost profile and failure modes—RAG is not a substitute for workflow design; agents are not free reliability.
Strong answer includes: Recommended pattern (rules, search, ML, GenAI, RAG, agentic) with rationale; rejected patterns and why; human-in-the-loop placement; alignment to latency, accuracy and explainability needs.
Weak answer anti-patterns: Pattern chosen from hype; RAG without retrieval quality plan; agents without tool governance; no spike for unknowns.
Learning Map topics: Generative AI and LLM Fundamentals, Retrieval-Augmented Generation, Agentic AI and Workflow Automation, AI Opportunity Discovery.
Q8. Which model and platform should be selected?
Why it matters: Model and platform choices affect cost, latency, compliance, data residency, eval reproducibility and exit strategy—not only benchmark leaderboard position.
Strong answer includes: Selection criteria weighted by client priorities; shortlist with trade-offs; deployment model (cloud region, private, on-prem); FinOps implications; lock-in and migration notes; eval plan to confirm choice.
Weak answer anti-patterns: Latest model by default; ignoring enterprise agreement or approved list; no cost model at expected volume; selection before requirements.
Learning Map topics: Generative AI and LLM Fundamentals, Cloud Platform Engineering, Vendor and Technology Evaluation, Performance and FinOps, Models landscape.
Q9. How will the solution integrate with existing systems?
Why it matters: Enterprise AI fails in the handoffs—CRMs, case tools, ERP, identity, audit logs—not inside the model call.
Strong answer includes: Systems map and integration pattern (API, event, RPA last resort); data contracts; sync vs async; error handling; ownership per interface; non-functional needs (SLA, idempotency).
Weak answer anti-patterns: "REST API later"; missing identity propagation; no reconciliation for write-back; integration as Phase 2 after budget spent.
Learning Map topics: Integration and Enterprise Systems, Enterprise Architecture, Software Engineering.
Q10. How will identity and access be controlled?
Why it matters: AI systems amplify credential and permission mistakes—over-broad retrieval and tool access cause breaches and audit findings.
Strong answer includes: AuthN/AuthZ model aligned to enterprise IAM; least privilege for data, tools and admin; service identities and secrets handling; break-glass process; logging of access decisions.
Weak answer anti-patterns: Shared API keys; same access for all users; admin console without MFA; retrieval without document-level ACL enforcement.
Learning Map topics: Security Engineering, Cloud Platform Engineering, Retrieval-Augmented Generation.
Q11. How will prompt injection and data leakage be prevented?
Why it matters: LLM applications introduce new attack surfaces—untrusted content in prompts, tool abuse, exfiltration via outputs and logs.
Strong answer includes: Threat model summary; input/output controls; separation of trusted vs untrusted content; tool sandboxing; DLP on outputs; monitoring for abuse; red-team or eval cases for injection.
Weak answer anti-patterns: "We use a guardrail product" with no design; trusting retrieved web pages; tools with broad network access; logging full prompts with secrets.
Learning Map topics: Security Engineering, Prompt and Context Engineering, Responsible AI Governance.
Q12. How will output quality be evaluated?
Why it matters: Without eval, quality is anecdote. Production AI needs datasets, metrics, gates and regression when models, prompts or data change.
Strong answer includes: Offline eval set and rubrics; online metrics; safety evals; release thresholds; human review workflow; CI regression plan; owner for dataset maintenance.
Weak answer anti-patterns: "Stakeholder will review"; demo questions only; no versioning; subjective acceptance with no rubric.
Learning Map topics: AI Evaluation and Quality Assurance, Prompt and Context Engineering, MLOps, LLMOps and Observability.
Q13. Where is human oversight required?
Why it matters: Regulators, model risk functions and operational reality require clear human roles—especially for consequential decisions, exceptions and failures.
Strong answer includes: Tasks that must not be fully automated; oversight UI and workflow; escalation triggers; training for reviewers; audit trail; alignment to RAI tier.
Weak answer anti-patterns: "Human in the loop" slogan; oversight undefined in TOM; no capacity planning for reviewers; automation of credit/medical/legal without tiering.
Learning Map topics: Responsible AI Governance, UX and Human Factors, Privacy, Legal and Compliance.
Q14. How will the system scale?
Why it matters: Pilot success at ten users fails at ten thousand without capacity, latency, cost and ops planning—and without batch vs online clarity.
Strong answer includes: Expected load profile; scaling model (horizontal, queueing, caching); latency targets; cost at scale; bottlenecks (vector DB, rate limits); DR and multi-region if required.
Weak answer anti-patterns: "Cloud autoscales"; no load test plan; ignoring embedding/index rebuild; linear cost surprise at token volume.
Learning Map topics: Cloud Platform Engineering, Performance and FinOps, Enterprise Architecture.
Q15. What will it cost to build and operate?
Why it matters: Investment committees approve TCO, not demos. Hidden run cost (tokens, support, eval, data pipelines) kills programmes in year two.
Strong answer includes: Build cost by phase (internal + external); run cost (infra, licences, tokens, support FTE); sensitivity to volume; comparison to alternatives; FinOps controls.
Weak answer anti-patterns: Build only; token cost ignored; no contingency; benefits case without cost-to-achieve.
Learning Map topics: Commercial and Financial Modelling, Performance and FinOps, Presales and Solution Shaping.
Q16. What are the alternatives?
Why it matters: Credible advisors compare do nothing, process fix, buy SaaS, outsource, build—not only their preferred build option.
Strong answer includes: At least three options with pros/cons; recommendation and conditions; do-nothing cost; exit paths; why now vs later.
Weak answer anti-patterns: Straw-man alternatives; build bias; no do-nothing case; vendor comparison without client-weighted criteria.
Learning Map topics: AI Opportunity Discovery, Vendor and Technology Evaluation, Consulting Problem Solving.
Q17. What are the major risks and dependencies?
Why it matters: Steering committees fund risk-adjusted plans. Unowned dependencies (data access, API from vendor X, board approval) slip every AI programme.
Strong answer includes: Top risks with likelihood, impact, mitigation, owner; dependencies on client, third parties, regulation; assumptions that invalidate the plan if false; RAID linkage.
Weak answer anti-patterns: Generic risk boilerplate; no owners; dependencies discovered in week twelve; risks without mitigation budget.
Learning Map topics: Delivery and Programme Management, Project Management overview, Stakeholder Management.
Q18. How will the solution be governed?
Why it matters: Production AI touches architecture boards, model risk, security, privacy, procurement and change advisory—governance gaps block go-live.
Strong answer includes: Forums and approvals required; evidence pack per gate; ongoing model/prompt change control; audit and retention; RAI alignment.
Weak answer anti-patterns: "Agile so no governance"; one-off sign-off; no change control for prompts; unnamed approvers.
Learning Map topics: Responsible AI Governance, Enterprise Architecture, Security Engineering.
Q19. How will users adopt it?
Why it matters: Unused AI is negative ROI. Adoption requires segmented comms, training, champions, workflow fit and resistance management—not launch day alone.
Strong answer includes: Change impact by role; comms and training plan; champions; adoption metrics; feedback loop; co-design evidence.
Weak answer anti-patterns: "Training deck at go-live"; ignoring union/work council; no time allocated for new workflow; UX afterthought.
Learning Map topics: Change Management and Adoption, UX and Human Factors, Communication and Executive Articulation.
Q20. How will it be supported in production?
Why it matters: Hypercare ends; BAU needs runbooks, on-call, incident process, model updates and vendor SLAs—or the system rots.
Strong answer includes: Support model (L1/L2/L3, vendor); SLAs; runbooks; monitoring; prompt/model update process; KT completion criteria.
Weak answer anti-patterns: Delivery team stays forever; no on-call; incidents ad hoc; no runbook for LLM-specific failures.
Learning Map topics: Operations and Production Support, MLOps, LLMOps and Observability, Delivery and Programme Management.
Q21. How will benefits be tracked?
Why it matters: Programmes that cannot show realised value lose renewal and reputation. Tracking closes the loop from Q4 and the value hypothesis.
Strong answer includes: Benefits register with baseline, target, owner, cadence; measurement method; honest attribution; review forum; link to optimisation backlog.
Weak answer anti-patterns: Benefits workshop only at start; no baseline; claimed savings without finance alignment; tracking stops after go-live.
Learning Map topics: Commercial and Financial Modelling, Business Fundamentals, Change Management and Adoption.
Q22. What decision does the client need to make now?
Why it matters: Every readout should end with a specific decision—fund, pause, pivot scope, assign owner, approve gate—not passive "thanks for the update."
Strong answer includes: One primary decision in one sentence; options with consequences; who decides and by when; what happens if deferred; prep materials required.
Weak answer anti-patterns: No ask; vague "continue discovery"; multiple competing asks; decision maker not in room plan.
Learning Map topics: Communication and Executive Articulation, Presales and Solution Shaping, Stakeholder Management.
Scoring rubric (Red / Amber / Green)
Score each question independently. Overall readiness uses the worst significant cluster rule: any Red on Q1–Q4, Q6, Q11, Q12 or Q22 caps overall at Amber until closed.
| Score | Definition | Typical evidence |
|---|---|---|
| Green | Written answer is specific to the case, decision-useful, with labelled facts/assumptions and no material gaps | Peer or manager would accept in a steering pack with minor edits |
| Amber | Directionally correct but missing ownership, numbers, or validation plan—or relies on unstated assumptions | "We need to confirm with legal" is Amber only if owner and date are named |
| Red | Cannot answer; answer is generic, wrong, or dangerously incomplete for the risk level | Would embarrass the firm if sent to client today |
Cluster guidance
| Cluster | Questions | Green bar |
|---|---|---|
| Commercial | Q1–Q4, Q15–Q16, Q21–Q22 | Problem, value, measurement and decision are linked and owned |
| Data & feasibility | Q5–Q8 | Data legal/quality and pattern/model choice defensible |
| Enterprise fit | Q9–Q10, Q14, Q17–Q18 | Integration, security, scale, risk and governance concrete |
| Trust | Q11–Q13 | Threat-led controls and eval/oversight defined |
| Adoption & ops | Q19–Q20 | Change and BAU plans exist with metrics and runbooks |
Overall readiness (manager lens)
| Profile | Rule of thumb |
|---|---|
| Staff solo on discovery | All commercial + data cluster Green or Amber; no Red on Q1, Q2, Q6 |
| Own design gate | Trust + enterprise fit largely Green |
| Lead pursuit to proposal | Q15–Q16, Q22 Green with VALUE-clean write-up |
| Capstone "strong ASE" | ≥18 Green, remainder Amber, zero Red |
Worked example: scoring one question (Q4)
Case: Retail bank — AI-assisted handling of routine service emails.
| Answer fragment | Score | Why |
|---|---|---|
| "We will improve customer experience." | Red | No metric, no baseline, not actionable |
| "We will reduce average handle time." | Amber | Metric named but no baseline, owner or measurement method |
| "We will reduce tier-1 email AHT from 11.2 min (Q1 ops dashboard) to 9.5 min by Q2 FY27, measured weekly by workforce analytics, owner: Head of Service Ops; attribution excludes concurrent telephony project." | Green | Specific, owned, measurable, time-bound, honest scope |
Use this granularity when self-scoring. Amber means one coaching session from Green—not "good enough for now" on a £1m programme.
Facilitation guide (managers)
When running a 45-minute competency review:
- Engineer sends written answers 24 hours before (no live improvisation only).
- Pick six questions on rotation (always include Q1, Q6, Q11, Q22 over two sessions).
- Ask " What evidence supports that?" for every green claim.
- Record gaps as rows on Core deliverables practice list.
- Reschedule Red items with owner and date—never verbal "we'll cover later."
Repeat monthly until capstone bar met.
Self-assessment
- I can answer each question in writing for my current case
- Facts, assumptions and hypotheses are labelled
- Every "I don't know" has an owner and a date to close it
- The client decision (Q22) is explicit
- Each answer scored Red/Amber/Green with gap-mapped topics
- At least one peer has red-teamed my Red and Amber items
Discussion
Comments
Share feedback or questions about this page. No account required.
Loading comments…