Vendor and Technology Evaluation
Executive view
Ask whether PoC measured production-like constraints—latency at peak, residency, audit logs, exit export—not demo latency on a clean corpus. Challenge awards driven by relationship or feature checklists.
Decision required: Is there a documented exit plan and TCO comparison including migration cost if the vendor fails or deprecates?
Technical view
Build scorecards before PoC starts; define pass/fail thresholds; run the same eval harness on all options; capture integration effort, not only licence line.
Document lock-in vectors: embeddings, prompts, agent graphs, fine-tunes, observability data.
Why this matters
AI vendor markets move faster than enterprise procurement cycles. Model capabilities shift quarterly; pricing changes overnight; startups get acquired; hyperscalers deprecate deployments. Teams that choose tools from conference demos or sales lunches discover:
- Vector DB that wins on latency fails residency and backup RPO
- Agent platform that ships fast locks prompts in proprietary runtime
- "Cheapest" LLM API costs 3x at production token volumes without caching
- Observability SaaS cannot export traces when you migrate
- SI partner staffs juniors after oral presentation with seniors
Structured evaluation turns subjective enthusiasm into evidence procurement and architecture boards can defend. It connects to topic 30 (contract terms reflect PoC thresholds) and topic 22 (FinOps validates TCO assumptions).
Weak evaluation produces technical debt, re-procurement within 18 months, and blame when the chosen vendor cannot meet production SLOs.
Learn
What to evaluate — categories of AI stack components
| Component | Evaluate for | Common trap |
|---|---|---|
| Foundation model API | Quality, latency, residency, data use policy, deprecation notice | Demo model ≠ production deployment |
| Vector / search | Recall, latency p95, hybrid search, backup, residency, export | Benchmark on clean data only |
| Orchestration / agents | Tool auth, versioning, eval hooks, portability | Pretty UI, no CI/CD |
| Observability | Trace export, PII redaction, eval integration | Vendor-only dashboards |
| Governance platforms | Policy, inventory, approval workflow | Checkbox vs API integration |
| SI / partner | Delivery method, references, staff quality, IP terms | Brand name, weak bench |
Evaluate integration surface as heavily as standalone features—AI value is systemic.
Evaluation criteria — the core dimensions
Function — Does it solve the use case with required human oversight?
Integration — Identity, APIs, ERP/CRM, data pipelines, existing cloud landing zone.
Performance — Latency p50/p95/p99 under contracted concurrency; batch vs streaming.
Cost — Licence + consumption + engineering + migration; 3–5 year TCO.
Compliance — Residency, certifications, DPA, training opt-out, audit logs.
Support — SLAs, escalation, engineering access during PoC and prod.
Portability — Export embeddings, prompts, configs, eval results.
Exit — Termination assistance, data deletion proof, alternative stack mapping.
Vendor viability — Financial stability, roadmap, customer concentration risk.
Weight criteria before vendor meetings—prevent post-hoc weight adjustment to favour incumbent.
Weighted scorecards — practical design
Example weighting for regulated bank internal copilot:
| Criterion | Weight | Rationale |
|---|---|---|
| Security & compliance | 25% | Non-negotiable gate |
| Functional fit | 20% | Workflow coverage |
| Integration | 15% | SSO, audit, existing Azure |
| TCO (5 year) | 15% | FinOps sustainability |
| Performance | 10% | p95 under load |
| Portability / exit | 10% | Avoid captive formats |
| Support & delivery | 5% | SI quality if applicable |
Score 1–5 with evidence column mandatory—"5" requires PoC metric or reference call.
Gate criteria: Any vendor scoring below 3 on security is disqualified regardless of total score.
Bake-offs — structure that produces decisions
Anti-pattern: Three vendors, three different scripts, executives vote on storytelling.
Better structure:
- Common brief — Same use case, data sample, success metrics, time box
- Common eval harness — Same queries, same load profile where possible
- Blind scoring — Technical panel scores before commercial reveal
- Structured orals — Risk, exit, roadmap, staffing; not only demo
- Decision memo — Options, scores, recommendation, dissent
Bake-off duration: 2–4 weeks PoC per finalist; not one-day demo unless tier-low tool.
PoC design — production-like constraints
PoC must answer questions the demo hides:
| Dimension | PoC test | Pass example |
|---|---|---|
| Data | Messy PDFs, scans, duplicates, PII redaction | Recall ≥85% on agreed set |
| Load | 200 concurrent users or agreed RPS | p95 latency ≤5s |
| Failure | Provider throttle, index partial outage | Graceful degrade documented |
| Security | SSO, RBAC, injection attempt on agent tool | Block unauthorised tool call |
| Ops | Log export, trace ID, rollback one version | Complete in <30 min |
| Cost | 7-day token meter on realistic traffic | Within ±20% of model |
PoC exit report template: hypothesis, method, results, gaps, recommendation, lock-in assessment.
Lock-in — identify before you feel it
| Lock-in type | Symptom | Mitigation |
|---|---|---|
| Data format | Proprietary embedding store | Open export; dual-write period |
| Prompt/runtime | Agent DSL not portable | Abstract orchestration layer |
| Model coupling | Features require one vendor API | Adapter pattern; eval on swap |
| Observability | Traces trapped in SaaS | OpenTelemetry export requirement |
| Commercial | Egress fees, minimum commit | Exit cost in TCO; negotiation |
| Skills | Team trained only on one UI | Document patterns; hire for fundamentals |
Acceptable lock-in: Conscious trade for speed with documented exit cost and time-boxed review (e.g. 18 months).
TCO comparison — beyond licence line
Three-year TCO components:
TCO = Licence/subscription
+ Consumption (tokens, compute, storage, egress)
+ Engineering (build, integrate, maintain)
+ Operations (on-call, FinOps, eval re-runs)
+ Migration / exit (year 3 scenario)
+ Risk reserve (vendor failure, price increase)
Model low / base / high consumption scenarios. Bank copilot example: base £420k/year run-rate vs low £280k vs high £680k at 2x traffic growth.
Include opportunity cost of engineering weeks spent on vendor-specific work.
Build vs buy vs partner — decision inputs
| Option | When it wins | When it loses |
|---|---|---|
| Buy (SaaS/platform) | Standard workflow; speed; vendor roadmap aligns | High customisation; strict residency; exit fear |
| Build | Differentiation core; full control; long horizon | Time to market; skills gap |
| Partner (SI) | Capacity gap; regulated delivery record | IP leakage; weak knowledge transfer |
Hybrid common: buy platform, build adapters and eval—score both parts.
Frameworks and methods
Vendor evaluation lifecycle
PoC charter template
Objective — One sentence decision enabled
Scope in / out — Data types, users, integrations
Duration — Start/end; effort cap per vendor
Success metrics — Numeric thresholds
Environment — Residency, identity, network parity with prod target
Deliverables — Config export, eval results, incident simulation log
Governance — Named client owner; weekly checkpoint
No charter → no PoC.
TRADE scorecard — qualitative guardrails
Beyond numbers, score Trust (reference integrity), Roadmap (credible vs vapour), Alignment (partner incentives), Delivery (staffing honesty), Exit (export demonstrated).
Reference call protocol
Ask references matched to your scale and industry:
- What broke in production first six months?
- Actual vs quoted TCO?
- Migration or export experience?
- How did vendor handle model deprecation?
- Would you choose them again—why/why not?
Take two references minimum per finalist; one should be churned customer if obtainable.
Exit plan one-pager
| Element | Content |
|---|---|
| Trigger | Vendor failure, 2x price, residency breach, acquisition |
| Alternative | Named second vendor or open stack |
| Data export | Embeddings, prompts, logs, fine-tunes — format and duration |
| Migration effort | Engineering weeks estimate |
| Parallel run | Dual-stack period length |
| Cost | One-time migration budget |
Attach to decision memo before award.
Integration effort estimation
| Integration | Small | Medium | Large |
|---|---|---|---|
| SSO / RBAC | 1–2 weeks | 2–4 weeks | 4+ weeks |
| ERP read | 2–4 weeks | 4–8 weeks | 8+ weeks |
| ERP write (agent) | 4–8 weeks | 8–16 weeks | 16+ weeks |
| Legacy search replace | 4 weeks | 8–12 weeks | 12+ weeks |
Add to TCO; do not assume "API available = free integration."
Real-world scenarios
Scenario A — Global bank: LLM API bake-off
Context: Tier-1 bank; three shortlisted API providers for internal research copilot; 15,000 users; data classification internal confidential; must stay EU.
PoC design: Common 800-query eval set; 150 concurrent users load test; injection test on 20 agent scenarios; 14-day run.
Results:
| Vendor | Faithfulness | p95 latency | EU residency | 3yr TCO (base) | Exit score |
|---|---|---|---|---|---|
| A | 91.2% | 3.1s | Yes | £4.2m | 4/5 |
| B | 89.8% | 2.4s | Yes | £3.6m | 2/5 (export weak) |
| C | 92.0% | 4.8s | Partial | £3.1m | 3/5 |
Decision: Vendor A wins despite highest TCO—exit score and agent tool audit pass; B fails gate on trace export; C fails residency gate for subset of use cases.
Numbers: Projected token growth 40% YoY; high scenario TCO £6.1m vs £4.2m base—FinOps alerting at £350k/month spend.
Scenario B — Retailer: vector database selection
Context: UK retailer; product knowledge RAG; 2.3m SKUs; demo showed Vendor X 120ms p95; Vendor Y 210ms.
Deep PoC: Hybrid search on dirty catalog (duplicate titles, missing attributes); backup restore drill; GDPR deletion of SKU subset.
Findings: Vendor X 120ms degrades to 890ms at 500 QPS; backup RPO 24h vs requirement 4h; Vendor Y 210ms stable at 500 QPS; RPO 1h; open export to Parquet.
Decision: Vendor Y selected; 8% slower at demo load acceptable for production stability and exit.
Numbers: Infrastructure cost £18k/month vs £14k/month; avoided £200k+ re-platform estimate at year 2.
Scenario C — Pharma: agent platform vs custom orchestration
Context: Top-10 pharma; field medical workflow; vendor pitches 6-week agent platform go-live vs 16-week custom on open orchestration.
Scorecard weight: Compliance 30%, portability 20%, time-to-value 15%.
PoC: MLR audit trail requirement; agent must log prompt hash, model version, source docs; platform vendor missing immutable audit API (roadmap Q3).
Decision: Custom orchestration on approved cloud; 16 weeks; platform disqualified on gate despite faster demo.
Numbers: Build $480k vs platform $220k year-one licence—but $1.2m regulatory risk modelled if audit gap exploited in inspection.
Scenario D — Manufacturer: SI partner evaluation
Context: Industrial equipment maker; predictive maintenance copilot; two SI finalists; oral presentations excellent for both.
Reference calls: SI-1 reference cites 40% staff turnover mid-project; SI-2 delivered similar project with same core team start-to-finish.
PoC: Both build two-week integration spike to SAP PM module; SI-2 completes API auth + read; SI-1 blocked on security review 10 days.
Decision: SI-2 awarded at 8% higher day rate—lower total cost due to schedule and rework avoidance.
Numbers: SI-1 quoted €1.1m fixed; SI-2 €1.19m; SI-1 projected €280k overrun from delay model.
Reference architecture for vendor comparison — neutral baseline
Before bake-off, document client reference architecture (neutral, not vendor-specific):
Score vendors on fit to reference—reduces vendor-driven architecture lock-in.
PoC data pack specification
Issue identical data pack to all finalists:
| Asset | Description | Volume |
|---|---|---|
| Corpus sample | PDF + HTML mixed | 500 docs |
| Eval queries | With golden answers | 200 |
| Load script | k6 or Locust | 150 VU |
| Abuse set | Injection attempts | 25 |
Hash and version data pack—proves fairness if vendor claims unequal input.
Decision memo template for award recommendation
To: Procurement / Architecture Board
From: [Name]
Re: Award recommendation — [Category]
Recommendation: Award to Vendor [X] subject to [conditions].
Summary score: [table]
Gate failures: [none / Vendor Y disqualified on residency]
TCO (3yr base): [table]
Exit plan summary: [one paragraph]
Dissent: [Name] supports Vendor [Z] on [ground] — addressed by [fact].
Conditions precedent: DPA signed; PoC export test passed; insurance cert.
Signature: _______________
File in document control—audit trail for years-later regulator question.
Vendor scorecard — worked example (bank LLM)
| Criterion (weight) | Vendor A (5=max) | Vendor B | Evidence |
|---|---|---|---|
| Security (25%) | 5 | 3 | A: SOC2+EU; B: US-only processing |
| Functional (20%) | 4 | 4 | Tie on eval |
| Integration (15%) | 5 | 3 | A: Azure landing zone fit |
| TCO 5yr (15%) | 3 | 5 | B cheaper £800k |
| Performance (10%) | 4 | 5 | B faster p95 |
| Portability (10%) | 4 | 2 | B export gap |
| Support (5%) | 4 | 4 | Tie |
| Weighted | 4.25 | 3.55 | A wins |
Sensitivity: if TCO weight +10%, B still loses on security gate.
Anti-lock-in contract clauses to recommend (topic 30 handoff)
- Data export on termination: embeddings, prompts, logs within 30 days
- Format: open standard (Parquet, JSONL, ONNX where applicable)
- Assistance: person-days included at fixed fee
- Deletion certificate after export
- No penalty for multi-model routing during term
Evaluation feeds contract—not separate concern.
Practice exercises
Primary exercise — Weighted scorecard for bank LLM vendor (75 minutes)
Brief: Regional bank; internal policy copilot; finalists Vendor Alpha and Vendor Beta; PoC results provided (faithfulness Alpha 90%, Beta 88%; Alpha TCO $1.1m 3yr, Beta $0.85m; Beta weaker on audit log export).
Tasks:
- Define 8 criteria and weights totalling 100%.
- Set gate on security/compliance.
- Score both vendors 1–5 with evidence notes.
- Write one-page decision memo — recommend one vendor; document dissent case for the other.
- Draft exit plan bullets for recommended vendor.
Acceptance criteria:
- Weights declared before scores
- Total weighted score calculated transparently
- Recommendation not purely lowest price
- Exit plan mentions export format
Stretch exercise — Full PoC charter and TCO model (half day)
Brief: Insurance claims document extraction; compare build on open stack vs vendor SaaS; 500k documents/year; 3-year horizon.
Tasks:
- PoC charter: scope, metrics, duration, environment.
- Load and failure test scenarios.
- Lock-in assessment for SaaS option.
- TCO spreadsheet narrative: low/base/high for both options including migration at year 3.
- Integration effort estimate for CRM attachment.
- Bake-off agenda for executive session — no unstructured demo vote.
Acceptance criteria:
- At least six PoC pass/fail thresholds
- TCO includes engineering and exit, not only subscription
- Decision criteria pre-registered in charter
Scenario G — Public sector lowest-price tension
Context: Framework mandates MEAT (most economically advantageous tender) but evaluators overweight price 60%; best technical vendor scores 2nd.
Intervention: Clarification letter to contracting authority; cite operational cost of failure; request weight rebalance to 40% price / 35% technical / 25% social value including carbon.
Outcome: Retender with revised weights; technical vendor wins; 5-year value £3.2m vs cheapest £2.1m headline—£890k rework avoided on cheapest vendor's weak export (reference call).
Numbers: Cheapest vendor PoC faithfulness 76% vs winner 91%—MEAT defence documented.
Scenario H — Joint venture SI and IP split
Context: Client contracts JV of SI-A and SI-B; AI prompts written by SI-A; fine-tune by SI-B; dispute at exit.
Prevention in evaluation/contract phase:
- Joint IP schedule pre-signed between SIs before bid
- Client owns all deliverables; SIs assign rights upstream
- Escrow for fine-tune weights
Evaluation must flag JV blur as risk in decision memo—not only post-award legal fight.
Vendor evaluation maturity model
| Level | Characteristic |
|---|---|
| 1 Ad hoc | Demo picks vendor |
| 2 Repeatable | Scorecard exists |
| 3 Defined | PoC charter standard |
| 4 Managed | TCO + exit on every deal |
| 5 Optimising | Post-award eval feeds library |
Aim level 4 minimum for tier-high AI spend >£500k/year.
FAQ — vendor and technology evaluation
Q: How many vendors in a PoC?
A: Two to three finalists maximum—more wastes everyone's time unless RFI poorly filtered.
Q: Can incumbent skip PoC?
A: Only if last PoC <18 months and architecture unchanged; otherwise credibility gap with challengers.
Q: What weight for exit/portability?
A: Minimum 10% for tier-medium+; 15% if regulated or multi-year lock-in risk.
Q: Build vs buy—who decides?
A: Architecture board with business case—evaluation provides evidence, not automatic buy.
Q: PoC failed—now what?
A: Document failure; widen RFI; adjust requirements; do not award based on relationship alone.
Evaluation artefact retention
Keep 7 years or per client policy: scorecards, PoC reports, decision memos, reference notes—supports audit and renewal disputes.
Monthly vendor governance cadence (post-award)
| Month activity | Owner | Output |
|---|---|---|
| Usage vs commit review | FinOps | Variance report |
| Eval regression on vendor upgrade | ML lead | Pass/fail log |
| Support ticket SLA | Service owner | Trend chart |
| Roadmap review | Architect | Deprecation risks |
| Exit drill element | Platform | Export test result |
Feeds renewal at month 18—evaluation never truly ends.
Scorecard anti-patterns to audit
- Weights changed after scores known
- Single evaluator without review
- Demo scored as "functional fit" without harness
- TCO excludes internal engineering
- Reference calls only from vendor-supplied list without churn check
Audit your own scorecards before steering—integrity matters more than winner.
Complete PoC results report outline
- Executive summary — Pass/fail vs charter; recommend award/continue/reject
- Environment — Parity gaps documented
- Functional results — Eval scores table
- Performance results — Latency under load graph narrative
- Security results — Injection test outcomes
- Export test — Screenshots or logs of successful export
- TCO actuals — Consumption during PoC extrapolated
- Lock-in assessment — Red/amber/green by category
- Issues and vendor responses — Open risks
- Recommendation — Aligns with decision memo
Ten sections prevent thin PoC reports that procurement cannot defend.
Weighted scorecard sensitivity analysis example
If security weight 25% → 30%, Vendor A score 4.25 → 4.30; Vendor B 3.55 → 3.40 (failed gate remains). Winner stable—robust decision.
If TCO weight 15% → 25%, Vendor B 3.55 → 3.72 but still below A—document in memo price not sufficient due to export gate failure.
Sensitivity table in appendix satisfies audit that award was not arbitrary.
Build vs buy decision record — required fields
- Problem statement (one paragraph)
- Build effort (weeks × roles)
- Buy TCO (3yr from scorecard)
- Lock-in assessment
- Time-to-value comparison
- Recommendation and review date (revisit if build estimate wrong by >30%)
Attach to architecture repository—prevents permanent buy default without evidence.
Vendor lock-in register — example entries
| Component | Lock level | Exit cost est. | Mitigation |
|---|---|---|---|
| Embeddings | Medium | £80k migration | Parquet export tested |
| Agent DSL | High | £200k rewrite | Abstraction layer in build |
| Model API | Low | £40k retest | Adapter + second provider |
| Observability | Medium | £60k | OTel export mandatory |
Update register at every major vendor decision—feeds renewal and architecture board.
PoC success criteria — numeric examples by use case
| Use case | Faithfulness | p95 latency | Export test |
|---|---|---|---|
| Internal FAQ | ≥85% | ≤3s | JSONL embeddings |
| Regulated RAG | ≥90% | ≤5s | Full config bundle |
| Document extraction | F1 ≥95% | ≤8s | Model + pipeline |
| Agent (read tools) | ≥88% + 0 harmful | ≤6s | Prompt + tool defs |
Pre-register thresholds in charter—post-hoc lowering invalidates bake-off integrity.
Questions you should be able to answer
- What criteria and weights apply before any vendor meeting?
- What gate disqualifies a vendor regardless of total score?
- Does the PoC use production-like data, load and failure modes?
- What is the three-year TCO under low, base and high consumption?
- Where is lock-in—embeddings, prompts, agent runtime, observability?
- What is the documented exit plan and estimated migration cost?
- Did all finalists run against the same eval harness where possible?
- What integration effort weeks attach to each option?
- What did reference calls reveal that demos hid?
- How does the vendor handle model deprecation and notice period?
- Is residency and subprocessor list verified in PoC environment?
- Who scored the bake-off—technical panel before commercial reveal?
- What happens if the vendor is acquired or deprecates the SKU?
- How do FinOps assumptions feed the award decision?
- What is the build vs buy vs partner rationale in one paragraph?
Long-list to short-list — RFI question bank
Use RFI to filter before expensive PoC:
- Describe data residency options per region with subprocessors listed.
- Provide export formats for embeddings, prompts, conversation logs, eval results.
- What notice period for foundation model deprecation?
- Describe audit log contents and retention; SIEM integration.
- Provide reference customers at similar scale in same industry.
- Training data policy—is client data used to improve shared models?
- Pricing model—tokens, seats, capacity; egress fees.
- SLA for harmful output incident response vs API uptime.
- Certifications—SOC2 Type II date, ISO 27001, ISO 42001 if claimed.
- Exit assistance—duration, cost, data deletion certificate.
Score RFI pass/fail gates before inviting to PoC—save 6 weeks.
PoC environment parity checklist
PoC that does not mirror production invalidates results:
| Dimension | Production target | PoC must match |
|---|---|---|
| Identity | SSO + MFA | Same IdP integration |
| Network | Private egress | No open internet shortcut |
| Data classification | Confidential | Representative redaction |
| Load | 500 concurrent | ≥50% of prod target |
| Observability | SIEM export | Same log schema |
| Model deployment | Named deployment | Same SKU/version family |
Document PoC exceptions in decision memo—adjust scores if parity gap material.
Scoring calibration workshop
Before final scores, half-day workshop with technical, security, procurement, FinOps:
- Review gate failures—any disqualification disagreements?
- Normalise 1–5 scale—what does "3" mean with example?
- Blind score review—outliers >2 points investigated
- Sensitivity analysis—if weight ±5%, does winner change?
If winner flips on small weight change, criteria are unstable—fix before award.
Vendor viability and concentration risk
| Signal | Green | Red |
|---|---|---|
| Revenue trend | Growing or stable public parent | Startup burning cash, no funding |
| Customer concentration | Diverse | One client >40% revenue |
| Roadmap | Published, dated | Vapourware demos |
| Support | Named TAM in contract | Email-only |
| Acquisition | Clear product continuity clause | Recent M&A, roadmap silence |
For startup vendors, negotiate source escrow or transition assistance in contract (topic 30 link).
Technology category-specific evaluation notes
Foundation model API: Eval on your domain set; test deprecation notification; compare fine-tune export; measure refusal behaviour on policy queries.
Vector database: Hybrid search quality on noisy data; multi-tenant isolation; backup restore drill; filter performance with metadata.
Agent platform: Tool authZ; max iteration limits; human approval hooks; prompt version control; eval integration in CI.
Observability: OpenTelemetry export; PII scrubbing; eval dashboard or API; retention alignment with privacy.
Governance SaaS: API vs manual UI; integration to ticket workflow; not standalone checkbox.
TCO spreadsheet — row-level guidance
Build TCO with auditable rows:
Licence/subscription (Y1–Y3)
+ Token/compute consumption (low/base/high)
+ Professional services (implementation)
+ Internal engineering (weeks × rate)
+ Training and change
+ Security review and pen test
+ Ongoing ops/on-call allocation
+ Eval re-run labour
+ Egress and storage growth
+ Exit/migration (Year 3 scenario)
- Vendor credits or incentives
= TCO
Present NPV if client finance requires—use their discount rate, not yours.
Competitive intelligence without unethical behaviour
Do: Public docs, patents, conference talks, analyst reports, prior PoC learnings, reference calls.
Do not: Misrepresent competitor; share confidential client data across bids; accept vendor gifts that violate policy.
Document intel sources in decision memo for audit.
Re-bid and incumbent challenger dynamics
When incumbent faces challenger:
- Incumbent advantage: integration sunk cost, known weaknesses
- Challenger advantage: price pressure, modern architecture
- Forced innovation: Incumbent may improve only under credible exit
Run PoC both if incumbent skipped prior eval—relationship is not evidence.
Bake-off facilitation script — 4-hour executive session
| Time | Activity |
|---|---|
| 0:00 | Chair opens criteria; no vendor in room |
| 0:15 | Technical scores presented (blind) |
| 0:45 | Each vendor 20 min structured Q&A—not demo |
| 1:45 | Demo only on common script (optional) |
| 2:45 | FinOps TCO comparison |
| 3:15 | Exit plan comparison |
| 3:45 | Decision or defer with named gaps |
Executive Decide or defer—never "we liked Vendor A vibe."
Post-award vendor management hooks
Evaluation continues after award:
- Quarterly business review — roadmap, incidents, cost
- Eval regression on vendor upgrades
- Exit drill annually—export sample data
Evaluation artefact is living—feeds renewal re-bid at month 18–22 of 36-month deal.
Scenario E — SaaS observability: export trap
Context: Mid-size insurer; selects observability vendor on beautiful eval UI; £95k/year; 18-month contract.
Month 14: Architecture board mandates OpenTelemetry standard; vendor export incomplete—agent spans missing.
Re-eval: Exit cost £140k engineering; 6-month migration; delayed unified SRE.
Lesson: PoC must test export schema, not only dashboards—portability gate at 10% weight minimum.
Numbers: Total cost of wrong choice £95k × 1.2 years + £140k migration = £254k vs alternative £110k/year with full OTel £220k over same period—cheaper alternative wins on 3yr TCO if evaluated correctly upfront.
Extended Learn — bake-off scoring normalization
When evaluators use different scales, normalise before weighting:
- Convert each criterion to z-score across vendors (optional for small sets use rank)
- Apply weights to normalised scores
- Document outlier justifications where one vendor scored 5 and others 2
Prevents charismatic presenter skew on functional fit.
Extended Learn — PoC failure modes catalogue
| PoC failure | Cause | Prevention |
|---|---|---|
| Scripted demo only | Vendor control | Mandatory harness |
| Synthetic data | Client fear | NDA + masked real sample |
| Wrong SKU | Sales promises | Architecture validates deployment ID |
| No security test | Time pressure | Gate: injection test minimum |
| Team swap | Bait-and-switch | Named staff in PoC charter |
Extended Learn — five-year TCO narrative example
Narrative for steering: "Vendor B wins 3-year TCO by £1.1m on licence, but exit at year 3 costs £900k and 26 weeks—Vendor A total 5-year cost £4.2m vs B £4.8m when exit included. Recommend A unless client accepts captive risk."
Numbers tell story procurement can defend to audit.
Extended Learn — open-source vs commercial evaluation
| Factor | Open-source stack | Commercial SaaS |
|---|---|---|
| Licence cost | Low | High |
| Engineering cost | High | Lower |
| Support | Community/self | Vendor SLA |
| Exit | Easier | Contract-dependent |
| Compliance evidence | Self-attest | Vendor certs |
Score total ownership—open-source rarely free at enterprise scale.
Extended Learn — multi-vendor strategy
Some clients adopt best-of-breed: model API A, vector B, observability C.
Evaluation extra: Integration PoC across three vendors—interface contracts between them; blame matrix in runbooks (topic 32).
Risk: finger-pointing in incidents—contract joint support clause or prime integrator.
Extended Learn — vendor diversity and resilience
Dual-vendor strategy for tier-high:
- Primary + secondary model API in production router
- Eval both monthly
- Contract minimum spend split caps concentration
Evaluation includes failover PoC as weighted criterion 10%.
Negative cases — when vendor evaluation fails
Demo-driven award
Symptom: Executive favourite after scripted demo; no common eval.
Impact: Production quality 30–50% below PoC promise; re-procurement within 12 months.
Fix: Common brief and harness; blind technical scoring.
Cheapest licence wins
Symptom: Procurement awards on subscription line; integration €400k over budget.
Impact: Project delay; vendor blame; architecture compromise.
Fix: TCO with integration and ops; FinOps scenarios in award pack.
PoC on clean data only
Symptom: 95% accuracy on curated sample; 71% on production corpus.
Impact: UAT failure; contract dispute.
Fix: PoC data representativeness checklist; sign-off by process owner.
Ignored exit until renewal
Symptom: Year 3 renewal; vendor 2.5x price; no export path.
Impact: Captive client; emergency migration.
Fix: Exit plan at award; export tested in PoC.
Feature checklist scoring
Symptom: Vendor ticks 47/50 features; none used in workflow.
Impact: Complex platform; low adoption.
Fix: Weight functional fit on use-case journey, not feature matrix.
SI bait-and-switch
Symptom: Seniors present; juniors deliver; weak architecture governance.
Impact: Rework; security gaps; timeline slip.
Fix: Contract staffing schedule; reference calls on actual team; milestone quality gates.
Practice checklist
- I defined weights before scoring any vendor
- I designed a PoC with load, failure and export tests
- I calculated TCO beyond licence line
- I documented exit plan for my recommended option
- I completed the primary scorecard exercise
- I can name three lock-in vectors for my stack
- I linked evaluation to FinOps and contracting (topics 22, 30)
Related playbook content
- RFP, Procurement and Contracting — Contract terms reflect PoC thresholds
- AI FinOps and Commercial Design — Consumption and TCO modelling
- Performance Engineering and AI FinOps — Load and cost at scale
- Enterprise Architecture — Standards and integration patterns
- Security Engineering — Security gate criteria
- Responsible AI and Governance — Vendor risk tier alignment
- MLOps, LLMOps and Observability — Observability export requirements
- Presales and Solution Shaping — Competitive positioning discipline
- How to use this Learning Map — Reference-depth study method
- 8D Framework — Build vs buy gates
Discussion
Comments
Share feedback or questions about this page. No account required.
Loading comments…