The build-versus-buy question for conversational BI is no longer "can our engineers make a chatbot answer questions about revenue" — most can — but "will the thing we build still be correct, secure and maintained in year three", and that is a cost question almost nobody prices honestly.
The question changed, and most evaluation frameworks did not
Five years ago the build-versus-buy debate in analytics was about dashboards: could packaged tools express your KPIs. In 2026 the object of comparison is an answer engine — a system that takes a natural-language question from a sales director in a WeChat Work group and returns a governed, correctly scoped, permission-respecting number with provenance. That reframing dissolves the old intuition. Dashboards are visual; answer engines are decision infrastructure. The comparison is not between two products but between two ongoing obligations.
The trap in most internal build-versus-buy decks is that they compare the vendor's price tag against the internal team's initial build estimate, and stop there. Both numbers are usually roughly right and both are roughly irrelevant. The honest comparison is three-year total cost of ownership including failure modes — and the failure modes are asymmetric. A bought platform that disappoints can be replaced at contract renewal. A half-finished internal platform that disappoints has already absorbed eighteen months of your scarcest people, and the political cost of admitting that usually exceeds the technical cost of finishing it badly.
What "build" actually involves: the stack beneath the demo
The demo takes two weeks: wire an LLM API to a schema, paste in a few example questions, and it works convincingly. The platform that survives enterprise contact has at least seven components, each a real engineering commitment:
- Schema and metadata layer: curated table and column descriptions, join paths, and business context so the model knows that "GMV" excludes cancelled orders. This is not a one-off document; it is a living asset that decays as pipelines change.
- Semantic layer: canonical metric definitions (revenue, active customer, margin) implemented once so that every answer draws from the same governed calculation. Without it, the chatbot re-derives metrics per question and eventually disagrees with the finance deck.
- Query generation and execution path: text-to-SQL or a semantic-API call layer, with dialect handling, query guards, timeouts and result caching.
- Evaluation harness: a versioned set of hundreds of real business questions with known-correct answers, run on every model or prompt change. This is the single most underestimated component; without it you cannot tell whether last week's model upgrade improved or silently degraded accuracy.
- Permissions and row-level security: the answer engine must respect the same entitlements as the underlying warehouse, per asking user, per query. Getting this wrong is a data-leak incident, not a bug ticket.
- Audit and observability: every question, generated query, result and correction logged, for both performance tuning and the compliance conversation that arrives in year two.
- Deployment surface: integrations into WeChat Work, DingTalk, Feishu, Teams or WhatsApp — each with its own auth flow, message limits and admin consoles.
Two-to-four senior engineers can assemble this. Keeping it correct while your warehouse, models, business definitions and LLM vendors all keep moving is the actual job, and it never ends.
The hidden costs of build, itemised
The costs that sink internal builds are the recurring ones, because they are staffed by the people the organisation can least afford to redirect.
The semantic layer is a permanent programme, not a project. Defining fifty metrics properly with finance and operations takes months of meetings; keeping those definitions current as the business changes is a standing tax. Teams that skip it ship an analytics chatbot whose numbers drift from the official reporting — and one publicised discrepancy in an executive meeting ends the internal tool's credibility faster than any technical failure.
The evaluation harness is the difference between an asset and a liability. Industry experience through 2024–2025 consistently shows answer accuracy degrading in unmeasured ways after model upgrades, prompt edits and schema changes. A serious harness costs one engineer-month to start and continuous maintenance thereafter. Teams without one navigate by anecdote: the CFO found an error, so nobody trusts the tool.
Security review is where many builds die quietly. A system that reads the data warehouse and writes SQL has to clear application security, data governance and often a penetration test. In regulated sectors — financial services, real estate with tenant PII — add model-risk review. Six months of review latency is common, and it is rarely in the build plan.
Maintenance is three coupled upgrade treadmills, not one. Your LLM provider deprecates models on its schedule; your warehouse changes drivers and query APIs; your IM platforms revise bot APIs. Someone owns all three simultaneously, forever.
| Hidden cost | First-year reality | Years 2–3 recurring | Common failure symptom |
|---|---|---|---|
| Semantic layer and metric dictionary | 3–6 engineer-months with business input | 0.5–1 FTE ongoing | Chatbot numbers disagree with finance |
| Evaluation harness | 1–2 engineer-months to bootstrap | 0.25–0.5 FTE | Accuracy drifts unnoticed after upgrades |
| Security and model-risk review | 3–6 months elapsed, 0.5–1 FTE in meetings | Re-review per major change | Launch slips past two quarters |
| IM channel integrations | 1–2 engineer-months per channel | API breakage upkeep | Tool lives in one group chat only |
| Model and platform upgrades | Variable | 0.5–1 FTE combined | Nobody dares to upgrade the model |
| On-call and incident response | Underestimated | 0.25–0.5 FTE | Silent wrong answers, no alerting |
Sum the recurring column for a mid-sized build and you are typically looking at 2–3 FTE permanently, plus elapsed-time risk on review. That is the number the build-vs-buy deck should carry — and it is the number most decks omit.
The buy side is not free of risk either
Honesty cuts both ways. Buying a conversational BI platform moves the engineering burden out but imports four distinct risks that deserve explicit diligence:
- Metric governance dependency. If the vendor's semantic definitions cannot express your business logic — the way your brokerage computes net fee revenue, or how your factory defines on-time-in-full — you inherit either a configuration project or silent misdefinition. Test this in the pilot with your ten hardest real questions, not ten demo questions.
- Data residency and security model. Where does the vendor's inference run, what is logged, and does the architecture require your data to leave your tenant? In Hong Kong and the mainland GBA, cross-border data posture is a board-level question, not an IT detail.
- Vendor lock-in through the semantic layer. Ironically, the same component that makes build expensive makes buy sticky: once your metric dictionary lives in the vendor's format, migrating means re-definitions and re-validation. Mitigate by demanding export of definitions and query logs in open formats, contractually.
- Model churn you do not control. The vendor will swap underlying models for cost and capability reasons. Your protection is their evaluation discipline — ask to see it, and ask what notice you get before a model change reaches your answers.
Pricing structures in 2026 vary widely — per-seat, per-question, per-connection — and each shapes behaviour differently. Per-question pricing punishes the experimentation that drives adoption; flat licensing rewards heavy usage but can strand value if adoption stalls. Model the pricing against your actual question-volume forecast, not against the vendor's reference customer.
A worked 3-year TCO example
The table below models a mid-sized enterprise: roughly 500 named users, three IM channels, an existing cloud warehouse, and a data team of 8–12. Ranges are illustrative estimates drawn from 2024–2026 industry benchmarks and should be recalibrated to your salary bands and vendor quotes — the shape of the comparison matters more than the exact figures.
| Cost line (3-year, USD) | Build (in-house) | Buy (platform) |
|---|---|---|
| Engineering build-out (semantic layer, query path, eval, integrations) | 450,000–750,000 (2–4 FTE × 12–18 months) | Included in implementation, 30,000–60,000 |
| Implementation services | 50,000–100,000 internal effort | 30,000–60,000 |
| Licences / infrastructure / model inference | 60,000–150,000 | 150,000–350,000 |
| Steady-state maintenance (years 2–3) | 400,000–900,000 (2–3 FTE) | 50,000–120,000 (0.25–0.5 FTE admin) |
| Security, risk and compliance programme | 80,000–200,000 of internal time | 30,000–80,000 (shared with vendor attestations) |
| Re-platforming / failure risk provision | 150,000–400,000 expected-value cost | 0–60,000 (exit/renewal effort) |
| Indicative 3-year total | 1.19M–2.50M | 290,000–730,000 |
Three honest footnotes to that table. First, the build column's range is wide because scope creep is the norm: most internal builds grow from "one team's questions" to "company-wide" without a proportional budget conversation. Second, the buy column assumes the platform can express your metrics — if it cannot, configuration costs rise and the comparison narrows. Third, neither column includes the option value of the data team's time: every FTE spent maintaining an internal answer engine is an FTE not building the demand forecasting model. On most CDO scorecards, that opportunity cost is the largest line item of all.
The baseline you are actually comparing against
Every build-versus-buy analysis shares an unexamined competitor: the status quo. The comparison as usually framed assumes the alternative to the new platform is zero cost, when in reality the organisation is already paying continuously for a broken question-and-answer loop. Making that baseline visible changes the arithmetic and, more importantly, changes who in the room supports the change.
The components of the status-quo cost are identifiable and countable. The analytics backlog: most data teams above a certain size run a de facto ticket queue of report and extraction requests, and industry experience through 2024–2025 consistently places the share of analyst time consumed by repetitive ad-hoc requests in the 30–50 percent range. Dashboard sprawl: large enterprises routinely accumulate thousands of reports with no single source of truth, and a meaningful fraction are silently stale. Shadow analytics: finance and operations staff maintaining private spreadsheet models that answer governed questions with ungoverned numbers — the audit risk nobody has budgeted. And decision latency: the churn analysis that took three weeks, the pricing review that waited for month-end. None of this appears in either column of a conventional TCO table, yet it is usually the largest number on the page.
Conversational BI's economic function is precisely to shorten the path between a business question and a governed answer, and the IM-native form matters here: the question gets asked where the conversation is already happening, in the WeChat Work or Teams thread, and the answer arrives with provenance in the same thread. Every governed question answered in-channel is a ticket that never entered the backlog and a spreadsheet that never got born.
The honest reframing, then, is this: the correct comparison is not build cost against licence cost. It is status-quo cost plus chosen-option cost, against status-quo cost plus the other option. Since the status-quo term is common to both sides, the TCO table from the previous section still decides the choice — but the status-quo figure determines whether the project deserves to happen at all, and it is the line item executives should demand to see quantified: request tickets per quarter, average handling hours, loaded analyst cost, plus a conservative estimate of decision-latency value. In our client assessments, that combined number routinely exceeds both the build and buy columns by a multiple.
The contract terms that decide year-three economics
Because buy-side risk is contractual, the negotiation checklist matters as much as the product evaluation. Five clauses separate a workable three-year relationship from a slow hostage situation:
- Definition and data export rights. Semantic or metric definitions, question logs and evaluation datasets must be exportable in documented, non-proprietary formats, at no penalty, contractually. This single clause converts lock-in from a structural problem into a commercial one.
- Model-change notice. Thirty days' written notice before a model change affects your answers, plus the vendor's evaluation deltas on your own question set. You are buying their eval discipline; inspect it.
- Data processing and residency schedule. Named inference regions, explicit no-training-on-your-data language, and logging posture documented. For GBA enterprises this is the clause the security committee actually reads.
- Accuracy remedy. A true accuracy SLA is rare in 2026 — but how a vendor responds when asked reveals its maturity. A vendor that says "we publish accuracy benchmarks and give you the harness to verify" is a fundamentally different counterparty from one that says "the AI is probabilistic".
- Exit provisions. Transition assistance, defined data-deletion timelines and certificates, and a price schedule for a one-year wind-down. Exit terms negotiated at signature cost nothing; negotiated at renewal they cost leverage.
Run the pilot before signing anything, because the pilot window is your leverage window: afterwards the leverage flips to the vendor. Insist the pilot runs on your real data, under your real permission model, against the ten-question test described earlier, with success metrics agreed in writing before it starts.
The build path needs its own contractual analogue: an internal charter. Name a single product owner, ring-fence the permanent maintenance funding as its own budget line rather than a draw on team goodwill, and write sunset criteria — the conditions under which you would retire the internal build and buy instead. Internal platforms rarely die from bad code; they die from diffuse ownership, where everyone uses the tool and nobody is accountable for keeping it correct.
Decision guidance by data team size
The right answer correlates strongly with the size and mandate of your data organisation, more than with budget.
- Data team under ~10 people (analytics and engineering combined, supporting the whole company): buy, almost unconditionally. A build consumes 20–30 percent of your entire capacity on non-differentiating infrastructure. Your differentiation is the semantic definitions and the decisions they enable — buy the plumbing, own the definitions, insist on exportability.
- Data team of 10–40 with a platform mandate: hybrid. Buy the conversational layer and IM-native deployment; invest the saved engineering capacity in a rigorous internal semantic layer and evaluation practice. Negotiate the right to bring your own semantic definitions or metric store. This is where most 2026 enterprises with real data organisations land.
- Team of 40+ with regulatory constraints, unusual data residency needs, or genuinely novel question types: build becomes defensible, but scope it honestly — build the answer engine only for the domain where packaged products provably fail, and buy everywhere else. And budget the maintenance line as a permanent product team, because that is what it is.
One decision test cuts through every deck: pick your ten hardest real questions from the last quarter — the ones involving metric edge cases, permission nuance, and cross-domain joins. Run them through the vendor pilot and through your best internal estimate of the build. Compare not just correctness but the elapsed time and the number of people involved. In our deployment experience, the ten-question test predicts the three-year outcome better than any spreadsheet.
A 30-day decision process you can actually run
Compress the evaluation into four weeks to avoid the six-month committee drift that kills these decisions. Week one: inventory the questions that matter — pull the real query logs or top report requests, cluster them into question families, and agree the metric definitions sheet with finance. Week two: run the vendor pilot against those questions inside your actual IM environment with real permissioning, and score accuracy, permission behaviour and answer latency. Week three: build the honest TCO from the worked example above with your own salary bands, and take it to the CIO, CFO and security lead simultaneously — not sequentially. Week four: decide and write the decision memo including exit criteria for whichever path you chose. The memo matters: teams that write down the conditions under which they would reverse the decision make calmer choices at renewal time, and the discipline of writing exit criteria forces the lock-in questions to the surface while you still have negotiating leverage.
