Single-turn Q&A is relatively easy: the user asks a question, the AI answers. Multi-turn conversation is harder: the user asks a follow-up that references the previous answer — "and now by month", "only for Shanghai", "what about last year?" — and the AI must maintain context across turns. This is what separates a useful BI assistant from a glorified search engine.
What Is the Context Window Problem?
The context window is a finite resource, and every previous turn consumes it. Large language models have a context window — the amount of text they can process at once — and in a multi-turn conversation, each prior turn, query, result, and schema snippet eats into it. After 5–10 turns, the model starts "forgetting" earlier context, which is exactly when users start repeating themselves and losing trust.
The numbers explain why. A single query result rendered as text can consume thousands of tokens, and enterprise schemas with hundreds of tables cannot fit alongside a long conversation at all — even 128K or 200K token windows fill quickly when every turn drags the full history along. The solution is architectural: maintain a structured conversation state outside the model and inject only the relevant context for each new turn. The model should never carry the whole conversation; the application should.
How Should an AI Resolve 'That' and 'By Month'?
The hard part of multi-turn BI is resolving what "that", "it", and "by month" refer to. When a user says "now show that by month", the AI must resolve "that" to the previous query's subject — revenue by region, say — which requires tracking the conversation's entity graph: what metrics, dimensions, and filters have been discussed, and which are active in the current turn.
Reference resolution is where naive systems fail in visible, embarrassing ways. A user who asks "revenue by region for Q3" and then "now filter to electronics" expects the electronics filter applied to the Q3 revenue-by-region query; a system that guesses wrong produces a confident, wrong answer. The resolution layer should also handle implicit carry-over — a user asking "and what about inventory?" mid-conversation expects the same dimensions, time range, and filters to apply until overridden. When the reference is genuinely ambiguous, the right behaviour is a clarifying question, not a guess: "which region did you mean?" costs one second; a wrong chart costs a meeting.
Enterprise glossaries make resolution tractable. When metric names, dimension values, and synonyms are defined in one place — "revenue" versus "net revenue", "region" versus "territory", the three different "month" conventions across business units — the resolver can map user language to the schema deterministically instead of guessing from the conversation alone. Glossaries also catch the silent failures: a user in one business unit using a term that means something different in another gets a clarification, not a confidently wrong chart.
How Should Context Be Managed Across Turns?
The winning pattern is externalising memory: keep the conversation state in an object, not in the prompt. Effective context management follows three steps: (1) store each turn's query, result, and metadata in a conversation state object; (2) for each new turn, generate a compact context summary rather than the full history; and (3) include the summary, the current question, and any specific entities referenced.
The summary should be structured, not a paraphrase: active metrics, active dimensions, active filters, time range, and the last analytical action. That structure is what makes reference resolution deterministic — "that" resolves against the entity graph, not against a wall of text. The approach also keeps token usage flat as conversations lengthen: a 20-turn conversation uses roughly the same context as a 5-turn one, because what is injected is state, not history.
Design the failure path explicitly. When the state cannot be resolved — an ambiguous reference, an entity that does not match any active dimension — the system should say so and ask, rather than fabricating a plausible context and answering anyway. Graceful degradation also means letting the user correct the record: "that wasn't what I meant" should update the conversation state and re-run the query, which turns a wrong turn into a learning signal instead of a dead end.
When Should Conversation Context Be Reset?
Not all turns belong to the same analytical thread, and the system must know when to start fresh. If a user switches from "revenue analysis" to "headcount trends", the AI should detect the topic shift and reset the analytical context — while keeping the conversation history available if the user returns to the previous topic.
Topic detection is a design decision as much as a model capability. Clear signals — a new metric family, an explicit "switch to", a different time horizon — trigger a reset; ambiguous ones trigger a confirmation. Sessions also need natural expiry: a question asked an hour after the last one is a new analysis, not a follow-up, and the state object should reflect that. The goal is a conversation that behaves the way a good analyst does — remembers what you asked, knows when you have moved on, and never makes you repeat yourself.
How Should You Design Multi-Turn BI Conversations?
Design for the way analysts actually work: ask, refine, drill down, compare. A multi-turn BI assistant earns its place not by answering one clever question but by sustaining a ten-turn analytical dialogue — which requires the four components described above: externalised state, reference resolution, compact summaries, and topic detection.
Deployment is where good design becomes a product. The assistant must live where the conversation already happens — in IM-native tools like WeChat Work — because the value of multi-turn analysis compounds when the follow-up is as easy as typing another sentence between meetings. Enterprises adopting conversational BI typically see analysis time drop sharply, because the analyst-to-insight loop loses its handoffs and its waiting periods. With a two-week deployment and a managed service, the context-management layer is delivered, tuned, and maintained — so the hard engineering of multi-turn conversation is invisible to the people who benefit from it.
Evaluate the assistant the way you would evaluate an analyst: on tasks, not on vibes. Give users a standard analytical workflow — a five-step drill-down with filters, comparisons, and a topic switch — and measure completion, correction rate, and time-to-answer against the pre-AI baseline. Turn-level accuracy matters, but the metric that decides renewal is whether a business user can complete a real analysis in the IM thread without calling for help.
What Are the Key Takeaways?
Multi-turn context is an engineering problem with known solutions. These are the design choices that make a BI assistant feel like an analyst rather than a search box.
- Externalise memory: conversation state lives in an object, not in the prompt — so context does not degrade as turns accumulate.
- Resolve references deterministically: an entity graph of metrics, dimensions, and filters makes "that" and "by month" precise.
- Inject state, not history: compact structured summaries keep token usage flat across long conversations.
- Detect topic shifts: reset the analytical context on topic change while keeping history retrievable.
- Meet users where they talk: IM-native conversational BI makes multi-turn analysis a natural extension of the workday.
What Should Teams Build First?
Multi-turn conversation is the difference between a BI tool and a BI analyst. The mechanics — context windows, reference resolution, state management, topic detection — are engineering problems with proven solutions, and the user experience they enable is the one people actually want: ask, refine, drill down, and get an answer that remembers what you meant.
The enterprises that win with conversational BI will be the ones that treat context as a designed system rather than an accident of prompt length. With the state machine handled properly, a finance team can hold a fifteen-turn dialogue about margins in the middle of a WeChat Work thread — and the AI will still know exactly what "that" means at turn fifteen.
How Should Conversation State Be Modelled?
Externalising memory is the right instinct, but an ad-hoc state object tends to accrete fields until nobody is sure which one is authoritative. A cleaner model treats the conversation state as a small, typed structure with four slots, each of which has an explicit precedence rule when a new turn arrives.
- Subject — the metric or measure under discussion: revenue, margin, headcount, units shipped. A new turn that names a metric replaces the subject; a turn that does not leaves it in place.
- Dimensions and filters — region, channel, product, time period, and any explicit constraints. These are the most frequently modified part of the state, and they are what "by month" or "only for Shanghai" actually changes.
- Operation — comparison, ranking, trend, decomposition, or threshold check. This is the part most state models omit, and its absence is why an assistant that handled "compare to last year" well then fails on "which regions are worst?"
- Result pointers — references to the previous result sets, so that "of those, which grew fastest?" can be resolved without re-running the original query.
The precedence rule matters more than the structure. When a user says "now break that down by channel, but only for the top three regions," three slots change at once and one stays. Systems that resolve slot-by-slot with a defined order — filters before dimensions, dimensions before subject — behave predictably; systems that re-derive the whole state from the latest utterance produce the erratic behaviour users describe as the assistant "losing the thread."
What Breaks Multi-Turn Conversations in Practice?
Beyond context-window exhaustion, four specific problems recur in production deployments, and each is solvable with a specific technique.
Ambiguous numeric references. "What about last year?" could mean the same period last year, the full prior year, or a year-over-year comparison at the current granularity. Rather than guessing, the highest-quality behaviour is to resolve the most probable interpretation, state it explicitly in the answer ("same period last year, January to March"), and offer the alternatives. Stating the interpretation costs one sentence and eliminates most silent misinterpretation.
Drill-down chains that exceed the schema. A user asks for revenue by region, then by city, then by store, then by product — and the fourth level does not exist at that granularity. The correct response is not an error or a hallucinated chart but a graceful boundary statement: what the finest available granularity is, and an offer to approximate it. Handling the edge of the schema well is a large part of what makes an assistant feel competent.
Topic drift inside a single thread. A conversation about margins slides into a conversation about headcount and back again. Topic detection should therefore be soft rather than binary: maintain a stack of analytical contexts rather than a single one, so that returning to a previous topic restores its state instead of starting again.
Permission leakage across turns. As the conversation state accumulates filters and entities, it can accumulate access to data the user should not see in combination — a filter that was legitimate in one context exposing an aggregation that is not permitted in another. Every resolved query must be re-authorised against the full accumulated state, not just against the newest turn.
How Do You Test a Multi-Turn Assistant?
Single-turn evaluation misses most of what goes wrong. The unit of testing has to be the conversation, not the question, and the practical approach is a scripted dialogue suite: twenty to fifty multi-turn transcripts covering the patterns users actually produce — refinement, drill-down, comparison, topic switch, return, and correction — each with known-good final states and expected answers.
Three metrics come out of running that suite. State accuracy: after N turns, does the resolved query specification match the expected one? Answer accuracy at depth: accuracy measured separately at turn one, turn five, and turn fifteen, because the whole point of the architecture is that accuracy should not degrade with depth — if it does, the externalised state is not working. Recovery rate: when the user corrects the assistant ("no, I meant by quarter"), how often does the next turn get it right?
Add one production signal that is cheap and highly diagnostic: the rate at which users abandon a thread and start a new one. High abandonment after three or four turns is the clearest indicator that context handling is failing, and it is visible in logs without any labelling effort.
What Breaks Context in Multi-Turn BI, and How Do You Fix It?
Multi-turn analytics fails when the system forgets what was already said. The usual culprits are stateless request handling, where each question is treated as isolated; silent assumption changes, where "last quarter" means different things across turns; and lost references, where "compare it to the North America region" has no anchor because the previous turn was dropped. Users experience this as having to repeat themselves or, worse, receiving a confident answer to the wrong question.
The fix is to treat context as first-class state. Maintain an explicit session object that records the resolved entities, filters, time ranges, and metric definitions from earlier turns, and carry it into every new query so follow-ups build on established ground. When a reference is ambiguous, resolve it against that state rather than guessing, and confirm with the user when genuinely unclear. Periodically summarise long sessions into a compact context snapshot so the model does not lose the thread as the conversation grows. With state handled deliberately, a ten-turn investigation feels like working with an analyst who was in the room the whole time.
