API-First Data Integration: Connecting Legacy and Modern Systems: Part 2

API-first data integration is the discipline of treating every legacy source as a versioned product with a published contract, rather than as a database to be queried directly — and it is the difference between an integration programme that compounds and one that collapses under its own exceptions. Most enterprises already know the cost of the alternative. Gartner's long-running estimate that poor data quality costs organisations an average of $12.9 million per year is not really a data-quality number; it is an integration-design number. When every consumer reaches into a source system on its own terms, no one owns the definition, no one owns the break, and every downstream report becomes a negotiation. Part 1 of this series covered why the API-first mindset matters. This part is about the mechanics: how you actually get from a mainframe, a thirty-year-old ERP, and a dozen departmental databases to a governed, real-time data fabric that a business user can interrogate in plain language.

The stakes are concrete. Integration debt does not show up as a line item; it shows up as a six-week wait for a new report, a nightly batch window that overruns into the business day, and two executives quoting different revenue numbers because they queried different copies of the same table. An API-first fabric fixes the plumbing once, then lets every consumer — BI, machine learning, operational dashboards, and increasingly, conversational analytics — work from the same governed contract.

Why Do Legacy Integration Programmes Stall?

Integration programmes rarely fail for technical reasons. They fail because of three recurring structural mistakes, and each one is a design choice rather than a constraint.

The first is point-to-point sprawl. A team needs customer data, so it writes a direct JDBC connection to the billing database. Another team needs the same data and writes its own extract. Within two years there are forty connections into eleven systems, each with its own refresh schedule, its own filtering logic, and its own definition of an "active customer." Nobody can safely change the billing schema, because nobody knows who depends on it. The cost is not the connections; it is the paralysis they cause in the source systems.

The second is batch thinking. Nightly extracts made sense when compute was expensive and the business could wait a day. They make no sense when a fraud decision has to be made in 200 milliseconds or a store manager needs to know current stock before promising a customer a pickup time. Batch also hides errors: a job that fails at 02:00 is discovered at 09:00, and by then the window for correction has closed.

The third is schema coupling. When consumers depend on the physical structure of a legacy table, every source-system upgrade becomes an integration project. The fix is a published contract that sits between the two — an interface the source team commits to maintaining and the consumer team commits to using, with explicit versioning, so either side can change independently as long as the contract holds.

• Symptom: every new report requires a new pipeline. Cause: no reusable contract layer.
• Symptom: numbers disagree between departments. Cause: per-consumer business logic embedded in extracts.
• Symptom: source upgrades are frozen. Cause: direct schema coupling with no ownership boundary.
• Symptom: incidents are discovered by business users. Cause: no contract-level observability.

What Does an API-First Data Contract Actually Contain?

A data contract is not a Swagger file with a table name in it. A useful contract has five parts, and skipping any of them pushes cost downstream.

Schema and semantics. Field names, types, nullability, and — critically — meaning. "customer_status = 3" is not a contract; "customer_status: one of ACTIVE, SUSPENDED, CHURNED, where CHURNED means no invoice issued in 180 days" is a contract. This is where most programmes under-invest, and it is exactly the layer a semantic model later consumes.

Versioning and compatibility policy. Semantic versioning with an explicit promise: patch changes are additive and backwards compatible; minor changes add optional fields; major changes break and require migration with a published deprecation window. Without this, every change is a negotiation.

Service-level objectives. Freshness (data is at most N minutes behind the source), availability (99.9% of requests succeed), and latency (p95 under X ms). These are measurable, and they turn "the data is wrong" from an argument into a ticket.

Ownership and change process. A named producing team, a named consuming steward, and a review step before any breaking change ships. The review is where the semantic definitions get argued out — once, in the open, rather than forty times in forty reports.

Access and entitlement rules. Who may read which fields, enforced at the contract boundary rather than in each consumer. This is the difference between row-level security that works and row-level security that is re-implemented — and subtly broken — in every downstream tool.

How Do You Expose a Legacy System Without Rewriting It?

The common fear is that API-first implies a rewrite. It does not. There are three established patterns, and mature programmes use all three depending on the system.

Change data capture (CDC) with an outbox. For transactional systems of record, read the database's transaction log rather than querying tables. Log-based CDC captures inserts, updates, and deletes without adding load to the source, and it preserves ordering. The captured stream lands in a log (Kafka, Kinesis, or a managed equivalent), and a connector service exposes it as a versioned API. The source system is untouched; you have added a tap, not a rewrite.

The strangler-fig facade. For monoliths with no usable log, put an API gateway in front of the existing interface and migrate routes one at a time. New consumers call the facade; the facade forwards to the legacy path until a route is reimplemented, then switches. The name comes from the fig that grows around a host tree: over time the new implementation replaces the old one without a cutover event. This is the lowest-risk pattern for systems nobody fully understands any more.

File-drop bridges for the genuinely unintegratable. Some systems can only emit a nightly flat file. Accept it, but treat the file as a transport detail: parse it immediately, validate it against the contract, reject on schema violation, and publish the result to the same API surface as everything else. The consumer never learns that the source is a mainframe extract, and when the source is finally upgraded, only the bridge changes.

The decision criteria are simple: prefer CDC where the log is available and stable; use a facade where the interface is the only stable surface; reserve file bridges for sources you intend to retire. Do not build a custom extract where a standard connector already exists — that is where maintenance cost accumulates fastest.

Should Integration Be Event-Driven or Request-Response?

Both, and the split should be deliberate rather than accidental. The test is whether the consumer needs the current state or the history of changes.

Request-response (REST or gRPC) suits lookups: "what is this customer's current credit limit?", "what is the stock level for SKU X in store Y?" These are synchronous, cacheable, and easy to reason about. They also fail in predictable ways, which makes them straightforward to secure and monitor.

Event streaming suits propagation: an address change should reach billing, fulfilment, and CRM without each of them polling. Events decouple producer from consumer — the producer publishes "customer address changed" and does not know or care who listens. New consumers can be added months later and replay the log to reconstruct state.

The failure mode to avoid is using events as a database. If a consumer must know current state, give it a materialised view built from the event log, not a replay of the full history on every request. And if a consumer needs an answer that spans several systems, do not make it call five APIs and join in memory — that is a distributed join with five failure modes. Put the join behind a composite service, or better, behind a semantic layer that already knows how the entities relate.

How Does API-First Connect to Conversational Analytics?

This is where the architecture pays for itself in business terms. A governed API fabric solves the plumbing; it does not solve access. Business users still cannot query an API, and they will not learn to. The result is the familiar queue: every question becomes a ticket, and the analytics team becomes a bottleneck measured in days.

Conversational analytics is the access layer that sits on the contract layer. Because every source already publishes a versioned, semantically documented contract, a query engine can answer "which customers downgraded last quarter and what was their support contact volume in the preceding 60 days?" by composing two published services rather than by asking an engineer to write a join. Because entitlement rules live at the contract boundary, the answer respects the same permissions the API would enforce — a regional manager asking the question sees their region's rows and nothing else.

Concretely, Beehive Strategy connects the API-first fabric described above through MCP connectors and a semantic layer, so data stays where it is while queries unify it in real time. Users ask questions in the messaging tools they already use — Microsoft Teams, Slack, WhatsApp — and receive grounded answers in seconds, with the underlying SQL and source contracts visible for audit. The platform deploys as a managed service in about two weeks, because the hard part — the contracts — is already done. That is what turns an integration architecture into a decision advantage rather than a cost centre.

What Governance Does an API-First Fabric Require?

Governance in an API-first world is lighter than in a warehouse-only world, because it is enforced at fewer points. Four mechanisms do most of the work.

A schema registry as the source of truth. Every contract is registered, versioned, and validated in CI. A pull request that changes a field type without a major version bump fails the build. This single check prevents the majority of breaking-change incidents.

Authentication and authorisation at the edge. OAuth 2.0 for service identity, short-lived JWTs carrying the caller's entitlements, and enforcement at the gateway rather than in each service. Centralising this means an access review is a configuration change, not a code audit across forty services.

Lineage from contract to consumer. When a definition changes, you need to know who is affected before you ship. Lineage captured at the contract level gives you that list automatically — and it is the same metadata that lets a conversational engine explain where a number came from.

A deprecation policy with teeth. Publish deprecation dates, emit warnings to consumers of deprecated versions, and actually retire them. Contracts that are never retired accumulate as orphaned services that nobody dares to turn off.

How Do You Keep Legacy Systems From Being Overwhelmed?

The oldest systems are usually the most fragile under load, and a new API surface can multiply traffic by an order of magnitude. Four defences, applied in order:

• Caching at the contract boundary. Read-through caches with explicit TTLs derived from the freshness SLO. A reference-data lookup that changes weekly should not hit the mainframe on every request.
• Rate limiting per consumer. Quotas prevent one runaway dashboard from consuming the capacity the order-entry system needs. Make the limits visible so consumers self-correct.
• Circuit breakers and bulkheads. When a legacy system degrades, fail fast and return a stale-but-labelled answer rather than letting threads pile up. A dashboard that says "data as of 14:32, source degraded" is far more useful than a spinner.
• Read replicas and CDC taps instead of direct reads. Never let analytics traffic touch the transactional path. If the only option is a direct read, schedule it outside business hours and treat it as technical debt.

The target is not zero load on legacy systems; it is predictable, bounded, observable load. A mainframe that serves 200 requests per second from a cache is happier than one that serves 20 unpredictable queries.

What Does a Realistic Migration Sequence Look Like?

Sequencing determines whether the programme survives its first budget cycle. A workable sequence runs in five stages, each delivering something the business can feel.

Stage 1 — inventory and triage (2–3 weeks). Catalogue every source, every consumer, and every current extract. Score each source on change frequency, business criticality, and integration difficulty. You will usually find that 20% of sources carry 80% of the analytical value; start there.

Stage 2 — the first two contracts (4–6 weeks). Choose one high-value, technically tractable source. Publish the contract, stand up CDC or the facade, and migrate two or three existing consumers onto it. Retire their old extracts. The retirement matters: without it you have added a pipeline rather than replaced one.

Stage 3 — access layer (2 weeks). Put conversational analytics on the published contracts and let business users ask real questions. This is the stage that converts executive sceptics, because it is visible.

Stage 4 — breadth (3–6 months). Work through the triage list, publishing two to four contracts per sprint. Resist the urge to standardise everything before delivering anything; the standard emerges from the second and third contracts, not from a design document.

Stage 5 — decommission. Turn off the point-to-point connections and nightly extracts that the fabric replaced. Track this explicitly as a metric, or it will never happen.

Which Metrics Tell You Whether It Is Working?

Vanity metrics ("APIs published") tell you nothing. Six measures do:

• Time to onboard a new data source — target: under one sprint, down from months.
• Time to answer a new business question — target: minutes for questions covered by existing contracts.
• Number of direct source connections retired — the honest measure of whether you are consolidating.
• Contract SLO attainment — percentage of time freshness, latency, and availability promises are met.
• Breaking-change incidents per quarter — should trend toward zero as CI validation matures.
• Analyst hours spent on data preparation — the number the CFO cares about; typical reduction is 40–60%.

Publish these monthly. Integration programmes lose funding when they cannot demonstrate progress, and these six make progress legible.

What Are the Most Common Pitfalls?

Building an API for every table. An API that mirrors the physical schema has inherited the problem it was meant to solve. Model the domain entity, not the table.

Versioning everything as major. If every change is breaking, versioning conveys no information. Invest in additive compatibility.

Treating the contract as documentation rather than as code. Contracts that are not validated in CI drift within weeks.

Ignoring the delete case. CDC captures deletes only if the source records them. Soft-delete-only sources produce APIs that resurrect dead customers.

Letting the access layer wait. Programmes that defer business access until "the fabric is complete" usually run out of patience first. Put a working question-answering layer on the first two contracts and grow it.

No plan for the file-drop systems. They are unglamorous and they are often the ones carrying regulatory reporting. Bridge them early, so they are not the reason the programme cannot decommission anything.

Where Should You Start Next Week?

Pick the source that appears in the most arguments. Every organisation has one — a revenue figure, a customer count, a stock position — where two teams routinely disagree. That disagreement is a contract waiting to be written. Draft it with both teams in the room, publish it, and put the question to a conversational interface so anyone can verify the answer. The first contract is the hardest because it forces the semantic argument that everyone has been avoiding; the tenth is routine, because the pattern, the CI validation, and the access layer already exist. API-first is not a migration you finish — it is a capability you acquire, and it starts with one honest definition.