A modern factory with 1,000 machines, each generating 50 data points per second, produces over 4 billion data points per day. Without the right architecture, this data is noise. With it, this data becomes predictive maintenance, quality optimisation, and production intelligence that transforms the factory floor. The difference between those two outcomes is not the hardware — it is the architecture that moves data from sensor to insight.
What does a smart factory data architecture actually do?
A smart factory data architecture moves information from machines that produce it to decisions that need it, without losing fidelity or time along the way. That sounds simple and is not, because the two ends of the pipe operate on incompatible assumptions.
Machines produce signals: vibration readings at kilohertz, temperature every second, cycle counts, torque curves, camera frames, PLC tag changes. These are high-volume, low-semantic, and time-critical. Decisions consume facts: this bearing will likely fail within nine days, this batch is drifting out of tolerance, this line is losing four percent of theoretical throughput to changeovers. These are low-volume, high-semantic, and time-sensitive in a completely different way.
The architecture's job is the translation between the two, and it has four responsibilities that are usually described as layers. Collection acquires signals at source without disrupting control systems. Ingestion moves them reliably and in order. Processing and storage turns signals into contextualised, queryable facts at the right latency and cost. Analytics turns facts into predictions and recommendations that someone acts on.
The part that most descriptions omit is the context model that sits across all four. A vibration reading is meaningless without knowing which asset, which product, which recipe, which shift, and which maintenance history it belongs to. Most factory data projects fail not because they lack a stream processor but because they never built the model that makes the stream interpretable.
Why do most factory data projects stall?
The pattern is consistent: a pilot proves value on one asset class, the business case is approved, and then scale-up stalls at twenty to thirty percent coverage. Three causes account for most of it.
Connectivity heterogeneity. A plant typically contains equipment from four or five decades of vendors: modern machines with OPC UA servers, older PLCs with proprietary protocols, and legacy equipment with no digital interface at all that requires added sensors. Each integration is a small project, and the long tail does not get cheaper with volume. The mistake is assuming uniformity and discovering heterogeneity after the business case is fixed.
Operational technology constraints. OT networks are engineered for determinism and safety, not for data extraction. You cannot simply install an agent on a PLC, and in many plants you cannot open outbound connections from the OT network to the cloud at all. Security architecture, network segmentation, and data diodes are not obstacles to be worked around — they are requirements that shape the design.
No owner for the outcome. The pilot was championed by an enthusiastic engineer. Scale-up needs a funded owner with authority over maintenance scheduling, production planning, or quality. Without that, the system produces alerts that nobody is accountable for acting on, and within two quarters people stop looking.
A fourth, subtler cause deserves mention: pilots are usually run on the best-instrumented asset in the plant. The economics that worked there do not transfer to equipment where a sensor retrofit costs more than the insight is worth.
What does the edge collection layer need to handle?
The edge layer acquires data at or near the machine, and its constraints are different from anything in the IT world.
Protocol translation. OPC UA, Modbus, MQTT, PROFINET, EtherNet/IP, and a long tail of vendor-specific protocols. The edge gateway's first job is speaking all of them and normalising what comes back into a consistent schema. Budget more effort here than seems reasonable.
Sampling discipline. A high-frequency vibration signal is valuable; a hundred kilohertz stream from every sensor on the plant is not, and it will saturate whatever network you have. The design decision is what to sample at full rate, what to aggregate at the edge, and what to transmit only on exception. The right answer is driven by the failure mode you are trying to detect — bearing wear needs high-frequency capture, temperature monitoring usually does not.
Buffering and store-and-forward. Plant networks drop. If the gateway cannot buffer locally and replay when connectivity returns, you will have silent gaps in exactly the periods when something went wrong. Local buffering of at least several days is a reasonable design target.
Isolation and safety. Read-only access to control systems, enforced at the network level rather than by convention. Nothing in the data architecture should be capable of writing to a PLC. This is a hard boundary, and it is also what makes the OT security review pass.
Time synchronisation. Underrated and routinely wrong. Correlating events across machines requires timestamps that agree to within milliseconds, which means PTP or a properly managed NTP hierarchy — not whatever default each device shipped with. Correlating data that is seconds apart produces confidently wrong conclusions about cause and effect.
How should streaming ingestion be designed?
Streaming ingestion moves data from plant edge to wherever it will be processed, and the governing principle is that the ingestion layer must never lose data silently.
Use a durable, partitioned, ordered log as the backbone — one logical stream per plant or per line, with partitions by asset or asset class. Durability matters because you will reprocess. Every factory analytics team eventually discovers that a transformation was wrong, or that a new model needs features that were never computed, and reprocessing from a retained log is the difference between a two-day fix and a six-month data archaeology project.
Separate the hot path from the warm path. The hot path carries the small subset of signals needed for real-time decisions — typically state changes, alarms, and derived aggregates — and feeds a low-latency store queried by dashboards and models. The warm path carries everything, in batch, to cheap object storage for history, reprocessing, and training. Mixing them is the most common cost and performance mistake in the layer.
Handle late and out-of-order events explicitly. Plant networks produce both, and a windowed aggregation that simply discards late data will silently undercount. Define a watermark policy, measure how often it is breached, and design downstream aggregates to be correctable rather than final.
Finally, make schema evolution explicit from day one. Equipment gets retrofitted, tags get renamed, firmware updates add fields. A schema registry with compatibility rules prevents the class of incident where a silent tag rename causes a model to read a different signal than it was trained on.
Where should processing and storage live — edge, plant, or cloud?
The placement question has a clean answer if you ask it in the right order: start with the decision's latency requirement, then the data's sovereignty and volume constraints, then cost.
At the edge, process what must be acted on within milliseconds to seconds: safety interlocks, closed-loop control adjustments, high-frequency anomaly detection on a single asset, and the filtering that decides what leaves the plant. Edge processing is also where you reduce volume — feature extraction on a vibration signal can cut data volume by two or three orders of magnitude with no loss of diagnostic value.
At the plant level, process what must keep running when the WAN is down, and what needs cross-asset correlation within a line or cell. A plant-level historian or edge cluster is the right home for short-term high-resolution history, typically days to weeks, queried by operators and maintainers who cannot tolerate a cloud round trip.
In the cloud or core data platform, process what benefits from fleet-wide scale: cross-plant benchmarking, model training, long-term history, and anything that integrates manufacturing data with ERP, supply chain, or quality systems. This is where the context model lives and where the expensive compute happens.
The practical failure is putting everything in the cloud because it is easier, and then discovering that a plant with unreliable connectivity has no analytics during the outage — which is precisely when analytics matter most. Design for degraded operation explicitly: the plant must remain useful when disconnected.
What does the analytics layer require?
The analytics layer is where signals become decisions, and three capabilities determine whether it delivers.
Contextualisation. Every analytic needs to join signal data to the asset hierarchy, the production context (which product, which recipe, which order), and the event history (maintenance, changeovers, quality incidents). This is a modelling exercise, not a tooling one, and it is the single highest-leverage investment in the layer. Without it, every analysis starts with manual data wrangling and every model is bespoke.
Appropriate model families. Predictive maintenance on rotating equipment usually starts with physics-informed thresholds or simple anomaly detection on derived features, not deep learning — because labelled failure data is scarce and the cost of being wrong is high. Quality prediction and yield optimisation tend to use supervised models on batch data, where labels exist. Computer-vision inspection is the mature exception, where supervised learning is clearly established. Choose the simplest approach that meets the accuracy requirement, because the constraint is usually maintenance of the model over years, not initial accuracy.
Delivery into workflow. The most accurate prediction in the world delivers nothing if it arrives in a dashboard that nobody opens. Maintenance predictions belong in the CMMS as work orders; quality alerts belong in the operator's station or the MES; throughput insights belong in the production review. Integration into the system where the decision is actually made is not a nice-to-have — it is the difference between a system that is used and one that is demonstrated.
One more requirement that separates working systems from abandoned ones: feedback capture. When a prediction is acted on, was it right? That answer is the training data for the next version, and systems that do not capture it plateau permanently.
Where do sensor-to-insight pipelines most often fail?
Between analytics and action. The most common failure point is not data collection or modelling — it is the last hundred metres. A prediction that surfaces in a standalone dashboard, with no owner, no workflow integration, and no feedback capture, generates no value regardless of its accuracy.
At the context join. Signal data that cannot be reliably joined to which product was being made produces models that work in validation and fail in production. Recipe and product context is usually held in the MES, and integrating it is often treated as a later phase that never arrives.
At scale-up economics. Per-asset integration cost that does not fall with volume caps coverage. This is a design problem: standardise on a small number of connection patterns, build reusable asset templates, and refuse bespoke integrations for asset classes below a materiality threshold.
On time synchronisation. Cross-machine correlation with unsynchronised clocks produces confidently wrong causal conclusions, and this failure is hard to detect because the data looks perfectly plausible.
On ownership at handover. The project team moves on and no operational function owns the system. Alerts go untriaged, model accuracy drifts, and within a year the platform is nostalgically described as "the pilot we did in 2024".
What should factories do first?
Start with the decision, not the data. Name the operational decision that will change — a maintenance task scheduled on condition rather than calendar, a quality hold raised earlier, a changeover sequence reordered — and work backwards to the minimum instrumentation, context, and delivery required to support it.
Pick the first use case where three conditions hold: the failure mode is well understood by maintenance or process engineers, the signal that indicates it is already available or cheap to add, and the decision it informs has a clear owner. Predictive maintenance on a critical rotating asset with existing vibration monitoring is the canonical example; vision-based inspection of a high-scrap operation is another.
Build the context model in parallel with the first use case rather than after it. The asset hierarchy, the product and recipe mapping, and the maintenance history links are reusable across every subsequent use case, and building them once is far cheaper than rebuilding them per project.
Design for the plant you have, not the plant in the vendor's reference architecture. If connectivity is unreliable, plan for disconnected operation. If the OT security review takes four months, sequence it first, not last. If the long tail of legacy equipment has no digital interface, decide up front which assets are worth retrofitting and which are not.
Finally, set the scale-up criterion before the pilot starts. "Eighty percent coverage of critical rotating assets within eighteen months at under X per asset" is a criterion. "Let's see how the pilot goes" is a way to guarantee the stall described at the beginning of this article.
What Are the Key Takeaways on Smart Factory Data Architecture?
Smart factory architecture is a translation problem: high-volume, low-semantic signals in; low-volume, high-semantic decisions out. The context model that makes translation possible is the highest-leverage investment in the stack.
- Design four layers — collection, ingestion, processing and storage, analytics — and build the context model across all of them from the first use case.
- At the edge: translate protocols, sample by failure mode, buffer for days, enforce read-only isolation, and synchronise clocks properly.
- In ingestion: durable ordered log, separate hot and warm paths, explicit watermarking, and a schema registry from day one.
- Place processing by latency requirement, then sovereignty and volume, then cost — and design for degraded operation when the WAN fails.
- Deliver predictions into the system where the decision is made, and capture whether they were right.
- Set the scale-up criterion before the pilot, and standardise connection patterns so per-asset cost falls.
