Education: Personalised Learning Pathways Through Data: Part 2

Personalised learning fails less often because the algorithms are weak and more often because the underlying model of what a student knows is missing. Adaptive systems need three things to work: a representation of the knowledge domain, an estimate of the learner's current state within it, and a rule for choosing the next step. Most programmes build the third, buy the second, and skip the first — then wonder why the recommendations feel arbitrary to teachers and pointless to students. Part 1 of this series covered the opportunity. This part covers the mechanics.

Getting it right matters because the upside is real. Bloom's famous two-sigma finding — that students tutored one-to-one performed two standard deviations better than those taught in conventional classes — has been revisited and qualified many times, but the underlying result, that one-to-one tutoring produces very large learning gains, has held up. The engineering question is how much of that gain can be recovered by a system rather than a human, and honest evidence suggests a meaningful fraction of it can.

Why Do Most Personalised Learning Programmes Underdeliver?

Four recurring causes, and they are organisational at least as often as technical.

No knowledge model. The system recommends "the next item" based on completion and score rather than on a map of concepts and prerequisites. Without the map, a student who fails a question on fractions cannot be routed to the specific prerequisite they are missing, so the system offers more fractions.

Engagement mistaken for learning. Time on task, videos watched, and modules completed are easy to measure and weakly correlated with durable learning. Systems optimised on engagement produce students who are busy rather than students who know more.

Teacher bypass. A recommendation the teacher cannot see, understand, or override will be ignored, and then quietly disabled. This is the most common reason adaptive systems end up as expensive content libraries.

Data silos. Assessment, attendance, homework, and behaviour data live in different systems with different identifiers. Without integration, the system sees a fraction of the signal and the teacher sees a tool that contradicts what they observe.

What Data Does a Learning Pathway Actually Need?

Three layers, and the third is the one that makes the first two useful.

Interaction data. Item-level responses: which question, which concepts it tests, correct or incorrect, time taken, hints used, attempts, and whether the answer was changed. This is the raw material for estimating knowledge state, and it needs to be captured at item level rather than aggregated to scores, because the aggregation destroys the diagnostic signal.

Learner context. Prior attainment, language background, attendance, and — used carefully — engagement patterns. Context explains variance that item responses alone cannot: a student who answers inconsistently across sessions may have a knowledge gap or may simply not be attempting seriously.

The knowledge model. The domain structure: concepts, their prerequisite relationships, and the mapping from assessment items to concepts. Without this layer, the first two are just numbers. With it, a wrong answer becomes a specific diagnosis and the pathway becomes a sequence rather than a queue.

Two practical rules. Capture the data at the moment of interaction, not in an overnight batch — an adaptive decision made in a lesson has to reflect what happened in that lesson. And keep the item-to-concept mapping under version control, because curriculum changes are frequent and a stale mapping silently corrupts every downstream estimate.

How Do You Build a Knowledge Model That Works?

This is the hard part, and it is curriculum work rather than data science. Four steps:

1. Define concepts at the right grain. Too coarse and the model cannot diagnose; too fine and no assessment covers each concept adequately. A workable rule of thumb for school mathematics is that a concept should be teachable in a single lesson and assessable with three to five items. Expect 80 to 200 concepts per year of a subject.

2. Map prerequisites explicitly. For each concept, state what a student must already know. This is the structure that makes a pathway a pathway: if a student fails long division, the prerequisite map says whether the gap is multiplication facts, place value, or the division procedure — and each has a different remedy.

3. Tag every assessment item to concepts. Ideally multiple concepts per item, with weights. This mapping is the hinge between data and decision, and it is where quality is won or lost. Double-tag a sample and measure inter-rater agreement; below roughly 0.7, the mapping is noise.

4. Validate against reality. Check that the model's predicted difficulty ordering matches observed item difficulty, and that students who the model says have mastered a concept actually pass harder items testing it. A knowledge model that has never been validated is a hypothesis with a user interface.

Budget honestly: for one subject and one year group, expect a curriculum specialist and a data person working for four to eight weeks. This is the single largest cost in the programme, and it is the one most often omitted.

Which Algorithms Should You Use for Pathway Recommendation?

Start simple, and add complexity only when it measurably beats the simple version.

Knowledge tracing — the workhorse. Bayesian Knowledge Tracing estimates, per concept, the probability a student has mastered it, updating on each response with learn and slip parameters. It is interpretable, cheap, and well understood. More recent variants — including deep knowledge tracing — fit the data better but are harder to explain to a teacher, which matters more than the accuracy gain in most deployments.

Mastery learning with thresholds. A student advances when mastery probability exceeds a threshold, and receives remediation on prerequisites when it falls below one. Simple, defensible, and it aligns with how teachers already think. This is the right default.

Multi-armed bandits for content selection. When several resources plausibly teach the same concept, bandit algorithms allocate students across them and learn which works best for whom. Useful once you have enough traffic for the allocation to be statistically meaningful — which for most single institutions it is not, until you pool across schools.

Spaced repetition for retention. Scheduling review based on forgetting curves is well-evidenced and cheap to implement. It is often the highest-return component of the whole system, because most learning loss is forgetting rather than never learning.

What not to do: do not start with deep learning on raw interaction logs. It requires volumes most institutions do not have, it produces recommendations nobody can explain, and it will lose the argument with a teacher on the first day.

How Do You Measure Whether Learning Actually Improved?

Measurement is where personalised learning programmes are most often caught out, because the easy metrics are the wrong ones.

• Learning gain, not activity. Pre- and post-assessment on the same construct, with the gain compared against a control group. Items completed is not a learning metric.
• Retention at a delay. Test again four to twelve weeks later. Programmes that look excellent on immediate post-tests frequently show no advantage at delay, because they optimised for performance during practice rather than for durable change.
• Transfer. Can the student apply the concept in a novel context? This is the hardest to measure and the most meaningful.
• Time to mastery. The efficiency measure: does the pathway get the student to the same standard faster? This is often the strongest business case, because time is the constraint teachers actually feel.
• Equity of outcome. Disaggregate all of the above by prior attainment, language background, and socio-economic group. A programme that raises the mean while widening the gap is not an improvement.
• Teacher time saved on diagnosis. The operational return that funds the programme: hours per week spent marking and diagnosing, before and after.

Run it as an experiment. Randomise at class or student level where you can, pre-register the metric, and publish the result including the null. Programmes that never publish a null result are not evidence-based programmes.

How Do You Avoid the Surveillance Trap?

Learner data is sensitive, the subjects are often minors, and the difference between personalisation and surveillance is governance rather than technology. Five principles:

Collect for a stated purpose, and use only for that purpose. Behavioural data gathered to support learning should not be repurposed for discipline, streaming, or marketing. Write this down and enforce it technically, not just in policy.

Prefer aggregate to individual where it suffices. A teacher needs to know that six students have not grasped a concept; they rarely need a per-student engagement score on a leaderboard.

Make the learner's view constructive. Students should see their own progress and next steps, not a comparative ranking. Comparative dashboards reliably increase anxiety and reduce effort among lower-attaining students.

Set retention limits and deletion paths. Interaction logs should have a defined retention period, and leaving the institution should trigger deletion.

Involve parents and students in the design. Consent is a floor, not a ceiling. Programmes designed with families are both more ethical and, in practice, far more durable.

What Does Teacher-in-the-Loop Actually Mean?

Not a rubber stamp on an algorithmic recommendation. Four concrete capabilities:

See the reasoning. The teacher can see which concept the student is stuck on, what evidence supports that conclusion, and what the prerequisite analysis says. Opaque recommendations are overridden, and correctly so.

Override with one action. Assigning different work, marking a concept as mastered, or scheduling a human explanation should each take seconds. If overriding is slow, teachers stop doing it and the system degrades.

Receive the signal at the right moment. A notification during the lesson is useful; a weekly report is too late. The value of adaptive data decays quickly.

Contribute to the model. Teacher overrides are the best available label on why the system was wrong. Capture them as feedback, review them monthly, and use them to correct item tagging and prerequisite links.

Design for the teacher first and the student second, because the teacher controls whether the student ever sees it.

How Do You Handle Privacy and Compliance for Minors?

Jurisdictions differ in detail and agree on principles. Wherever you operate, assume the strictest applicable regime.

• Lawful basis and consent. Educational provision may cover core use, but anything beyond it — research, product improvement, third-party sharing — needs a specific basis, and for minors usually parental involvement.
• Data minimisation. Do not collect behavioural telemetry you cannot act on. Every field you hold is a field you must protect and justify.
• Location of processing and storage. Many jurisdictions restrict where student data may be held. Check per country, and check the sub-processor chain, not just the vendor.
• No training on student data by default. Confirm contractually that learner interactions are not used to train vendor models, and that any use for improvement is aggregated, de-identified, and opt-in.
• Access control and audit. Who viewed which student's record, when, and why. Log it and review it.
• A named accountable person. Someone in the institution owns the register, the retention schedule, and the incident response.

What Does a Realistic Implementation Look Like?

Phase 1 — knowledge model for one subject and year group (4–8 weeks). Curriculum specialist plus data engineer. Concepts, prerequisites, item tagging, and a validation pass. Deliverable: a map that teachers agree with, which is the only acceptance criterion that matters.

Phase 2 — data integration (2–4 weeks). Connect the assessment, homework, and attendance systems at item level with resolved student identifiers. Deliverable: a live estimate of mastery per concept per student.

Phase 3 — pathway engine and teacher console (4–6 weeks). Mastery thresholds, prerequisite routing, spaced review, and a teacher view with one-click override. Deliverable: recommendations a teacher is willing to act on.

Phase 4 — measured pilot (one term). Randomised or quasi-experimental, with pre-registered metrics including retention at delay and equity disaggregation. Deliverable: a published result, positive or not.

Phase 5 — expand. Each additional subject reuses the pipeline and the console. The second subject costs a fraction of the first, because the hard part was learning how to build the map.

For the access layer, Beehive Strategy connects the assessment, attendance, and homework systems through MCP connectors and a semantic layer, so educators ask questions in plain language — "which students have not mastered the prerequisite for this week's topic, and what is their attendance pattern?" — and receive permission-scoped answers in the messaging tools staff already use. Deployed as a managed service in about two weeks, with row-level security so a teacher sees their classes and a head of year sees their year group.

What Are the Failure Modes?

Buying a platform before building the map. The platform cannot compensate for a missing knowledge model; it will just organise content badly.

Optimising engagement. Produces activity, not learning. Measure learning gain.

Streaming by stealth. Adaptive pathways that quietly sort students into fixed tracks entrench inequality. Keep pathways revisable and make the criteria visible.

Item tagging rot. Curriculum changes and the mapping drifts. Version it and re-validate each term.

Over-claiming in evaluation. Comparing against a historical cohort without controlling for cohort differences is the most common way programmes convince themselves.

Ignoring the teacher. The consistent finding across two decades of educational technology: the teacher determines the outcome. Design for them, or the tool is decoration.