Imad Bouzkraoui
FRGet in touch
All posts

4 min read

Rules aren’t enough: adding ML to data quality checks

Hard-coded quality rules catch the problems you anticipated. Here is how I layer ML anomaly classification on top — without drowning people in alerts.

During my final-year project in the Data Governance entity of Banque Centrale Populaire, I worked on Data Watchdog: a system that checks whether the data the bank relies on is complete, fresh and correct. It started where most data quality setups start — with hand-written rules — and ended as a hybrid of rules, ML classification and human review.

This post is about why that hybrid matters, and how I would build it again.

What rules are good at

Rules are cheap, explainable and deterministic. For the invariants you know about, nothing beats them:

Check Example
Completeness amount is never null
Validity amount >= 0, currency is an ISO code
Uniqueness one row per transaction_id
Referential integrity every account_id exists in accounts
Freshness the latest batch landed less than 2 hours ago

Most of them collapse into a single profiling query per batch:

SELECT
  COUNT(*)                                        AS row_count,
  AVG(CASE WHEN amount IS NULL THEN 1 ELSE 0 END) AS null_rate_amount,
  SUM(CASE WHEN amount < 0 THEN 1 ELSE 0 END)     AS negative_amounts,
  COUNT(DISTINCT account_id)                      AS distinct_accounts,
  MAX(loaded_at)                                  AS last_loaded_at
FROM staging.transactions
WHERE batch_date = CURRENT_DATE;

Keep these. They are your first line of defence, and they should block a pipeline when they fail.

Where rules stop working

Rules only find what someone thought of in advance. In practice, the incidents that hurt are rarely a single null column:

  • Combinations. Each field is valid on its own, but the combination is not — a perfectly normal amount on an account type that never sees amounts that size.
  • Drift. Nothing is “wrong” today, but the distribution has been sliding for three weeks.
  • Volume oddities. Half the usual rows arrive, all valid. A rule that says row_count > 0 is happy.
  • Maintenance. Every new rule is code. After a year you have hundreds, some contradictory, and nobody remembers why threshold 0.37 was chosen.

Layering ML on top

The idea is not to replace rules with a model. It is to add a second layer that looks at patterns instead of individual values.

Step 1 — profile every batch. The profiling query above already produces a feature vector per batch: row counts, null rates, means, percentiles, distinct counts, deltas versus the previous run. Store them; they become your training data.

Step 2 — start unsupervised. On day one you have no labels, so an unsupervised detector is a sensible start:

import pandas as pd
from sklearn.ensemble import IsolationForest

profiles = pd.read_parquet("dq_profiles/transactions.parquet")  # one row per batch
features = ["row_count", "null_rate_amount", "negative_amounts", "distinct_accounts"]

train, today = profiles.iloc[:-1], profiles.iloc[[-1]]
model = IsolationForest(n_estimators=200, contamination=0.02, random_state=42)
model.fit(train[features])

is_anomaly = model.predict(today[features])[0] == -1
score = model.decision_function(today[features])[0]  # lower = more anomalous

Step 3 — move to classification. An alert that says “something is odd” gets ignored. An alert that says “this looks like a duplicate load” gets fixed. Once people have reviewed enough alerts, you have labels — and you can train a classifier that predicts the category of anomaly, not just its existence.

The human loop is the product

This is the part I underestimated at first. In Data Watchdog, business owners accept or reject each proposed correction. That decision does two things:

  1. It keeps a human accountable for what changes in critical data.
  2. It produces labelled examples — exactly what the classifier needs to improve.

Two consequences for design:

  • Optimise for precision, not recall. Every false alarm costs trust. A system that cries wolf ten times will be muted on the eleventh.
  • Make the review fast. Show the record, the rule or model that fired, the suggested fix and one click to accept or reject. If reviewing takes five minutes, nobody does it.

Batch, streaming and governance

Not every flow needs real time. I used Prefect for batch profiling and scoring, and Kafka to push alerts on critical flows as they happen. Around it, governance tools answer the questions you get the moment an alert fires: who can see this data (Apache Ranger) and what else is affected downstream (lineage in Apache Atlas).

Finally, a quality system needs its own dashboard. A Power BI report on completeness, freshness and anomaly trends turns “data quality” from a feeling into a metric that teams can own.

A checklist if you are starting

  • Write rules for every invariant you know. Make the critical ones blocking.
  • Profile every batch and keep the history — it is free training data.
  • Add an unsupervised detector on the profiles; tune for few, high-quality alerts.
  • Put humans in the loop, make reviewing fast, and store every decision.
  • Graduate to a classifier when you have enough labels.
  • Track the quality system itself: alert volume, acceptance rate, time to resolution.

Rules tell you whether data matches what you expected. The ML layer tells you when your expectations are out of date — and the human loop keeps both honest.

Chat on WhatsApp