Production · Banking
Data Watchdog: data quality that learns
From hard-coded rules to ML anomaly classification — with streaming alerts and a human validation loop — for a bank’s Data Governance team.
- Rules + MLhybrid anomaly detection
- Batch + streamPrefect & Kafka
- Human-in-the-loopbusiness validation
Architecture
- 01PostgreSQLsource data
- 02Rulesknown checks
- 03ML classifierscikit-learn
- 04Kafkareal-time alerts
- 05Business reviewaccept / reject
↺ Review decisions feed back into the models. Prefect orchestrates the batch side; Atlas and Ranger govern the whole flow.
Context
Banque Centrale Populaire is the central institution of Groupe Banque Populaire, one of Morocco’s largest banking groups. Its Data Governance entity makes sure the data the bank relies on is complete, fresh and correct.
The challenge
Quality controls were hard-coded rules. Rules catch the problems someone anticipated and miss the rest; every new one is more code to maintain, and alerts without context get ignored.
What I built
- A rules layer for known invariants — completeness, validity, freshness — kept as the first line of defence.
- scikit-learn classification models that flag and categorise the complex anomalies rules miss.
- Batch and streaming orchestration with Prefect and Kafka, emitting real-time alerts.
- A human validation loop: business owners accept or reject proposed corrections, and their decisions feed continuous improvement.
- Governance and security: access control with Apache Ranger, lineage and cataloguing with Apache Atlas.
- Power BI quality reporting on completeness, freshness and anomalies.
Outcomes
- 01Detection no longer limited to anticipated, hand-written rules.
- 02Real-time alerts instead of after-the-fact reports.
- 03Business teams in the loop, so the system improves with use.
- 04Governed end to end: access control, lineage and catalogue.
Stack
- Python
- SQL
- Prefect
- Kafka
- PostgreSQL
- scikit-learn
- Apache Ranger
- Apache Atlas
- Power BI
- Docker
Facing a similar problem?
I’m happy to talk it through — whether you’re hiring or need a freelance mission.