Databricks · Delta Lake · TDA
Computational Neuroscience Lakehouse (NeuroLab)
A Databricks lakehouse and topological-data-analysis pipeline for sleep EEG.
NeuroLab is our research data-engineering project: raw sleep EEG is organised into Bronze / Silver / Gold layers for a traceable, reproducible pipeline.
The data are public PhysioNet Sleep-EDF recordings. Raw EDF files arrive incrementally in Bronze via Auto Loader; Silver applies band-pass filtering, artefact detection and 30-second epoching and detects sleep spindles and slow oscillations; Gold aggregates time-domain, frequency-domain and topological features.
Topological features use methods such as persistent homology and complement classical time–frequency features.
To validate methods, we reproduced published sleep-oscillation research in Python: on a public intracranial EEG dataset we detect slow oscillations, spindles and ripples and analyse how they nest, using phase–amplitude coupling and cluster-based permutation tests.
The same methods power enterprise data platforms: layered modelling, data-quality checks, metadata management and reproducible computation.
Architecture
System architecture
01 · Bronze
- Auto Loader incremental EDF ingest
- Schema evolution
- Raw records retained
02 · Silver
- 0.5–30 Hz band-pass filter
- Artefact detection, 30 s epochs
- Spindle and slow-oscillation detection
- Delta Live Tables expectations
03 · Gold
- Time, frequency and topological features
- Subject-level aggregates
- OPTIMIZE and Z-ORDER
04 · Apps & governance
- MLflow sleep-staging models
- Persistent-homology research
- Unity Catalog RBAC, column masking, lineage
Highlights
Engineering highlights
Declarative data quality
Delta Live Tables expectations validate timestamps and signal quality in Bronze and Silver; bad records are stopped instead of silently flowing downstream.
Governance and access
Unity Catalog grants access by role (data engineering, research, ML), masks sensitive columns and keeps audit logs and lineage.
Environments and releases
GitHub Actions and Databricks Asset Bundles automate tests and deployment across dev, staging and production.
Reproducible science
The reproduction is driven by a single YAML config; every detector documents its method and citation, with CI across several Python versions.
Book a free 30-min call
Free call