Earthquakes Lakehouse (Azure)
ADF → ADLS Gen2 → Databricks Delta (Bronze/Silver/Gold) with dbt marts and Azure Monitor alerts.
Role: Data Engineer•2024
18:00
ADF Schedule
Daily refresh time
B/S/G
Medallion
Bronze → Silver → Gold
dbt tests
Quality
not_null, accepted_values
Azure Monitor
Alerts
Failure notifications
Technology Stack
AzureADFADLS Gen2Databricks/DeltadbtGitHub ActionsAzure Monitor
Contents
Problem
Operationalize ingest and curation for USGS earthquake events with reproducible schedules, quality checks, and business-ready marts.
Architecture
ADF (Copy, params) → ADLS Gen2 (raw) → Databricks (Bronze → Silver → Gold) → dbt marts (eq_daily_metrics, eq_top100)- ADF: Parameterized Copy (start/end date, min magnitude) and daily schedule
- Databricks: Bronze persist, Silver normalization (event time, magnitude bucket, lat/lon/depth)
- dbt: Gold marts with tests (not_null, accepted_values) enforced in CI
- Ops: GitHub Actions CI; Azure Monitor alerts on failures; SPN + Secret Scopes
- Runbook: Structured logging with row counts for triage
- Security: Least-privilege IAM, Secret Scopes for OAuth
- Cost: Partitioning and pruning in Delta tables
- Repository README
Results & Impact
- Daily SLAs with alerting via Azure Monitor
- Contracted marts powering dashboard-friendly metrics
- CI-enforced tests gatekeeping quality