Project 02: HIPAA-Compliant Lakehouse on Azure Databricks // Corrupted Glitch

Full build log: a Terraform-provisioned Medallion lakehouse with Zero Trust boundaries — RBAC, VNET injection, and Private Link protecting PHI at scale.

The mission: give analysts a modern lakehouse over patient data without ever exposing PHI to a public endpoint. Everything provisioned by Terraform; nothing clickable in the portal.

[ BUILD PARAMS ]

PLATFORM: Azure Databricks (VNET-injected)
PATTERN: Medallion (bronze / silver / gold)
COMPLIANCE: HIPAA · Zero Trust boundaries
NETWORK: Private Link only — no public ingress
IAC: Terraform, 100% of resources

#Architecture

 ┌────────────────────── PRIVATE VNET ──────────────────────┐
 │                                                          │
 │  ┌─ BRONZE ─┐     ┌─ SILVER ─┐     ┌── GOLD ──┐          │
 │  │ raw PHI  │ ──▶ │ cleansed │ ──▶ │ de-ident │──▶ BI    │
 │  │ ADLS gen2│     │ conformed│     │ aggregate│   (RBAC) │
 │  └──────────┘     └──────────┘     └──────────┘          │
 │        ▲                                                 │
 │  Private Link ◀── ingestion (ADF, self-hosted IR)        │
 └──────────────────────────────────────────────────────────┘
        NO PUBLIC ENDPOINTS · CMK ENCRYPTION · AUDIT LOGS ON

#Step 1 — Network Skeleton First

VNET injection cannot be retrofitted. Provision the network before the workspace:

terraform/network.tf HCL

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
resource "azurerm_virtual_network" "lakehouse" {
  name                = "vnet-lakehouse-prod"
  address_space       = ["10.40.0.0/16"]
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

resource "azurerm_subnet" "dbx_private" {
  name                 = "snet-dbx-private"
  # Databricks requires delegated public+private subnets;
  # "public" here still has NO internet ingress — NSG-locked.
  address_prefixes     = ["10.40.2.0/24"]
  virtual_network_name = azurerm_virtual_network.lakehouse.name
  resource_group_name  = azurerm_resource_group.main.name
  delegation {
    name = "databricks"
    service_delegation { name = "Microsoft.Databricks/workspaces" }
  }
}

#Step 2 — Workspace with No Public IP

terraform/workspace.tf HCL

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
resource "azurerm_databricks_workspace" "phi" {
  name                = "dbx-phi-prod"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  sku                 = "premium" # required for RBAC + audit
  public_network_access_enabled         = false
  network_security_group_rules_required = "NoAzureDatabricksRules"
  custom_parameters {
    no_public_ip        = true
    virtual_network_id  = azurerm_virtual_network.lakehouse.id
    private_subnet_name = azurerm_subnet.dbx_private.name
    public_subnet_name  = azurerm_subnet.dbx_public.name
  }
}

#Step 3 — Medallion Zones with RBAC That Means It

Bronze (raw PHI): service principals only. No human reads bronze.
Silver (cleansed): data engineers, break-glass audited.
Gold (de-identified aggregates): analysts and BI.

jobs/deidentify_gold.py PYTHON

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from pyspark.sql import functions as F

def to_gold(silver_df):
    """Gold = de-identified. Safe Harbor's 18 identifiers gone."""
    return (silver_df
        .drop("patient_name", "ssn", "mrn", "street_address", "phone")
        .withColumn("zip3", F.substring("zip", 1, 3))   # geo, coarsened
        .withColumn("age_band",
            F.when(F.col("age") >= 90, "90+")           # HIPAA: 90+ collapses
             .otherwise(F.floor(F.col("age") / 10) * 10))
        .drop("zip", "age", "dob"))

#Step 4 — Prove It

Compliance is a plan-time failure, not a review meeting. terraform plan runs in CI with policy checks: no storage account without CMK, no subnet without an NSG, no workspace with public access. A violation fails the PR.

Control	Mechanism	Verified by
PHI never public	Private Link + `no_public_ip`	CI policy check
Least privilege	Zone-scoped RBAC	Quarterly access review
Encryption	CMK on ADLS + managed disks	`terraform plan` policy
Auditability	Workspace audit logs → SIEM	Log pipeline heartbeat