From Lab Bench to Lakehouse:
Genomics Pipelines at Petabyte Scale with Delta Lake

01 · Why Genomics Defeats the Standard Warehouse

Three properties make genomics a worst case for conventional platforms. Shape: the atoms are huge semi-structured files (FASTQ, BAM/CRAM, VCF/BCF) with deeply nested per-sample fields — flattening them into warehouse rows either explodes cardinality or amputates the science. Reproducibility: a published cohort analysis must be re-derivable against the exact data, reference build, and code that produced it — sometimes seven years later. Audit: clinical-trial submissions answer to regulators who ask precisely the questions warehouses can't: show me what you knew, when.

The lakehouse resolves the tension because it refuses the false choice: raw artifacts stay in object storage as the immutable scientific record, structured variant tables live beside them with database semantics, and one transaction log binds both into a provenance chain.

02 · The Lakehouse Architecture for Variant Data

Partitioning is the load-bearing decision: partition by chromosome (and date for ingestion tables), Z-order on (sample_id, position) so both access patterns — "everything about this sample" and "everything in this region across the cohort" — prune effectively. Whole-genome scans become targeted reads; the difference at biobank scale is hours and four figures per query.

03 · Ingestion: VCF/BCF as a Streaming Problem

Sequencers don't batch politely — they emit runs continuously, and a sequencing center's output is a stream wearing a filesystem costume. Treat it that way:

Auto Loader — continuous VCF discovery into bronze (PySpark)
(spark.readStream
   .format("cloudFiles")
   .option("cloudFiles.format", "text")
   .option("cloudFiles.schemaLocation", chk("vcf_bronze"))
   .load("s3://seq-center/landing/vcf/")
   .transform(parse_vcf_lines)            # header-aware; per-sample fields → MapType
   .withColumn("source_file", input_file_name())
   .withColumn("ref_build",  extract_reference(col("source_file")))   # pinned, always
   .writeStream
   .format("delta")
   .option("checkpointLocation", chk("vcf_bronze"))
   .trigger(availableNow=True)            # cost-controlled micro-batch
   .toTable("genomics.bronze.vcf_records"))

A sustained 50GB/hour per ingestion lane is a comfortable design target on a modest Spark cluster, and lanes scale horizontally — sequencing throughput becomes a provisioning decision, not an architecture one. The rules that keep bronze honest: preserve the raw line per parsed row (re-parse history when a parser bug surfaces, instead of re-requesting files from the lab); pin the reference build as a column, never an assumption; and checksum-verify against the sequencer manifest before a file is considered landed.

04 · Variant Processing as Governed Spark Pipelines

Joint genotyping, annotation, and cohort filtering become Spark jobs over Delta tables instead of shell-script chains over files. The wins are mundane and decisive:

Maintenance is scheduled, not heroic: OPTIMIZE with Z-ordering after large ingests, file compaction for streaming landings, and VACUUM windows aligned to the retention policy (not the default — Section 07 explains why).

05 · ACID + Time Travel = Reproducible Research

The killer feature is not speed; it is the audit answer. Every table version is retained and queryable:

the inspection query — re-derive a 2024 analysis exactly
-- What did the cohort look like when the paper was submitted?
SELECT * FROM genomics.gold.cohort_cardio
  TIMESTAMP AS OF '2024-03-15 00:00:00';

-- Bind the full provenance chain for the audit response:
DESCRIBE HISTORY genomics.gold.cohort_cardio;   -- who, what job, which commit
-- + pipeline git SHA pinned in table properties
-- + reference genome checksum pinned per row
-- = the rerun IS the audit response

A pipeline run is a transaction: it commits whole or not at all, so a failed annotation job can never leave a half-updated cohort for a downstream model to silently train on. Pair table history with pinned pipeline-code versions and reference checksums and reproducibility stops being a policy and becomes a property — the difference between "we believe the analysis was correct" and "here is the rerun" when a trial faces inspection.

06 · Where GPUs Earn Their Cost — and Where They Don't

GPU acceleration is real and it is not universal. The honest engineering move is per-stage benchmarking: run each pipeline stage on CPU and GPU fleets, divide speedup by cost ratio, pin each stage to its winner.

Mixed fleets orchestrated per-stage are normal and unexciting — which is what you want. The waste pattern we see in the field is symmetrical: estates paying for GPUs on I/O-bound stages, and estates grinding CPU weeks on alignment because "GPUs are expensive." Both lose to a benchmark spreadsheet that takes two days to build.

07 · Governance for Human Genomes

A genome is the most identifying datum that exists, consent is per-sample and revocable, and jurisdictions disagree about where genomes may rest. The platform encodes all three:

consent enforcement — revocation as a platform property
-- Gold cohorts are consent-filtered views, never copies:
CREATE OR REPLACE VIEW genomics.gold.cohort_cardio AS
SELECT v.* FROM genomics.silver.variants v
JOIN governance.consent_registry c USING (sample_id)
WHERE c.study_scope = 'cardio_2026'
  AND c.status = 'active';            -- revocation is immediate, by construction

-- Erasure (GDPR-class) is provable, not aspirational:
DELETE FROM genomics.silver.variants WHERE sample_id = :revoked;
-- deletion vectors mark immediately; VACUUM physically removes
-- past retention window; both events land in the audit log

Plus: study-scoped access grants (a researcher holds grants to studies, not tables), region-pinned storage with governed sharing instead of replication for cross-border collaborations, and the audit log itself as a queryable product. A 12PB estate is a realistic scenario target for a national-scale biobank on this design — and the controls are identical at 50TB, which is the right time to install them.

08 · Lessons Learned: The Hard Truths

• Normalization is the cohort. Decomposition and left-alignment differences between labs are the silent killer of cross-site comparability. Normalize deterministically in silver, record the tool version per row, and re-normalize history when the tool upgrades.
• Preserve the raw line. The month-nine parser bug is a certainty, not a risk. Bronze rows that carry their source line turn it into a re-parse backfill instead of a lab-relations incident.
• Small files will eat the metadata layer. Streaming VCF landings produce file counts that degrade planning long before storage hurts. Compaction is a scheduled job from day one, not a remediation.
• VACUUM and audit retention fight — decide deliberately. Time travel depends on retained versions; storage budgets want them gone. Set retention per zone from the compliance calendar (gold: years; bronze: months) and document the trade once, not per incident.
• Consent as a join, not a pipeline step. Filtering consent during ETL bakes yesterday's consent into today's tables. Consent-filtered views make revocation instantaneous and auditable by construction.
• Benchmark GPUs per stage, then stop arguing. The two-day benchmark spreadsheet ended a six-month internal debate. Alignment and DL calling won decisively; everything else stayed CPU. Numbers beat positions.

09 · Key Takeaways for Practitioners

The lakehouse engineering pattern here is the one we run in production — documented in the geospatial AI lakehouse case study (a different high-cardinality science domain, same architecture spine). For format selection beneath the pipelines, see Delta vs Iceberg vs Hudi.

FAQ · Frequently Asked Questions