Production Engineering Series
A Practitioner's Guide
Spark, Delta Lake,
and Databricks
in Production
Real Problems, Real Fixes, No Theory
A practitioner's guide to the failure modes, configuration traps, and architectural mistakes that no getting-started guide covers. Every chapter opens with a real production incident and ends with a diagnostic checklist you can use tomorrow.
Book Progress 2/24
Pt 1
Pt 2
Pt 3
Pt 4
Pt 5
Recently Published
Front Matter
Part 1 Spark Runtime Failures
01 OOM Decoded: Driver vs Executor
Coming Soon
02 The Broadcast Join Trap
Coming Soon
03 Shuffle, Spill, and Skew After AQE
Coming Soon
04 Dynamic Allocation on K8s and Serverless
Coming Soon
05 UDF Performance Traps and Serialization
Coming Soon
Part 2 Delta Lake in Production
06 The Small Files Tax and How to Stop Paying It
Coming Soon
07 Liquid Clustering: The Gotchas Nobody Mentions
Coming Soon
08 VACUUM: The Operation That Breaks Everything Downstream
Coming Soon
09 MERGE at Scale: Concurrency, Performance, and Correctness
Coming Soon
10 Schema Evolution and Type Widening
Coming Soon
11 CDF, Time Travel, and Deletion Vectors: Hidden Costs
Coming Soon
Part 3 Structured Streaming
12 Checkpoints: The Fragile Foundation
Coming Soon
13 Watermarks and Late Data: Getting the Tradeoff Right
Coming Soon
14 State Store Bloat and Memory Leaks
Coming Soon
15 Exactly-Once Semantics: What It Actually Guarantees
Coming Soon
16 Streaming on Serverless and Shared Compute
Coming Soon
Part 4 Databricks Platform Pitfalls
17 Cluster Economics: Job, All-Purpose, Serverless, Photon
Coming Soon
18 DBR Version Upgrades: What Breaks and Why
Coming Soon
19 Unity Catalog Migration: A Survival Guide
Coming Soon
20 From Notebooks to Production Code
Coming Soon
Part 5 Architecture Mistakes
21 Medallion Architecture: When Bronze-Silver-Gold Goes Wrong
Coming Soon
22 The Over-Partitioning Epidemic
Coming Soon
23 Catalog and Namespace Design You Cannot Undo
Coming Soon
24 Cost Optimization: The Decisions That Actually Move the Bill
Coming Soon
Stay Updated
New chapters delivered to your inbox
Subscribe to get full chapters as they publish, plus production war stories that don't make it into the book.
Subscribe on Substack →