Technical deep dives, architecture patterns, and best practices in data engineering.
Also published on Medium · Newsletter
We paused a streaming job in staging and forgot about it. Sixty days later it couldn't start, because the source table had cleaned up the log versions its checkpoint still pointed at. Same job, same error, twice.
A wall of ConnectionRefusedError in CI, one real OOM, and two lines that fixed it. Here's the debugging chain that led there.
Three streaming jobs died within 32 seconds of each other. The data plane was healthy the whole time. Here's what actually happened and what we changed.
I asked an LLM agent to get a Databricks job ID at runtime. It confidently proposed four approaches. All four were wrong. The fix was a 30-line Python script I could have written in ten minutes.
We know what the real answer is. We tested it. The code is ready. We're just waiting for the right moment, and that's a completely legitimate engineering decision.
Splitting into multiple tasks feels like the obvious fix after a multi-query partial failure. It isn't — not on a shared cluster. There's still one driver.
Most Databricks streaming failures don't look dramatic. No cluster termination, no red wall of errors. Just a job that says RUNNING while your customers report nonsense.
Learn how to inspect the Delta transaction log to understand your partition size distribution and make informed partitioning decisions.
Deep dive into Z-ordering, data skipping, and compaction strategies to maximize Delta Lake performance.
Proven strategies for optimizing Databricks cluster configurations and reducing cloud infrastructure costs.
Exploring medallion architecture, data mesh, and other patterns for building scalable lakehouse platforms.