Preface Introduction 01. OOM Decoded: Driver vs Executor 02. The Broadcast Join Trap 03. Shuffle, Spill, and Skew After AQE 04. Dynamic Allocation on K8s and Serverless 05. UDF Performance Traps and Serialization 06. The Small Files Tax and How to Stop Paying It 07. Liquid Clustering: The Gotchas Nobody Mentions 08. VACUUM: The Operation That Breaks Everything Downstream 09. MERGE at Scale: Concurrency, Performance, and Correctness 10. Schema Evolution and Type Widening 11. CDF, Time Travel, and Deletion Vectors: Hidden Costs 12. Checkpoints: The Fragile Foundation 13. Watermarks and Late Data: Getting the Tradeoff Right 14. State Store Bloat and Memory Leaks 15. Exactly-Once Semantics: What It Actually Guarantees 16. Streaming on Serverless and Shared Compute 17. Cluster Economics: Job, All-Purpose, Serverless, Photon 18. DBR Version Upgrades: What Breaks and Why 19. Unity Catalog Migration: A Survival Guide 20. From Notebooks to Production Code 21. Medallion Architecture: When Bronze-Silver-Gold Goes Wrong 22. The Over-Partitioning Epidemic 23. Catalog and Namespace Design You Cannot Undo 24. Cost Optimization: The Decisions That Actually Move the Bill

Chapter 22 of 24

The Over-Partitioning Epidemic

14 min read · Updated March 2026 · Dataset: NYC Taxi (samples.nyctaxi.trips)

Expands on: delta-table-partition-size-distribution

Coming Soon

This chapter is currently being written.

Subscribe to the newsletter to get notified when new chapters drop.