Preface Introduction 01. OOM Decoded: Driver vs Executor 02. The Broadcast Join Trap 03. Shuffle, Spill, and Skew After AQE 04. Dynamic Allocation on K8s and Serverless 05. UDF Performance Traps and Serialization 06. The Small Files Tax and How to Stop Paying It 07. Liquid Clustering: The Gotchas Nobody Mentions 08. VACUUM: The Operation That Breaks Everything Downstream 09. MERGE at Scale: Concurrency, Performance, and Correctness 10. Schema Evolution and Type Widening 11. CDF, Time Travel, and Deletion Vectors: Hidden Costs 12. Checkpoints: The Fragile Foundation 13. Watermarks and Late Data: Getting the Tradeoff Right 14. State Store Bloat and Memory Leaks 15. Exactly-Once Semantics: What It Actually Guarantees 16. Streaming on Serverless and Shared Compute 17. Cluster Economics: Job, All-Purpose, Serverless, Photon 18. DBR Version Upgrades: What Breaks and Why 19. Unity Catalog Migration: A Survival Guide 20. From Notebooks to Production Code 21. Medallion Architecture: When Bronze-Silver-Gold Goes Wrong 22. The Over-Partitioning Epidemic 23. Catalog and Namespace Design You Cannot Undo 24. Cost Optimization: The Decisions That Actually Move the Bill

Chapter 3 of 24

Shuffle, Spill, and Skew After AQE

16 min read · Updated March 2026 · Dataset: NYC Taxi (samples.nyctaxi.trips)

Coming Soon

This chapter is currently being written.

Subscribe to the newsletter to get notified when new chapters drop.