Stop caching everything in Spark: target medium‑size CTE results
Spark re‑evaluates CTEs each time, so blindly caching hurts. Cache only medium‑sized intermediate results that are reused at fan‑out points and fit comfortably in memory; skip tiny tables and datasets too big to stay resident. Use the Spark UI or eager cache calls to verify memory impact and keep jobs fast.