DataFusion 54.0.0 adds LATERAL joins and spill‑to‑disk, boosting SQL capabilities
The new release introduces LATERAL joins, SQL lambda functions for arrays, an Arrow‑based Avro reader, and spill‑to‑disk for memory‑heavy nested loop joins. Under the hood, join, scan and planning performance get a 5‑20% bump, and scalar subqueries run once instead of being rewritten as joins. These upgrades broaden DataFusion’s SQL expressiveness and cut query runtimes for complex analytics.