DataFusion now detects and exploits exact or partial sortedness in Parquet files automatically. For ORDER BY and ORDER BY ... LIMIT N queries it reorders scans using min/max statistics, applies dynamic filters and skips the blocking sort, cutting full‑scan costs and peak memory on time‑series or log data.
DuckDB's Iceberg extension adds CTAS, MERGE, INSERT, UPDATE and DELETE, making CRUD‑style Iceberg pipelines possible without Spark. Pair it with MinIO for S3 storage and LakeKeeper's Rust catalog and you get a fully local, lightweight lakehouse stack for data engineering workloads.
Backfilling isn’t a post‑mortem script, it should run through the same extract‑raw‑transform pipeline as daily loads, toggled by config flags. By treating migration and historical ingestion as production jobs (Opto uses Dagster), teams eliminate brittle one‑offs, improve correctness, and cut the endless firefighting that follows ad‑hoc backfills.
Dropbox’s Riviera platform, which transforms hundreds of thousands of files per second for products like PowerPoint previews and video transcoding, is now exposed as an API and Model Context Protocol for external developers. By sharing its plugin‑based architecture, engineers can reuse Dropbox’s high‑scale content extraction and conversion pipeline for AI, workflow automation, and search use cases.
When Planck scaled its LLM‑agent fleet to hundreds, latency jumped not because of the LLM APIs but due to a flood of tiny async CPU tasks. The authors pinpointed the hidden CPU contention and rewrote the dispatcher to eliminate the bottleneck, restoring fast response times.
Public benchmarks report about 80% accuracy for Text‑to‑SQL models, but Michael Stonebraker’s test on a 1,400‑table Oracle warehouse yielded only ~10% accuracy, rising to ~30% with explicit table hints. The gap stems from unseen proprietary data, far more complex queries, idiosyncratic naming, and tangled schemas. Teams risk over‑estimating AI tools if they rely on clean‑benchmark scores.
A new AI Impact Analyst watches Iceberg tables via Lakekeeper, captures schema edits in Kafka, and uses an Anthropic model to assess downstream impact within seconds. The service posts real-time alerts and remediation tips to Slack, cutting the costly downtime that schema drift normally triggers.
Mid‑2026 production pipelines now deploy specialist agent teams, triage, SQL, Python, tied together by the Model Context Protocol (MCP) and linked to GitHub, Slack, PostgreSQL. Persistent memory graphs harvest execution histories, letting agents build lasting knowledge and eliminating brittle orchestrated reasoning loops. The result is faster, more reliable AI‑driven workflows.
Aurora DSQL is a serverless, PostgreSQL‑compatible OLTP database that decouples compute, storage, and transaction coordination into scalable services. It delivers active‑active multi‑region deployments with strong consistency, requiring cross‑region coordination only at commit, enabling sub‑second latency and scaling to millions of transactions per second without manual replication or failover.
A single dubious column type or primary‑key change can lock a team into a multi‑quarter migration and a permanent performance tax. The blog maps schema changes onto a ladder of irreversibility, showing which tweaks are cheap and which are costly. It argues that proactive guardrails, not after‑the‑fact fixes, are the only sane strategy.
DuckDB processes data in cache‑friendly vectors of 2,048 rows, letting it apply pre‑compiled operators across the batch with a single selection vector. This push‑based pipeline slashes copying and improves CPU IPC, turning per‑row Volcano execution into a high‑throughput, columnar engine that runs queries dramatically faster.
Joe Reis demonstrates a prototype where every AI agent runs its own embedded DuckDB engine and trades immutable SliceRef objects, snapshot IDs plus content digests, instead of mutable snapshots. This sidesteps cache‑coherence bottlenecks and lets agents share data products directly, governed by Malloy‑defined semantic contracts.
You’re caught up.
That was all 12 stories in the 23 July issue.