Unloading Snowflake ACCOUNT_USAGE query_history to Parquet writes bytes_scanned as INT32, silently overflowing any value past 2^31 and corrupting your cost and scan analysis. A bare NUMBER(38,0) column keeps its narrow on-disk physical type; wrapping it in any expression or an identity CAST fixes the export. Simple cast before you unload or your aggregates lie.
A new paper pushes back on CDC hype: it durably records and replays the fan-out, fixing silent divergence, but destinations still need idempotency or duplicate rejection to be safe on redelivery. A retried index upsert is harmless; a retried email is a duplicate. The incidents just move downstream.
A benchmark pits DuckDB on ECS Fargate against Spark on EMR Serverless over 1,000 JSON files from S3, with some files spiking past 20MB to force real-world skew. DuckDB finished about 3.4x faster and at lower cost. A useful datapoint for teams weighing lightweight engines over managed Spark clusters.
Flexmon open-sourced Globe Trotter, a GPU-accelerated 4D globe engine for massive spatio-temporal data, under Apache 2.0. Its zero-copy Flex binary formats let a single browser tab render 1.47M hexagons across 1,440 epochs (2.1B points) at 60 FPS from static hosting, no tile or server infrastructure needed.
Hex built DataBench from messy, real-world analytics prompts after arguing existing benchmarks like Spider 2.0 test overspecified "pub trivia." Frontier agents gather evidence well but stumble on open-ended recommendations, often manufacturing confident answers when an honest "can't know" is right. That failure mode matters for anyone putting agents near production data.
Tailscale's uptime was wrecked for months by a rarely hit race between SQLite WAL checkpointing and write transactions, uncovered only after 19 separate corruption incidents. A masterclass in database forensics: the fix was a tiny logic check, but finding it took real archaeology. Their manual checkpointing made a 16-year-old bug far more likely to fire.
Partition pruning and min/max stats stop helping when you filter on high-cardinality keys, forcing full scans. Hudi's metadata-table indexes (record-level, secondary, expression, bloom filters) prune to the few matching files at planning time, taking a record-key lookup on a 400 GB table from 977 seconds down to 12. A clear primer on making selective reads act like a database.
DuckDB's v2.0 "Cyanoptera" lands this fall with a native server mode: any DuckDB process can serve databases over the network, with CONNECT pushdown straight to PostgreSQL and MySQL. It also adds triggers, a VARIANT type, asynchronous I/O, a new SQL parser, and a new default storage format.
Dataindex.us now catalogs federal data collections terminated, reduced, or altered under executive orders, from NCUA contractor diversity forms dropped entirely to CDC health surveys stripped of gender identity items. Each entry links the underlying agency record, making the scale and pattern of data loss visible at a glance.
You’re caught up.
That was all 9 stories in the 17 August issue.