Netflix runs two Flink autoscalers today: a cluster-level tool built in-house years ago, and Apache Flink's Autoscaler, which it now wants as the only one. The old system saved 25-45% on CPU, but scaled whole clusters off container metrics and missed operator-level state, causing telemetry gaps. The post walks through what broke and how the open-source autoscaler fixes it. A rare, honest teardown of throwing away your own too-clever tooling.
Bulk-export REST APIs that hand back rows and force pagination are giving way to a different contract. Producers now publish just metadata: Iceberg REST catalogs, Parquet files, and short-lived credentials on object storage. Consumers bring their own compute and read shared tables directly. The most valuable API in the stack never returns a single row, because the data never moves.
Lightstream is an open-source Rust data-streaming stack that multiplexes Arrow, Protobuf, and MessagePack frames over TCP, QUIC, WebSockets, HTTP, or more on a single connection. Its EC2 benchmarks show it measured faster than Apache Arrow Flight on every axis, even reconciling parallel streams into one globally ordered output. Python bindings implement Arrow's PyCapsule protocol, so DuckDB and similar tools can stream a Lightstream reader directly.