LodeHQSubscribe →

Zero-copy isn't free: Inside Hotstar's real-time ad engine

Data · 2026-07-27

Data Engineering
Inside Hotstar’s Real‑Time Ad Engine: How One Request Scales7 MIN

Hotstar’s engineering team details the end‑to‑end pipeline that serves a single ad request, from real‑time bidding and audience targeting to the high‑throughput infrastructure that processes millions of requests per second. The post reveals the micro‑service architecture, latency budgets, and scaling tricks that keep ad delivery fast and reliable at Disney+ scale.

Zero‑Copy Isn’t Free: Six Data‑Movement Patterns and Their Hidden Costs30 MIN

Zero‑copy is a catch‑all for six distinct architectures, from federated queries to sharing protocols. The post breaks down each pattern, exposing where data still moves, the governance trade‑offs, latency and compute costs, and how to pick the right one for your workload.

Zalando trades 7‑year homegrown ad join for Flink, cuts cost in half12 MIN

Zalando’s Ad Platform replaced a bespoke, in‑memory stream join that processed up to 200 MB/s with Apache Flink. The migration eliminated in‑memory state loss on restarts, reduced infrastructure spend by roughly 50%, and kept the 15‑minute event‑matching window intact. Unmatched events are now persisted until their counterpart arrives.

Analytics & Visualization
All 116 US Supreme Court Justices Mapped by Appointment and Tenure1 MIN

FlowingData visualizes every U.S. Supreme Court justice, showing when they served and which president appointed them. The timeline reveals appointment patterns across history, from early republic to modern era, letting analysts spot political swings and tenure trends.

ML & AI for Data
Run BM25, Dense Retrieval, and SPLADE on a 16 GB MacBook: a RAG baseline checklist13 MIN

The author reproduces three core retrieval baselines, BM25, dense retrieval, and SPLADE, on a consumer‑grade MacBook with 16 GB RAM, documenting out‑of‑memory crashes, native library failures, and API key hurdles that the official docs omit. Knowing how to run these baselines locally gives RAG developers a concrete quality reference before layering LLMs.

Airbnb uses a Transformer to turn years of guest actions into smarter search results9 MIN

Airbnb replaced hand‑crafted ranking features with a sequence model that ingests every view, booking and cancellation a guest has ever made. By encoding this massive event stream in a Transformer, the platform surfaces listings that match a traveller’s evolving preferences, boosting relevance and conversion. The approach scales to millions of users and unlocks richer personalization across the search funnel.

KimiClaw’s Managed AI Agents Beat Self‑Hosted OpenClaw for Ease and Scale7 MIN

KimiClaw packages OpenClaw’s autonomous‑agent capabilities in a cloud service, eliminating the hardware, setup, and maintenance overhead of self‑hosting. For data teams that need always‑on AI workflows but lack ops bandwidth, the trade‑off of reduced local control delivers faster deployment and comparable automation power.

Context engineering: the silent bottleneck slowing AI adoption21 MIN

Stack Overflow’s data‑science lead explains that LLMs can draft replies or code, but the missing piece is the surrounding context, email threads, Slack chats, internal tools, and permissions. Without automated context engineering, humans must cobble together and validate inputs, turning a quick AI assist into a multi‑step chore.

Give OpenAI LLMs Real Web Interaction via Playwright6 MIN

The walkthrough shows how to stitch OpenAI Agents SDK with Playwright MCP to create a loop where an LLM observes page structure and issues element‑targeted actions. A live example automates a support‑ticket console, proving that LLMs can navigate real web UIs without custom code.

Databases & Storage
RocksDB’s Snapshot Isolation: Inside Its MVCC Transaction Engine18 MIN

RocksDB tags every write with a monotonic sequence number and builds immutable SST files, then serves readers a consistent view via snapshot isolation. This design lets reads and writes run in parallel without locks, and forms the basis for its atomic batch updates and transactional API.

Postgres LISTEN/NOTIFY scales to 60K writes/sec with batching5 MIN

DBOS re‑engineered Postgres LISTEN/NOTIFY by buffering NOTIFY calls and adding a safety‑net poll, cracking 60 000 writes per second on a single server with millisecond latency. The trick sidesteps the global exclusive lock that traditionally throttles NOTIFY, proving the mechanism can serve high‑throughput, low‑latency streaming or pub/sub workloads without sacrificing durability.

CocoSQL: Write DuckDB SQL once, run on Postgres, ClickHouse, etc.10 MIN

CocoSQL uses DuckDB’s parser, binder, and AST to translate a single query into Postgres, ClickHouse, and other dialects, keeping literal values, ordering, and types intact. This lets their platform blend visual data transforms with custom SQL on any backend without users learning new syntax, cutting errors from dialect quirks.

Practice & Datasets
New OWID hub maps economic inequality data for practitioners36 MIN

Our World in Data launched a dedicated topic page that consolidates all its economic inequality charts, datasets, and analysis in one place. It organizes the material around the World Bank Poverty & Inequality Platform, the Luxembourg Income Study, and the World Inequality Database, letting analysts quickly compare measures and see where policy can shift outcomes.

Turn a One‑off AI Script into a Live Research Agent in One Sitting17 MIN

This guide walks you through every piece of glue code needed to turn a prototype LangGraph agent into a production‑ready web‑searcher that drafts a concise brief with source links. It closes the gap where most AI agents die after the demo, showing how to scope, containerize, and deploy safely.

SSOT Projects Fail When Teams Disagree on Metrics, Not When Tech Breaks5 MIN

Housing.com’s engineering team found that SSOT initiatives collapse under mismatched metric definitions and lack of governance, not under technical complexity. Aligning business owners, standardizing definitions, and establishing clear data governance turned their broken SSOT into a trusted foundation for decision‑making.

Get Data in your inbox, every issue.
Subscribe free
Get the app · Privacy · Terms · About · Contact
© 2026 LodeHQ