Startups bet on DataFusion, AI tracing, agent runtime

Data · 2026-07-25

Data Engineering

Startups Bet on Apache DataFusion to Power Custom Analytic Engines2 MIN

Apache DataFusion, a Rust‑written, Arrow‑backed query engine, lets startups spin up high‑performance analytics platforms without reinventing the wheel. Its composable design and growing ecosystem, Python, Java bindings and distributed runtimes, enable rapid launch of custom data solutions, a key advantage as they chase speed and flexibility.

ML & AI for Data

BERTopic + LLMs: Faster, More Accurate Topic Modeling for Unstructured Text11 MIN

BERTopic stitches together transformer embeddings, UMAP, HDBSCAN and optional LLM‑based representation models into a modular pipeline, delivering topic clusters that beat LDA in speed and coherence. The article walks through a concrete Python implementation, showing how swapping components, like a smaller SentenceTransformer, lets you tune performance for large corpora.

AI Tracing Turns Silent LLM Failures Into Debuggable Stories11 MIN

AI tracing captures every step of an LLM or agent request, prompts, data retrieval, tool calls, and model outputs, so teams can spot hallucinations and contradictory answers that normal latency and HTTP metrics miss. Monte Carlo’s 2026 survey found only 47% of builders can trace failures end‑to‑end, underscoring the need for a standardized trace layer.

Enterprise Agent Runtime Stack Emerges to Govern Autonomous LLM Workflows3 MIN

As LLM inference becomes cheap, enterprises now need infrastructure to run autonomous agents at scale. The blog outlines a five‑layer stack, agent builders, persistence engines, memory, web/tool interaction, and control planes, designed for durable execution, governance, and long‑term state. Without this stack, agent‑driven automation remains brittle and unmanageable.

Klaviyo Scales 100GB+ ML Pipelines with Ray Data, Train & Tune5 MIN

Klaviyo uses Ray Data to ingest and preprocess 100 GB+ tables, over 100 M rows, on a Ray cluster, achieving near‑linear scaling and minutes‑long jobs. Ray Train and Ray Tune then handle distributed model training and hyperparameter optimization, letting the team ship ML models faster without Spark.

Databases & Storage

When Feature Creep Turns a Report Generator into a Public SQL Console6 MIN

A decade of incremental feature requests transformed a device‑log report page from a simple date‑range UI into a free‑form SQL textbox, effectively exposing the entire database. The story shows how tiny UI tweaks can accumulate into an unintended “SQL Injection as a Service” risk, warning data engineers to guard against creeping query interfaces.