Apache Celeborn adds a remote shuffle layer to EMR, letting Spark jobs run entirely on Spot instances without risking data loss. By offloading shuffle data to a dedicated storage tier, it cuts recomputation after spot interruptions and avoids over‑provisioning, boosting reliability and cutting cloud spend.
Apache Celeborn adds a remote shuffle layer to EMR, letting Spark jobs run entirely on Spot instances without risking data loss. By offloading shuffle data to a dedicated storage tier, it cuts recomputation after spot interruptions and avoids over‑provisioning, boosting reliability and cutting cloud spend.
Prefect is buying Dagster Labs and keeping both brands, pricing and roadmaps intact. About 40 Dagster engineers join Prefect, merging Dagster's asset‑first model with Prefect's durable execution engine to create a single, AI‑ready orchestration platform.
AWS and Salesforce now let Data 360 query Apache Iceberg tables in S3 without copying files, using the Iceberg REST endpoint in the Glue Data Catalog. This "zero‑copy" federation cuts storage costs, speeds real‑time analytics, and keeps Lake Formation governance intact, letting AI agents and users work on fresh, governed data instantly.
On a 1TB TPC‑H benchmark, distributed Polars across 32 m8i.xlarge nodes edged out a single m8i.32xlarge only on I/O‑heavy queries, hitting 400 Gbps burst network bandwidth. For compute‑intensive joins the single node still wins, showing distributed Polars is not a universal speedup.
Cloudflare is bundling storage (R2), query (Workers), and Apache Iceberg support into a lightweight lakehouse, showing you can run simple analytics without leaving its edge network. The missing piece is robust orchestration, so teams still need external workflow tools to stitch jobs together.
Spark 4.2 folds metric governance, vector primitives, change‑data capture and an Arrow‑first Spark Connect client into the core engine. That gives teams a single, consistent source for dashboards, BI and AI agents, cutting duplicated logic and keeping data fresh for models.
Netflix engineers built a streaming‑first service topology platform that merges eBPF flow logs, IPC metrics, and distributed traces into a three‑stage pipeline. The system delivers sub‑second query responses and near‑real‑time freshness at millions of records per second, turning incident debugging from a weeks‑old map into an instant view.
Razorpay slashed its data warehouse refresh time by 90% by swapping full table scans for incremental graph traversals. A silver layer now deduplicates daily changes per primary key, secondary indexes store join columns, and a DAG scheduler only recomputes downstream models that actually changed. This cuts compute costs and speeds up merchant reporting.
cosmos.gl delivers GPU-accelerated network graph rendering directly in the browser via WebGL. It handles millions of nodes with fluid interaction, letting analysts explore massive relational data without server-side image generation. The library’s lightweight API and demos prove high performance even on modest hardware.
IBM Research shows that real‑world model routing is a systems optimization nightmare, not a simple classification task. Hidden costs like cache pricing, invisible task difficulty, and infrastructure latency can flip the expected cheap‑model‑wins on their head, forcing routers to juggle cost, quality, compliance, and speed simultaneously.
The post walks through building a unified agent memory from the ground up using MongoDB for text, vector, and graph storage, and a graph database for deep traversal. It shows why existing tools like Graphiti or mem0 still fall short and how a custom pipeline can give you fine‑grained control over ingestion, deduplication, and query serving.
Expedia Group codified a framework that forces every AI model to deliver measurable business impact, run on shared platform foundations, and be owned across product, AI, and engineering. The system includes release‑gate checklists and guardrails so models stay reliable, compliant, and scalable as they evolve.
Meta unveiled a transformer‑based graph model that learns hierarchical embeddings linking users’ latent interests to billions of advertiser offerings. By distilling sparse engagement signals into multi‑level interest tokens, the system boosts deep‑funnel ad relevance and opens new avenues for personalization across the ads stack.
Snowflake Horizon now supports the Apache Iceberg REST Catalog protocol in both directions, letting any engine read and write Iceberg tables without moving data. Databricks Unity Catalog only offers inbound access, making external tables read‑only and forcing migrations. This bidirectional support restores true data agency and cuts lock‑in risk.
You’re caught up.
That was all 15 stories in the 16 July issue.