Datafusion

Why do modern query engines pass around columns instead of rows? Because the hardware loves it. This post explains why columnar layout is so fast, how Apache Arrow represents it in memory, and how to build and manipulate Arrow arrays in Rust without treating the whole thing like black magic.

You write a SQL query, hit enter, and a few milliseconds later results appear. In between, a small compiler has already parsed your text, built a plan, optimized it, and executed it. This post walks through that pipeline with a real query and real Rust code using DataFusion.

In this article, I share my critical view on the current state of data engineering, dominated by heavyweight platforms like Spark and Databricks, and introduce Sail, an open-source engine built on top of Apache Arrow and DataFusion, written in Rust, that offers a new path: lightweight, efficient, and powerful.