Spark

A practical guide to setting up a clean and reproducible environment with Docker, Jupyter, and RustRover to work with Sail, whether as a user or contributor. From launching services with docker-compose to debugging locally without installing any dependencies on your machine.

In this article, I share my critical view on the current state of data engineering, dominated by heavyweight platforms like Spark and Databricks, and introduce Sail, an open-source engine built on top of Apache Arrow and DataFusion, written in Rust, that offers a new path: lightweight, efficient, and powerful.

Defining the serializer and transformation from Yaml to Core

Defining the model core from Scala to Spark

Exploring how to create Apache Spark ETL workflows with zero code.