DuckLake Comes to Apache DataFusion With Open-Source Integration
We implemented DuckLake for Apache DataFusion, supporting multiple databases as catalog backends. The integration gives DataFusion a lakehouse format that manages snapshots and catalog metadata for Parquet files stored in object storage. Our implementation is running in production, and we have open-sourced and donated it to the Apache DataFusion Contrib organization.…
RT @__AlexMonahan__: DuckLake is now multi-engine! You can use DataFusion, an open source Rust query engine, with DuckLake as the storage!…
🤔 Did you know that using DuckLake is not limited to DuckDB? 🤯 Despite the similarity of names, DuckDB is just one of the engines for DuckLake, but you are not limited to using DuckDB at all! 🔥 Hotdata's DataFusion DuckLake implementation is packed with features: it supports both reads and writes, as well as different catalog databases including PostgreSQL and SQLite. 👉 In today's blog post, Divya and Eddie from Hotdata explain how DuckLake fits into their agentic use cases and introduce their DataFusion DuckLake engine.
Original title: Bringing DuckLake to DataFusion
Samuel Times preserves the original link so every selection remains auditable.