Quack: The DuckDB Client-Server Protocol

DuckDB has introduced the Quack protocol, a new communication standard that allows multiple DuckDB instances to operate in a client-server setup, supporting concurrent writes and remote queries. This development broadens DuckDB’s applicability, enabling use cases that require multi-process access, which were previously limited by its in-process architecture.

The Quack protocol is designed to facilitate communication between DuckDB instances, whether on the same machine or across networks. It is built on proven technologies such as HTTP, emphasizing simplicity and speed. To use Quack, users install and load the extension in their DuckDB instances, then establish connections using specific commands, enabling data sharing and remote query execution.

DuckDB’s team noted that Quack was designed from scratch without legacy constraints, allowing for optimized performance and flexibility. The protocol supports various operations, including copying data between instances and executing complex queries remotely. It is available in DuckDB version 1.5.2, released as part of the core_nightly repository.

Why It Matters

This development matters because it enables DuckDB to support multi-user, multi-process environments—a capability it lacked due to its traditional in-process architecture. This expansion can make DuckDB suitable for larger-scale applications like telemetry collection, dashboards, and collaborative data analysis, where multiple processes need to read and write concurrently. It also reduces reliance on workarounds like external RPCs or migrating to other database systems, potentially simplifying deployment and improving performance.

Background

DuckDB, first released in 2019, is known for its in-process architecture optimized for data analysis and interactive use cases. Historically, it lacked native client-server support, limiting its use in multi-process environments. Prior to Quack, users relied on custom RPC solutions, extensions like Arrow Flight SQL, or external systems such as PostgreSQL to achieve multi-user access. The introduction of Quack marks a significant shift, aligning DuckDB more closely with traditional client-server database models while maintaining its core strengths.

“We see DuckDB as a universal data wrangling tool. If this means having a client-server protocol in addition to the in-process capabilities – fine. If this ends up unlocking a vast new set of cases in which DuckDB can be useful – excellent!”

— DuckDB team

“Designing Quack from scratch allowed us to optimize for speed and simplicity, drawing lessons from existing protocols like Arrow Flight SQL.”

— DuckDB team

What Remains Unclear

It is not yet clear how widely adopted Quack will become or how it will perform in large-scale, real-world deployments. Details about security, scalability, and compatibility with existing systems are still emerging. Additionally, the long-term integration with other DuckDB features and potential limitations of the protocol remain to be seen.

What’s Next

Next steps include broader testing by the community, integration into various workflows, and potential enhancements based on user feedback. The DuckDB team is likely to monitor adoption and gather performance data to refine Quack further. Future updates may include expanded features, security improvements, and documentation to facilitate adoption.

Key Questions

What is the Quack protocol?

Quack is a new communication protocol developed by DuckDB that enables instances to connect and interact as client and server, supporting remote queries and data sharing.

How do I set up Quack in DuckDB?

You need to install the Quack extension in your DuckDB instances, load it, and then establish connections using specific commands as documented in the latest release notes.

Does Quack replace the in-process architecture?

No, Quack adds client-server capabilities to DuckDB, complementing its in-process features, thus broadening its use cases without replacing existing functionalities.

What workloads can Quack support?

Quack supports workloads involving multiple concurrent writers, remote data access, and complex queries across instances, suitable for larger or collaborative data environments.

Is Quack secure?

Security features are still being developed; users should consider implementing additional security measures when deploying Quack in sensitive environments.