CockroachDB
The name CockroachDB was inspired by the insect that goes by the same name. Just like how cockroaches have been surviving for millions of years and colonizing the entire planet and thriving, CockroachDB instances are supposed to replicate and repair data, spread naturally across multiple availability zones, and survive total regional failures. Also, once CockroachDB becomes part of a given software ecosystem, it's impossible to get rid of or replace it, just like cockroaches. Here, we will discuss why there is a need for yet another database, known as Inspiration, and provide a high-level overview of CockroachDB.
Why yet another database?
As more companies shift from on-premises to the cloud, they are looking for SQL datastores on various cloud platforms to manage their transactional data. Most of the traditional databases such as MySQL, Postgres, and Oracle are not built for the cloud. This necessitates a cloud-native, consistent, distributed SQL that can scale with the growth of data. CockroachDB fills this gap.
Inspiration
As we previously discussed in the NewSQL section, in 2012, Google published a seminal paper on Google Spanner: a globally distributed database service and storage solution. Although Google Spanner combined the best of both SQL and NoSQL and was very useful for a lot of applications, it was not available for public usage. Also, Google Spanner was and still is not an open source project and has only been available on Google Cloud Platform since 2017. So, this created a necessity for an open source Spanner-like database that can be used in different cloud providers and on-premises. Around 2012, Spencer Kimball, Peter Mattis, and Ben Darnell were working at Google on the Google File System and Google Reader projects. They also got acquainted with both Bigtable and Spanner during their tenure at Google. They decided to build something very similar to Spanner to make it available for everyone and started an open source project on GitHub in 2014. After a year, they decided to leave Google and founded Cockroach Labs in 2015 before officially working on CockroachDB in June 2015.
Key terms and concepts
Before we look at the various functional layers, let's look at some of the key concepts and terms. A CockroachDB cluster refers to a group of nodes that act as a single logical unit. A node is a single machine that runs an instance of CockroachDB. CockroachDB stores all the data as sorted key-value pairs. These keys are divided into ranges. CockroachDB replicates each range and stores each replica on a different node. For each range, there will be a leaseholder, which acts as a primary owner of a given range and receives and coordinates all the traffic for that range. For each range, one of the replicas acts as a leader for write requests and ensures that the majority of the replicas are in consensus, before committing a given write. For each range, there will be a time-ordered log of writes, called a raft log, for which the majority of replicas agreed upon.
High-level overview
CockroachDB is a cloud-native, consistent, highly scalable relational database. Some of the primary goals of CockroachDB are to provide strong consistency, geo-distribution of data, high availability, SQL support, easy deployment, and less maintenance. Since we will be dealing with CockroachDB internals in detail in subsequent chapters, we will just provide a high-level overview here:
Figure 1.10 – High-level overview of the CockroachDB architecture
CockroachDB exposes a SQL interface, using which clients can interact with the database. Client requests can land on any node within a given cluster and work just fine since all the nodes are symmetrical.
CockroachDB can be divided into five functional layers:
- SQL
- Transactional
- Distribution
- Replication
- Storage
The SQL layer is responsible for receiving SQL queries and converting them into key-value operations. The transactional layer ensures that all CRUD operations that happen on multiple key-value pairs are transactional. The distribution layer is responsible for ensuring ranges are evenly distributed among all the available nodes in a cluster. The replication layer ensures that ranges are replicated synchronously, whenever there is a change. Finally, the storage layer is responsible for managing key-value data on the disk.