Ten years of NewSQL: Back to the future of distributed relational databases

Introduction

451 Research first used the term NewSQL 10 years ago to describe a new breed of relational database players. Little did we know that the term would be picked up by the wider industry to describe a new category of products, triggering numerous marketing campaigns and industry events, its own Wikipedia entry, and dedicated academic research papers. A decade later, it is interesting to look back on that first report to examine the fortunes of the various vendors we cited as examples of NewSQL, especially in the context of other trends that have shaped the market, such as NoSQL and database as a service, while looking ahead to see whether NewSQL will continue to be relevant in the next 10 years.

The 451 Take

Looking back at the report with which we introduced the term NewSQL, it is clear that the early NewSQL pioneers failed to have a material impact on the database market. It is especially telling to compare the survival rate of those early NewSQL players with the various NoSQL vendors that were also mentioned in the report. As such, it would be easy to conclude that NewSQL was a relative failure. However, that would be to overlook the complexity of what the various NewSQL database players were trying to achieve, as well as the lessons learned by the second generation of 'distributed SQL' vendors, which represent the evolution of one subset of the overall NewSQL category. Customer demand for globally distributed databases, especially those delivered as a service, is now significantly stronger than it was a decade ago, while the newer vendors have also learned from the success of the NoSQL vendors to focus not only on the technical elegance of their databases, but also developer engagement and ease of consumption. The term NewSQL may not survive the next decade, but its impact will be felt on the database market for many years to come.

New what?

When we started using the term NewSQL in early 2011, it was not our intention to define a market category. We were simply using the term to refer collectively to a group of emerging database products and vendors. What the various products included in this loosely affiliated group had in common was combining the benefits of the relational model and distributed architectures.

The various NewSQL offerings could be grouped into three sub-categories:

• Brand new distributed relational products.
• Transparent database sharding middleware.
• Distributed database-as-a-service offerings.

Additionally, we initially included database storage engines that were designed to improve the performance of relational databases to the extent that horizontal scalability was no longer a necessity. However, we had thought better of this by 2016 and our collaboration with Andy Pavlo, associate professor of Databaseology in the Computer Science Department at Carnegie Mellon University, on the definitive description of NewSQL.

Either way, our initial 2011 report described the various players we saw emerging in this space and explored the potential for merger and acquisition activity. While many of the vendors mentioned did indeed end up being acquired, others shuttered their doors sooner than expected, and not one achieved the big money exit the various founders and investors would have been hoping for.

Today, 451 Research estimates that the various NewSQL database providers generated revenue of $587m in 2020, which is a tiny proportion of the overall $35.6bn operational database market and substantially smaller than the $4.9bn generated by NoSQL database providers in 2020.

Exit, stage left

The relative failure of the original NewSQL vendors to make an impact in the database market is highlighted by comparing the fortunes of the various NoSQL vendors highlighted in our 2011 report. The majority of these are still going strong today. One of them (MongoDB) is a public company, and others (Couchbase, DataStax) are heading in the same direction.

In comparison, the NewSQL section of that 2011 report would today best be illustrated by a collection of tombstones. Many of the vendors mentioned were not long for this world, while other NewSQL vendors not mentioned in that original report also met an early demise.

• Having started out as a data grid provider for the communications sector, Xeround turned its attention to cloud databases in mid-2010, but by 2013 had announced that it was discontinuing its public cloud service.
• GenieDB was put back in the bottle in 2015, having previously shifted its ambitious plan to deliver a globally distributed relational database toward database as a service two years earlier.
• Having failed to drum up significant business for its shared-disk clustering database storage engine for MySQL, ScaleDB pivoted to database virtualization in 2013 before eventually closing its doors sometime in 2017.
• TransLattice may have acquired managed database provider StormDB in 2013, but it failed to build a business around its PostgreSQL-based TransLattice Elastic Database and quietly vanished without a trace.
• Deep Information Sciences promised much with its next-generation general-purpose database architecture, but failed to deliver – it became the open source Deep Software Foundation in 2017 and subsequently disappeared.

Even those that were acquired failed to land big exits.

• Having spent some time in the doldrums, hoping for a prevailing tailwind, sailing-inspired Schooner Information Technology was acquired by SanDisk in 2012 for its database acceleration technology.
• Akiban was acquired by FoundationDB in 2013 to add an SQL layer to the latter's NoSQL distributed database before FoundationDB was itself acquired by Apple in 2015 and later made available as open source in 2018.
• Launched in 2009, FathomDB was an early database-as-a-service pioneer, but failed to gain momentum and was acquired by Meteor for its development platform in 2014.
• Database clustering and replication specialists Continuent was acquired by VMware in 2014, before being spun off again two years later. It remains an ongoing concern.
• Tokutek was acquired by Percona in 2015, primarily for its distribution of MongoDB rather than its MySQL storage engine. The technology is now Percona Server for MongoDB.
• ScaleBase was acquired by fellow load balancing and database sharding player ScaleArc in 2015, before the latter was itself acquired by ESW Capital's Ignite Technologies in 2018.
• Having started out with a database hardware appliance and transitioned to offering a software product in 2014, Clustrix was acquired by MariaDB in 2018. The technology lives on today as MariaDB Xpand.
• Database-as-a-service specialist ClearDB was acquired by RDX in early 2019, prior to the latter company acquiring Navisite and assuming the cloud service provider's identity later that same year.
• Initially known as NimbusDB, NuoDB struggled to build a substantial customer base for its murmuration-inspired database, despite early support for cloud-native architecture, and was acquired by Dassault Systemes, its largest customer, in late 2020.

In fact, VoltDB is the only NewSQL vendor mentioned in that 2011 report that continues in the same guise today, although it has very much narrowed its focus and is now primarily targeting the telecommunications space (and 5G use cases, in particular). Additionally, SingleStore, formerly known as MemSQL, continues to target the opportunity for hybrid workloads – handling both OLTP and OLAP processing.

Others remain in operation with an altered focus: Database sharding specialist CodeFutures became AgilData in 2015 and is now a cloud- and data-centric services provider, while JustOne Database rebranded as Edge Intelligence in 2017 with a focus on analytics at the edge.

A successful failure?

It is clear from the list above that the original NewSQL vendors did not succeed in disrupting the database market. In fairness, they were setting out to tackle one of the most significant challenges in computer science – combining the scalability advantages of NoSQL with the structure, consistency, performance and transactional support provided by the relational data model (there's a reason why the speed of light can be considered a competitor when it comes to distributed transactions).

They were also competing in a sector of the market that is dominated by giant database and cloud incumbents, such as Oracle, IBM, Microsoft, Amazon Web Services and Google. Those early pioneers were, in hindsight, perhaps too early, and were ahead of customer demand for both distributed data processing and database managed services (there's a reason why we once described the cloud as where database startups go to die).

Additionally, while early NoSQL products and services lacked support for traditional relational database concepts, many of those have been added in the intervening 10 years, making NoSQL databases realistic alternatives for at least some potential NewSQL workloads.

The next generation of distributed relational database vendors has fared better, however. Cockroach Labs has more than 300 customers and a $2bn valuation; PlanetScale recently announced the launch of its developer-focused database service; and PingCAP launched the public preview of its TiDB Cloud managed service, having launched version 2.0 of its database in 2020 and raised a $270m series D funding round.

Meanwhile, YugaByte recently expanded to EMEA and APAC, having raised a $48m series B round to bring its total funding to $103m, and Fauna has established itself as a provider of a managed serverless database consumed by customers via an SQL API. We would also include horizontally scalable cloud database services in this next generation of NewSQL offerings, including AWS Amazon Aurora, Google Cloud Spanner and Azure Database for PostgreSQL – Hyperscale.

Together, these next-generation offerings could be argued to be delivering the 'truest' examples of NewSQL, given that they have rearchitected the concept of the relational database for a globally distributed architecture. As such, it is somewhat ironic that they have – by and large – stopped using the term NewSQL, preferring 'distributed SQL.' We don't blame them, 'distributed SQL' is a more descriptive term that better articulates their value.

In addition to highlighting the technical merits of a distributed architecture, it is notable that the various distributed SQL vendors are all also highly focused on developer engagement and highlighting their potential benefits in terms of ease of adoption, ease of use and productivity improvements. This is an area in which the first generation of NewSQL vendors certainly fell short in comparison with the various early NoSQL vendors.

Either way, there is growing potential demand for this second generation of offerings. While only 28% of respondents to 451 Research's Voice of the Enterprise: Data & Analytics, Data Platforms 2021 survey have a globally distributed database in production today, 58% expect to be using a globally distributed database within a year, and 70% expect to be doing so within three years.

Figure 1: Globally Distributed Database Adoption Plans

^{Source: 451 Research's Voice of the Enterprise: Data & Analytics, Platforms, 2021
Q: Which of the following best describes your organization's usage of globally distributed databases? n=560}