AI in the datacenter industry — hype or growth catalyst

While AI has been around in one form or another for quite some time, OpenAI LLC's ChatGPT and other generative AI services sparked a new interest in the technology, which seems to be fueling a new era for growth along the entire supporting infrastructure chain. From a datacenter infrastructure perspective, the industry is seeing new and interesting growth in some markets. It is worth pointing out that the current growth wave in the US started in 2021, nearly a year before anyone had heard of ChatGPT and the AI rush. Building on the back of the increased demand brought on by COVID-19, cloud service providers had already started ramping up deployments. Taking into account all the investments in the latest CPUs and (mostly) graphics processing units (GPUs) by cloud service providers, it would seem we can expect further changes for the datacenter industry.

The datacenter industry loves a good hype story. Remember when internet of things, blockchain, 5G and the metaverse were going to radically change the datacenter industry, ushering in a new wave of growth … and then it never happened? Or at least hasn't happened. Can the same be said for AI? Like the others, perhaps the jury is still out. It is worth remembering that "the cloud" was on the hype list, and has now radically changed the datacenter industry as we know it, and is ushering in a new wave of growth. The escalating rack densities that everyone was talking about 10-plus years ago were finally brought to fruition by the cloud and the way we currently think about infrastructure. Furthermore, most of the AI workloads today live in the cloud, so any amount of AI growth will inevitably fuel further cloud growth, which we are seeing play out in terms of cloud revenue, the number of datacenters and the infrastructure cloud providers are deploying. While we cannot attribute 100% of the current construction boom in certain datacenter markets to AI, it is reasonable to assume that AI growth is helping. Will AI build on the momentum of change the cloud providers have already started? It seems reasonable to think so at this point, but it remains to be seen how radical that change might be.

What we know (so far)

From an underlying infrastructure perspective, what we know is that a few markets across the US are seeing new and interesting growth, and that growth is being driven by the hyperscale cloud providers. Northern Virginia is somewhat hard to make heads or tails of, because it has seen consistent large-scale demand for many years, and continues to do so. However, Chicago, Dallas and Phoenix seem to have something "new" playing out — particularly Phoenix. According to the 451 Research Datacenter KnowledgeBase, Phoenix is one of the fastest-growing markets in the US, and it is being driven completely by the hyperscale cloud providers. Looking at both self-built and leased facilities, the Phoenix market is on track to grow by an additional 1.1 GW of capacity by the end of 2027. That is, if everything that has been announced gets built; the current feeling is that this won't be a problem.

Phoenix was an interesting market before the current cloud boom, as it sits strategically close to the West Coast markets of Southern California and Silicon Valley, with much cheaper power and land than either. While this is likely a factor in cloud providers choosing the city, it would seem that the city's power companies' ability to deliver power in a timely manner is probably just as big a factor. While other markets (like Ashburn, Va.) have struggled with distribution and transmission, the greater Phoenix metro hasn't had these issues — at least not until very recently in the East Valley, but even still the West Valley is pushing ahead.

Focusing on Phoenix as a possible bellwether, providers there noted that the power requirements per data hall have been on the rise. By our numbers, somewhere around the end of 2019, the watts per square foot (WPSF) on delivered datacenters started creeping up from about 150 WPSF to now over 300 WPSF for the latest builds in the metro. Generally speaking, we use 36 square feet per rack when calculating how many racks fit in a given data hall, which works out to an average of just under 11 kW per rack being provisioned in these datacenters. While that number is not groundbreaking, the reality is that the cloud providers are likely deploying fewer racks than our calculations, meaning there may be rows of racks operating at tens of kilowatts. Having said that, all the wholesale providers in the metro we spoke with stated that they weren't having to do any re-architecting of their data halls or cooling methodologies to accommodate the current requirements — yet.

Depending on who you ask, datacenter providers can generally adequately cool a datacenter full of 25-kW racks (some claim as high as 50-kW racks) without doing anything "special" from a cooling perspective. The providers did note that the hyperscale cloud providers were wanting to chart out a plan to accommodate over 50 kW per rack in future datacenters; likely within the next five years. In response, these same providers are currently working with liquid and other cooling technologies in order to best accommodate these requests, and some are already delivering liquid to the floor of the datacenters on a limited basis. So we can say that there seem to be changes coming rapidly for future datacenters, but not necessarily to the current batch coming out of the ground.

While hyperscalers are leasing a lot of datacenter space across the country, they are also buying up a lot of CPUs and GPUs. NVIDIA Corp. noted in its second-quarter 2023 earnings call that datacenter compute revenue had almost tripled year over year, and this was primarily being driven by an accelerating demand from the cloud service providers and internet content companies, including the likes of Amazon Web Services Inc., Google Cloud, Meta Platforms Inc., Microsoft Corp. (Azure) and Oracle Corp. (Oracle Cloud) — the very same companies building and leasing a lot of datacenter capacity in the Phoenix market, as well as other markets.

Furthermore, all three of the major hyperscale cloud providers noted in their recent earnings calls the intent to bolster their infrastructure spending, which will include further investments into AI-related infrastructure. Microsoft's CFO Amy Hood said that the elevated spending would continue well into next year. In all cases, company representatives noted that AI was only part of the reasoning for increased infrastructure investments. This is to say that these companies aren't slowing down on their other infrastructure offerings; AI-related services are simply part of the overall portfolio.

Reading between the lines of NVIDIA CEO Jensen Huang's comments during his keynote speech at SIGGRAPH 2023, it seemed a foregone conclusion to him that AI workloads would ultimately live in the cloud. Assuming that this is true, to say that AI is growing at a rapid rate is to say that the cloud is also seeing a lift. This was reflected in the latest earnings from Amazon.com Inc., Google LLC and Microsoft, in which each company saw boosted revenue attributed in some measure to early interest in AI. When we look at our Voice of the Enterprise data, respondents aren't quite as decided. Of the represented companies that noted having a strategy for AI/machine learning, 50.9% were developing applications using cloud-based AI and ML services as their primary strategy, while the rest were using other avenues. Ultimately, this is good news for datacenter providers, whether they are servicing the hyperscale cloud providers or enterprises leveraging the technology.

Conclusions

In reflecting on the events of the last 12 months or so, there are a number of things we can say are true:

– The hyperscale cloud providers are seeing growth as a result of AI-related services.

– The hyperscale cloud providers are planning to invest more in infrastructure that will ultimately support AI-related services, in addition to their other services.

– The hyperscale cloud providers have been leasing large amounts of space and power in several markets across the US at a pace not previously seen in those markets.

– The datacenter providers that are building for these hyperscale cloud providers are noting escalating density requirements, with further increases on the horizon, although the acceleration does not seem out of reach for the various datacenter providers.

Based on these points, it seems reasonable to conclude that the hyperscale cloud providers are ramping up their deployed infrastructure, and that this is likely due in some part to a broadening interest in AI-related services. However, based on the current build requirements, it does not seem as though there are entire datacenters being devoted to only AI (yet), rather AI-related infrastructure is being deployed alongside other infrastructure types. It is fair to say that this is really hard to know, given that the hyperscale cloud providers don't disclose what workloads are being deployed where. In any case, whatever is being deployed isn't pushing datacenter providers to totally reimagine their designs, at least not for now.