Next in Tech | Ep. 212: NVIDIA GTC Bursts at the Seams

View Full Transcript

Eric Hanselman: Welcome to Next in Tech, and S&P Global Market Intelligence podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Industry and Company Data at S&P Global Market Intelligence. And today, we're going to be digging into the NVIDIA GTC Conference with 451 Founder, John Abbott, who is out there at the conference. John, welcome back to the podcast. 
 

John Abbott: Hi, Eric. Thanks very much for having me back. 
 

Eric Hanselman: And thanks for being back. A very interesting conference this year. Interesting to see the evolution. What's your take on what was going on? 
 

John Abbott: Well, I think I've been to pretty much every GTC conference since 2009 when they really started having them. And to start with, it was all gaming, which wasn't really up much to do with my coverage area. I used to show the latest games on the screen and rhapsodize about ray tracing. But... 
 

Eric Hanselman: Shaders and all those other kinds of heavy-duty computations, right? 
 

John Abbott: Absolutely. And I'm not much of a gamer either, so it sort of went way over my head. But anyway, they were always looking to broaden their market opportunities. And over the years, it moved on to things like design automation, then you remember GPU virtualization and remote graphics and all that and then the term GPGPU, general purpose GPU. So, they're always looking for the opportunities, and it got more and more into enterprises and high-end supercomputing -- and they looked at parallel processing, but they never call it that because they felt that was too complicated and would put people off. And then machine learning and deep learning. And none of that was very long ago either really. It's -- and I think there's something sort of surprisingly unassuming in the fact that NVIDIA stays at the San Jose Convention Center, which is where they always ran their stuff. It's now a heaving when you go into the show itself because it's too small for 25,000 people. And they have to outsource the keynote to the local hockey stadium. 
 

Eric Hanselman: The only place to be able to hold enough people, so... 
 

John Abbott: Absolutely. And they called it actually this time, the Super Bowl of I don't know, AI, the Super Bowl of AI. anyway. And when you were at the stadium, it did feel like that a bit actually. But anyway, so... 
 

Eric Hanselman: It speaks to really what's happened with was this very niche nerdy kind of thing that was the graphics processing unit community. NVIDIA is the provider and... 
 

John Abbott: It always felt a bit like a sort of cottage industry in those days, even though it's quite a big market, but nothing like now. And now it seems NVIDIA still wants to remember its roots. So, it's not actually moving from San Jose. But even though I did hear that they try to book up the globe in Las Vegas couldn't because it's fully booked. 
 

Eric Hanselman: This is one of the challenges of growth. Well -- and they've been in this up and down journey. There was this big boom during cryptocurrency when suddenly GPUs were needed to mine Bitcoin. And then that fell away and then suddenly, this crazy little thing called AI happened. 
 

John Abbott: Yes. And it's still actually quite a vibrant show, I think, because it's still basically developers. So, I mean, I remember back to Java 1, which was really exciting for the first 10 years and then it declined into a sort of morass of nothingness. 
 

Eric Hanselman: Well, interestingly enough, I've got some research coming up because I was at the reinvigorated Java 1, which they haven't had. 
 

John Abbott: I saw that was on up the shore, Redwood Shores, yes. 
 

Eric Hanselman: Exactly, just a little north of San Jose in Redwood Shores. 
 

John Abbott: Is that -- is it reinvigorated... 
 

Eric Hanselman: It is very much so. Java turned to 30 this year. But again, very much developer focused, something that, I guess, maybe Java One aspires to the same kind of growth that the GTC is dealing with. 
 

John Abbott: It's just that at its peak, it was a huge show. It was a huge, massive show. And then it Java became slightly less central to things. I know it's still there and will be forever. But it then disappeared really effectively. And I think GTC is at that sort of peak stage now. So, it will be interesting to see whether next year, it's up or down. And I suppose that depends on quite a few other factors. 
 

Eric Hanselman: But the conference itself has that developer centricity to it, although, of course, now everyone on the planet wants to figure out what they're going to do and what's taking place. And to be honest, just the developer community. 
 

John Abbott: Absolutely. And that's what I felt was really interesting because I think CEO, Jensen Hung was actively aware of NVIDIA's place in the Magnificent 7 stocks and it's the responsibilities which go with that. So, he said, for instance, he called out in particular, the response to DeepSeek in January. The general response was completely wrong, and he was really aggressive about it really for him because he's a very sort of laid-back person really. I mean, the narrative then was that new ways of building models would mean less demand for GPU clusters in the future. 
 

Eric Hanselman: And for those of our listeners who hadn't heard, the Deep-Sek model developed in China under -- in an environment in which access to GPUs was constrained, actually went out and did a whole bunch of optimization, both primarily in training and built a model with a lot less GPU requirements simply because they were operating in an environment where they had to do those optimizations. And the stock market wiped $1 trillion or so dollars’ worth of valuation off a whole bunch of different companies in worry that, in fact, this was going to stunt the growth of all of the technologies that were associated with it and in the process, take a lot of value away from those who are building infrastructure to run AI. now coming on a couple of months later, there seems to be general agreement that, no, no, we're still going to need lots of AI infrastructure to support all this. But interesting to see that Jensen was out there reaffirming that this is really where they are. 
 

John Abbott: Yes. And his argument was just that you'll always need more GPUs even if things get more and more efficient. And it brought me to mind of the VMware situation when VMware came out and said, right, we're consolidating all these servers and you might need so many servers in the future, but that didn't work out. We had to buy even more servers because of VMware. So -- but also, he -- I mean, I spoke to quite a few staff members at NVIDIA, and it sort of burned in their brain this event on the 20th of January, which was -- which they call DeepSeek Monday. So, they obviously found it quite dramatic period after months and maybe years of sort of uninterrupted growth, suddenly, they saw their share price go down. 
 

Eric Hanselman: Well, it's been interesting watching the overall reaction because, of course, there was that very dramatic market reaction and then all of the back and forth around it, having to go back to economics textbooks to the point at which suddenly everybody is quoting Jevon’s paradox from the industrial revolution of if you make something -- if you make the consumption of something more efficient, you will, in fact, not reduce the consumption of it, but in fact, accelerate it because it's now less expensive to consume. Yes, it's been a wild ride. 
 

John Abbott: But also in the keynote, Jensen spent at least 30 minutes of his time talking about the math’s of tokens per query and how tokens are the main means by people who invest lots and lots of money in GPU infrastructure can claw some of that investment back. So, he spent a long time going through the math of that and how the shift from knowledge-based models, which typically deliver a one-shot response to a prompt based on pretrained data is now shifting towards this chain of thought reasoning models where the problem is broken down into multiple stages that hone the eventual answer and confirm its accuracy from multiple sources. And that process generates a lot more tokens at every stage. I think he was talking about hundreds of tokens for those kind of queries rather than a single one for a one-off shot thing. I mean that sounds a bit like a justification for spending lots of money on GPU infrastructure. But he felt that was important to do that in the keynote, which is normally focused on technical things. 
 

Eric Hanselman: Well, and interesting to identify that there's a shift from what he's calling out is the shift towards being able to get good answers. It was interesting hearing him talk about it about I caught the replay. It was interesting in hearing him talk about that and getting very specific about this is what really effective AI is going to require. And then saying, yes, it requires more tokens, but -- this is -- and calling out specific examples about, look, you can ask questions, and you can get an answer. But if you want something that is going to have depth and accuracy, then you really need to go to the next level of AI reasoning, getting into reasoning and reason-based approaches to be able to actually deliver this. To your point, really sort of doubling down on this idea that this is where we're headed, and that consumption is driving this. 
 

John Abbott: Absolutely. And I suppose the sort of main headline of the GTT show was the launch of Blackwell Ultra, which is the sort of enhanced memory version of Blackwell, which was announced last year's GTC, but it only really just recently begun sort of fully shipping. It's now fully shipping. He assured us and Ultra will come a bit later, and he also talked about some of the rack models, the 72 NVL72, which has got 36 Grace CPUs and 72 Blackwell B200 GPUs in it. So, there's really huge rack scale systems. And I always think -- and he has confirmed multiple times at the show that NVIDIA is not a chip company, it's a systems company, and it's now a systems and a software company as well, which is another big focus of the show. 
 

Eric Hanselman: Well, it's interesting watching because GTC, as you were pointing out, started out as a developer environment, which is how do you get the most out of your GPU environment? How do you code around them? How do you build applications with them, all those pieces to this. And that -- and that's evolved from NVIDIA just simply being a semiconductor device manufacturer to now one that supports a very large software effort -- if you take a look at how they're optimizing models, specifically for the CUDA software architecture and of course, enhancing the CUDA software architecture, this is no longer just a device pipeline. This is all of the software that wraps around it. Now these massive DGX systems that -- well, some massive, some small, but DGX systems that actually are built around them. So full built platforms around this, we're in a very different place. 
 

John Abbott: Yes. And on the software point, really, CUDA didn't get that much of a looking in terms of new developments, but he really did emphasize how much it continues to be an enabler. I mean, they're talking about 6 million developers now and 900 CUDA-X libraries as AI-related libraries that are available for people to -- so they don't have to start from scratch on everything from physics, data science, deep learning, quantum chemistry, weather, medical imaging and all that. And then on the... 
 

Eric Hanselman: This whole panoply behind him of all of the libraries that they made. 
 

John Abbott: That's right. I thought -- but then also on the business side as well, the less sort of glamorous side, if you like, not just the huge high-performance computing science projects that are going to save the planet or reduce disease or whatever, but real sort of business things are made everywhere at the show was this NVIDIA AI enterprise product with all the systems integrators, Accenture, EY, Deloitte and systems and storage companies like Dell HP and Nutanix, all showing this thing off with the various boxes of NVIDIA software components and then their own tact on alongside it. So that product has AI frameworks, application blueprints and the NIM container where they actually package up these applications and make them easier to deploy just like a container. So that was a much bigger emphasis than I've seen in the past on just sort of work a day software that's AI-enabled. -- if NVIDIA is really concerned that this stuff is not just shelfware or it's not just fueling the infrastructure build-out without actually making sure that people are actually using this stuff for real-world things at the same time. 
 

Eric Hanselman: Well, expanding that ecosystem across all of these various ecosystem players, storage players, server players, I mean, the fact that you got Michael Dell up there, it really speaks to the extent to which this has become a much larger ecosystem. 
 

John Abbott: Yes. 
 

John Abbott: I mean another thing that I thought was really interesting is they opened up their road map. So, for 4 years ahead. And they claim that this is something -- I'm not sure, but they said that most people go that far out. And certainly, Intel doesn't -- goes a certain distance. And AMD is a bit more secretive, I would say, about what's coming up next. But they laid it out for 4 years with the Vera CPU and Rubin GPU ahead and then beyond that, kind of the Fainman is the generation. 
 

Eric Hanselman: Omine. It's continuing that process of identifying various luminaries in various fields, but computing and physics, but... 
 

John Abbott: I mean the people who are investing, especially the GPU clouds, of which there are 7 or 8 there at the show floor, they do need to know that there's a road map and how it all fits together and what they're going to do in the future. 
 

Eric Hanselman: Well, because they are in a world now where -- I mean they have existed where there hasn't been a lot of strong direct competition. But now there is. You've got AMD coming out with the MX300i and all of its family beyond that. You've got a whole set of potential new semiconductor companies that are all looking at being able to capture some of that massive market. But if you look at where NVIDIA is because it is such an ecosystem play, because there is so much to it, it's going to be difficult to displace. 
 

John Abbott: Absolutely. They've built a moat around CUDA and all its ecosystem, and that's really hard to break into. So, AMD, I would say, has got credible technical products, but it's much weaker on the software side of things. And it's an effort for people to use an alternative to NVIDIA because they have to start reinventing the rest of that software stack as well. Jensen was very funny in a session he had with the analysts where people asked about some of the alternatives, and he went on a bit of a rant about APU,PU, CPU, DPU is saying splitting all these AI functions into sort of more discrete devices, that's stupid. And he thought GPU, GP, GPU is stupid as well. So, he started talking about what is a graphics processor. It's not graphics. It's not really general purpose. But -- and he basically said, it's the software and the programmability of that hardware that really is the distinctive feature that you've got to worry about. 
 

Eric Hanselman: It's not what it is. It's what you can do with it. 
 

John Abbott: Yes. So that was great. And where he ended up in the end, he started physical AI and robotics was sort of left at the end of the keynote, and it didn't get that much of a mention because it's still quite a long way out really, but he claimed that physical AI is where everything is ultimately heading. And things like Omniverse, which NVIDIA started talking about 4 or 5 years ago and hasn't really done very much yet. But he's still pushing that as the sort of operating system for digital twins where you get all this information in and start really utilizing it for design before you actually build the products and that. And Cosmos, which is the framework for physics aware framework. So, it knows the properties of all these materials and how they should react. -- and they introduced a new Newton physics engine as well, which was developed in conjunction with Google and Disney Research. So, all that's still sort of going along. And he's really saying that's where the end point is, and we're building intelligence into all these different vertical markets, and that's why AI is a big deal and where the value will be ultimately. 
 

Eric Hanselman: Well, because he was on stage with a robot, created an acute little one, courtesy of Disney. But I thought you make an important point, which is that this is where that journey is headed. And there are a lot of pieces that have to come into play. The idea of digital twins and being able to effectively train and simulate in digital environments -- this is something that Ian Hughes has been talking about quite a bit in terms of being able to build digital environments in which to train so that if you have a robot, if you have an autonomous vehicle, whatever that thing happens to be, you've got the ability to now spin it up and train it in a virtual world that responds in the way that the real world would respond that you have digital twins of the kinds of things that you're interacting with and use that as a way to more efficiently train your -- whatever your particular product happens to be and to leverage it that way. And it seems like we're finally starting to get to that convergence of an understanding of the needs for physical environments and where they have to be able to both interact and train. And the other thing I thought was really interesting that he commented on was the need for data around all this and the challenges that most organizations face in data. 
 

John Abbott: Yes. And in fact, they didn't go into it very much, but they've got a sort of storage framework, which they're working on with all the storage companies as well. And a lot of those storage companies I spoke to at the show -- they wouldn't go into many details, but they said, yes, we're working with NVIDIA on this on getting that software stack a bit more tuned for AI so that you can feed the GPUs with the data you need. 
 

Eric Hanselman: Well, yes, it's something that we've talked about on a number of different episodes, which is getting to that next stage of having a data pipeline as opposed to an application development pipeline and what's necessary for that. and being able to get the right data at the right scale that is robust enough and that the data part of this starts to become pretty significant. well, and we keep saying, in AI, the data is the application. You've got to be able to get beyond that idea of data is the stuff that we've just sort of stored to data really is the -- what we're building the end result on top of. 
 

John Abbott: Yes, absolutely. And another area related to that is the networking side really. And I think during the conference, I heard for the first time that Ethernet will probably be the ultimate outcome for its network, even though they've got InfiniBand and they're not giving up with it in any way for the big scale of scale out, they're looking at Ethernet, the spectrum switches... 
 

Eric Hanselman: Well, so how long ago? Actually, I should have looked this up. NVIDIA acquired Mellanox, a networking provider that was one of really the key proponents of InfiniBand and its own special low latency, high-performance interconnect in competition to Ethernet. Yes. 
 

John Abbott: But all the clouds use Ethernet, all the big clouds really, and they don't really want to have 2 separate stacks. So, I think ultimately, they'll push for that. The other thing about networking that came up with the show was this co-packaged optics where you reduce the expense of power-hungry optical transceivers by actually integrating that onto the chip, which they've been working on with TSMC. And that could take out a lot of power requirement and a lot of the cost of optical networking. -- inside. 
 

Eric Hanselman: The networking part of my heart was found that it was near and dear to me there. Yes, the idea that -- because the challenge, of course, is that we've had optical interconnects once you got outside the semiconductor world, but that everything that was within -- and we're now environments in which you've got chaplets in which you've got GPUs and memory, and all sorts of things all crammed on to lead frames or interposers or packaging. But the interconnect is all electrical at really high speed, which, of course, choose is flinging all those electrons around takes a lot of energy. And if we could do that optically, we cut down a lot of the power requirements. We also, as a result, cut down the heat dissipation so that so many of these devices now are in the hundreds of watts in a package. if we could start to roll that down, now we get to package them closer together, it uses less energy, all these other good things that start to compound on top of themselves. 
 

John Abbott: Yes. So, I mean, one of the -- there's a couple of things that they didn't really talk about very much. And that is despite what we've been saying there still could be a pullback from some of this AI infrastructure build-out. It's going on at an incredible pace. And that's a big danger, I think, for NVIDIA with alternative models coming up, they're using different techniques. So, they didn't really talk too much about that. And then the other thing they didn't talk about at all really was geopolitics. And I think that's because Jensen is a very unpolitical sort of person. he really doesn't want to get involved in that stuff. But -- but there's so much going on there, all the tariffs, which could affect NVIDIA's fabless semiconductor business model, which is based around, all the uncertainty around that. And even more so, I think, the environmental issues of this massive build-out and what that really means longer term. And I felt they didn't really address that. If things go on at even greater momentum, where will all the required energy come from and at what price to the planet? 
 

Eric Hanselman: A good question and actually something we've talked a lot about on the podcast. And those concerns about where does this go? And what does that growth rate look like? And again, getting back to the device power efficiency, hey, maybe optical buys you back some of that. But it is that question. We keep building and building and building and all those electrons have got to come from somewhere. So... 
 

John Abbott: They are definitely addressing the efficiency as much as they can, but no manner of -- well, I think all the efficiencies are really going to come from software. And that's -- and I think NVIDIA should maybe -- well, I'm sure they are, but maybe they should talk a bit more about how they're trying to get the software efficiencies to reduce the hardware level, which in some ways is counterintuitive to their business model of throw hardware at it. 
 

Eric Hanselman: But it is something that they've already invested significantly in. So, they spend a lot of time optimizing models so that they run most efficiently on NVIDIA GPU hardware. So, they've got certainly a vested interest to identify that. And it is interesting to have Jensen talking about token consumption. because, again, it's been that question of how do you actually monitor usage? What is that metric? And we're starting to get to a world in which organizations are looking to manage efficiency. You've got basically all of the AI platform folks that are out there, IBM, Red Hat, Microsoft, everybody talking about more efficient models, taking a step back a little bit from trillion parameter models and looking at efficiency and optimization and DeepSeek we were talking about, certainly was one, I guess, major milepost in terms of thinking about efficiency. So clearly, more to be done, that question of whether or not developers are willing to sacrifice some level of speed for efficiency. Interesting. So, what do you see in looking a little bit ahead? Where -- is this going to be something that the repercussions from GTC are pretty much shaken out? Or is there more that should be rolling out over time? 
 

John Abbott: Well, I mean, I think their road map is quite nice and clear, and it seems fairly solid that they're going to try and do this yearly cadence of new introductions. I think there was a bit of a schism there as to whether this will affect their previous GPUs. So, NVIDIA Jensen was talking about, you don't want to buy a hopper anymore because Blackwell is so much better. And he caly joked about himself being the chief revenue disposer or something along there. But actually, when you talk to companies, they're so they want as much hardware as they can get, and there's still -- quite a few of them are still pretty happy with hopper. So, I suppose it's how they follow through on this road map and broadening out their software stack to become almost like a VMware of AI, which seems to be the direction where they're heading. So, I'm looking forward to seeing how that plays out. 
 

Eric Hanselman: At least as long as they're not the Intel of AI. That's right. 
 

John Abbott   00:29:44

And with Intel now struggling for multiple reasons, it really has cleared the way for NVIDIA in this accelerated AI, which they're now saying everything is accelerated. So, if you haven't got an accelerator, you're irrelevant, and that's implied their opinion. 
 

Eric Hanselman: Well, it certainly seems to be the wave that is coursing through the markets as we stand today. Well, thank you, John. This has been great. Certainly, a lot more to look towards as this continues to roll out. 
 

John Abbott: Yes. I think so. 
 

Eric Hanselman: Well, so much more to discuss, but we are at time for this episode. Thanks to our audience for staying with us. And thanks to our production team, including Sophie Carr, Feranmi Adeoshun, Kyra Smith, and Adam Kovalsky on the marketing and events teams. I hope you'll join us for our next episode where not surprisingly, we're going to be looking at something that's also tied in AI on the data side, Page Bartley will be on talking about privacy, data exposure risks and the regulatory environment that sits around AI. I hope you join us then because there is always something Next in Tech.