Listen: Next in Tech | Episode 65: The operations side of AI/ML

View Full Transcript

Transcript provided by Kensho.

Eric Hanselman

Welcome to Next in Tech, an S&P Global Market Intelligence Podcast with the world of emerging tech lives. I'm your host, Eric Hanselman, Principal Research Analyst for the 451 Research Arm of S&P Global Market Intelligence. And today, we will be discussing the latest on machine learning and artificial intelligence with Nick Patience, a Research Director for that team.

Nick, welcome back to the podcast.

Nick Patience

Thanks, Eric. Good to be back.

Eric Hanselman

Great to have you, especially with the latest addition of the AI and ML study in hand. What's new? And what are the key takeaways?

Nick Patience

Yes. So as regular listeners to your podcast will now, we do a couple of these surveys a year on the Voice of the Enterprise AI and machine learning. One is focused on use cases and the other is focused on the infrastructure underpinning AI. And this one is use cases. And the central part of it, as the name suggests, is looking at use cases and, in our case, looking at it, it deep dives into 6 particular industries, and we can talk a little bit about that in a minute.

But in terms of what's new, we also focused more on ML Ops this time because ML Ops is crucial to enabling organizations to get AI and machine learning deployed at scale. And we also dug a bit deeper on skill shortages, which is always a question we ask and we ask what companies are doing to remediate those skill shortages. And we also looked into levels of trust organizations having their own AI applications and also a bit more about their appetite for AI regulations.

But on the ML Ops thing, I think just to dwell on that for a second. One of the questions we actually asked was, how do you define ML Ops? And the reason that's an issue is because the specialist ML Ops vendors arrived in the last 3 or 4 years, very much focused on the post-deployment phase of machine learning. So once the model has been trained, is deploying it and then what happens after that is it needs to be managed and monitored and sometimes retrained and often retrained. And that's what ML Ops used to mean really up until maybe 12 to 18 months ago.

And then the hyperscale cloud vendors started getting involved, and they offered some of these ML Ops tools, but they prefer and others as well prefer the term to mean the entire span of creating, developing and deploying machine learning models starting right at the beginning with the identification of data, the data integration issues and then the training and then the deployment and the post-deployment monitoring management.

So we actually asked our survey base, what do you think it means? And it was interesting because we found 46% of our respondents as there's 1,000 people in the U.S. and Western Europe who are very familiar with what their organizations are doing with AI and ML, 46% of them actually said it to them, it meant the full span meaning and 38% said it was specifically post-deployment and -- but additionally, 16%, even though they're familiar with what's going on, were actually unclear about how to define ML Ops. So it's still something out there, which needs to be defined.

Eric Hanselman

Well, I mean, we're still in a relatively early stage in terms of where the technology is. And even I think to the extent to which people just sort of interchangeably swap back and forth artificial intelligence and machine learning as though there isn't a particular distinction. And then when you actually get into the practical and pragmatic aspects of really the operational pieces of each of these disciplines, clearly, that's an area where it's still relatively early days.

Nick Patience

Yes, you're right. It is -- those terms I use interchange. I think, I mean, when we first started really going out this subject in earnest, and we started back 22 years ago, covering it to some extent, but really sort of 2015 onwards, we had to initially define this. And obviously, as I think most people here will know by now, the machine learning is a subset of AI, but it's machine learning, which is what's going on in enterprises.

And there's other kinds of AIs reinforcement learning. There's obviously deep learning and there's other kinds. But machine learning is the core thing, and it does presented on a challenges. In addition to the technology challenges and related to them as well, the ML Ops tries to solve. There's obviously skills shortages as always -- I mean, there's always skill shortages, especially in the last couple of years.

And so one of the things -- one of the questions we always ask organizations is, what are your areas of skill shortage as it pertains to machine learning? And while everybody might think it's data science, data science, data science, all the data scientist is joint top is also joined by software engineers and application developers as the top needs follow them by machine learning engineers and cloud architects.

And I think as we've probably said on this podcast before, that speaks, I think, to the -- although you said, we're still at a very early stage, machine learning is becoming more integral to regular application development. And so all those other roles that you have such as database administrators, even storage admins and things like that are becoming central to the success of machine learning. And in fact, only 8% of our respondents said they don't have a skill shortages related to AI machine learning. So you can see it's still a major issue.

Eric Hanselman

Yes. Well, and before we get too far along, I also want to clarify one particular point. In the realm of various terms get kicked around, there's also AI Ops. And I just want to clarify the ML Ops versus AI Ops piece and the -- where that fits in sort of the infrastructure versus the actual use case aspects?

Nick Patience

Yes, that's a good point. It is a strange one. It still does create a lot of confusion. So the difference is, once you know it, it's pretty clear, but the names are too similar. That's the problem. So AI Ops is the application of AI to IT Ops. So it's using AI and machine learning in particular, in applications used by the IT professionals essentially to run and manage their operations, whereas ML Ops is focused specifically on how you build and deploy and monitor and manage a machine learning model. So this is much more used by the data scientists and the machine learning engineers as the kind of 2 key protagonist data scientists at the earlier stage, machine learning at the later stage, machine learning engineers at a later stage.

And so it has -- they do have quite different meanings. I mean, AI Ops is the application of AI to IT Ops and then you could have other names for the application of AI to all to HR and to marketing. But thankfully, we don't. Every single role -- as analysts, I guess one of our jobs is to make up terms, but we're not good making up AI ops. So I think it is a slightly confusing one, but hopefully, that clears it up.

Eric Hanselman

Well, it gets to the key part of -- 3 episodes ago, we had Giorgio Baldassarri and Kieran Shand on from the credit risk and analytics tools team and everything that we were talking about there was the ML Ops piece. It was all of the -- how do you actually generate the pipelines for the data? How do you actually look at figuring out how you're going to build -- bring data sets together? How do you verify and clean the data that you're working with? How do you actually build the infrastructure that's going to support it? Are you using work benches? Where do you get your data sets from, all those kinds of things or the operational pieces to build up MLPs?

Nick Patience

Yes, that was a really interesting episode. And obviously, if people haven't listened to that, I'd encourage them to go back to it. And the other thing I'd point out, it's early for kind of promotion of our central flagship event of the year, which is called 451 Nexus, which is in October in Vegas. But I will be running a panel on ML Ops and the challenges organizations facing and getting AI and ML to scale and how ML Ops can help them with that.

Eric Hanselman

Never too early to talk for 451 Nexus, come on. Well, okay, so we talked about use cases or we talked about the ops piece, I wanted to talk about the use cases aspect. There are some really interesting perspectives about different verticals that are in the study. It would be great to dig into those. Where are -- what are the key verticals do you see and the interesting use cases that surround them?

Nick Patience

Sure. So we look at 6 industries here. So we look at financial services, retail, health care, along with pharmaceuticals, manufacturing, telecommunications and then energy utilities, oil and gas is one big thing, although they're quite different industries. And although we probably haven't got time to go through all of those, so maybe I'll just pick a few of them.

Financial services has obviously been an early adopter of all sorts of technologies and the same applies with machine learning. And I think what we're seeing here is, in addition to digital security being a primary use case in financial services, especially by the largest companies in that sector. So 75% of those companies with more than 10,000 employees chose it as their primary digital security as their primary use case.

We're also seeing interest in use cases that you would -- they're actually used by smaller companies. So we're seeing things like wealth management become a much higher use case, especially amongst companies with fewer than 250 people. So although UBS and those kind of companies obviously have massive financial wealth management arm, there's also a lot of this is done at a very small level. And so there's obviously a lot of applications being built and sold that using AI machine learning to help people with that -- with wealth management and also we see it in financial forecasting as well.

So it's showing how the maturity of the technology. And obviously, machine learning is enabling layer. It's not the end in itself, but it's obviously spreading its way through the application portfolio used by financial services companies at both extremely large ones and increasingly smaller ones.

Eric Hanselman

That was interesting because that's one of the points that Giorgio and Kieran had brought up, which was just simple things like being able to rebalance portfolios effectively and efficiently, things that normally would be relatively labor-intensive and you'd have to swift through lots of potential different investment options and manage that. But if you build a model that helps you drive that effectively saves you a lot of cycles.

Nick Patience

Yes, exactly. Exactly. I think it's -- and that's the first of these industries where we've really seen that happening, I think, going into the realm of the smaller companies.

Eric Hanselman

So what other verticals do you see?

Nick Patience

Yes, sure. So retail has obviously been a very interesting space in the last couple of years. And by far, the top use case there is digital customer engagement. And we gave the responders, particular examples. So we're talking about visual and voice search. We're talking about virtual assistance, voice ordering, AR, VR and digital mirrors, all those kinds of things. And it's interesting to think about how much behavior is going to stick? How much pandemic behavior is going to stick post-pandemic? We're also not post the year, but we're emerging. And you see actually -- I'm not going to go through all the stats, but if you look at the U.S. and U.K. government stats -- official stats on e-commerce or online commerce as a percentage of overall commerce, in the U.K., in particular, things seem to have shifted.

People are -- there's far more people still doing -- when you look at the numbers in kind of Q4 of 2021, the last available versus Q4 2019, the one the last ones just before the pandemic, it's quite interesting. Is it something like 10 percentage points above now. I think those things are going to be changing. So that digital customer engagement using AI and ML and other kinds of technologies is really important.

And we also ask in our surveys. We always ask, "What are your top use cases now and what are your top use cases in 2 years' time, why we're also assuming they're not going to abandon the ones they're doing now?" And so customer engagement was very strong. I think it was 40% chose it now and 15% -- an additional 15% chose it as a future use case.

And then there's other things such as fraud prevention and digital security, which has been popular as a use case for a long time, as is payment processing. And within payment processing, the use case of machine learning is often to do with some kind of fraud prevention. And then also within retail, strong use cases in the future include supply and demand forecasting, which is, for understandable reasons, customer identity profiling. So Customer 360 is obviously the holy grail in retail. We hear about it a lot.

And the other one is quite interesting is a strong future use case, while I find it interesting, is customer service case classification. So this is whether your case is a really important, medium important, low importance, and retail has been really slow on the uptake here compared to something like financial services, which have been using machine learning for this kind of classification problem for years. And I think this is, again, indicative of though the -- perhaps, let's just say, the lack of focus on aftersales care among traditional, what we used to call, brick-and-mortar retailers versus the digital natives, which are much more focused on that kind of care. And so I think that's interesting.

Eric Hanselman

Yes, that's one of those places that's near and dear to Sheryl Kingstone's heart, and that was actually one of the things that Sheryl had called out specifically, which was that post-sale care of how do you ensure that a customer doesn't get handed from person to person to person to person or a website to website or person to website so that, in fact, you can actually manage that post-sale interaction seamlessly, which, again, there are a lot of different constituents. If you have to bop somebody over to return something to a different team, yes, that's a potential hiccup in that relationship with the customer.

Nick Patience

Totally. Yes. And i know She's got -- she has data about most things up that she says and so here's some more for her arsenal.

Eric Hanselman

Yes. That's fuel for the fire.

Nick Patience

Indeed. Indeed. Just one more industry, I think, is an interesting one. Manufacturing has been a pretty early adopter of machine learning, one we've tracked very closely. And I know a lot of technology vendors, both small and now increasingly extremely large, are very interested in it. And quality assurance is by far the primary use case. That's primarily about defect prevention rather than defect detection, which is to do with quality control.

And so it's -- we think it's just a really interesting example of innovative use of machine learning because obviously, it's much better to spot these problems before they happen than it is to kind of remediate them once they've already happened. And so we saw 43% of manufacturing respondents choosing it as a use case now, I think it's something like 7 percentage points clear of the next one. And then a further 18% choosing it as a primary use case in the future. And we saw a much more -- it's obviously much -- QA is much more important in high-precision manufacturing, where slips in quality standards can be much more expensive or more dangerous. And then we look to those manufacturers, we cut them by various issues.

And one of the issues we cut all of our surveys buyers do have machine learning and production versus do you have it in the POC stage versus do you have it neither and you have it, you're planning in the next 12 months. Those with manufacturers and production, 53% chose that as a primary use case. So already -- that indicates is actually becoming pretty widespread. So if the average is 43%, but 53% of those in production, it means it's pretty much is out there and it's really filtering through.

Eric Hanselman

I'm curious, does quality assurance tie into predictive maintenance because predictive maintenance is another one of those ML applications in which monitoring machine performance, vibration, those sorts of things? Do they interact? Or do you see those as distinct?

Nick Patience

We ask them as distinct and they came actually 1 and 2. So the one that was immediately below it, but 7 percentage points was predictive maintenance or condition-based maintenance and things like that. So it's -- I think they are related because obviously, if you have a -- you can prevent the problems happening, prevent defects, but also there's always some maintenance needs doing using machine learning models to predict when these things have to happen is not even 2 sides of the same coin. It could be the same side of a slightly larger coin really.

Eric Hanselman

Yes. So it's a piece of -- I mean the predictive maintenance is, I want to keep it from breaking and the quality assurance presumably is I want to keep the entire process with intolerant. So maybe I'm looking at oven temperatures or injection pressures or other aspects of the manufacturing process that may not be about the break but are going slightly a skew. And you'd figure out that, "Hey, if you lose focus on the tolerances, then your quality is going to suffer."

Nick Patience

Yes, you said you're right. QA is part of the larger the entire process versus focusing on specific machines. And one of the ways is -- machine learning enables us to happen is through a computer vision. As I like to say, sometimes, if all machine learning or AI, in general, gave us was the ability for computers to see then that is revolutionary in itself. And manufacturing has obviously been such an adopter of this. And by using cameras everywhere, organizations can do all sorts of things with QA and with predictive maintenance.

And actually another one of our strong future use cases is employee safety, including social distancing. And obviously, that became -- there was a -- the social distancing was a really large aspect, but it also opens people's eyes to the possibility of spotting using cameras and automating that process to make sure that people are wearing the right equipment, they're not straying into areas they shouldn't strain because they're too dangerous, all those kind of aspects. And if you can at least semi automate that part of the process, you can save people from having to stare at screens all day to try and to figure out if something is going to go wrong.

Eric Hanselman

That's the thing that I find is fascinating. I'm such a geek in the industrial side. My expectation had been that, of course, everybody is going to be instrumenting with all these sensors and they're going to do specialized accelerometers and temperature monitors and all this other stuff, but the #1 case is stick a camera on it and then there's so much we can do. Once you've got a camera on it, you can do so much in terms of understanding what's actually going on, whether or not it's the process, it's the people or all these other pieces. That's the thing that just a real mind shift for me.

Nick Patience

Yes. It totally -- it enables you to do preventative things. It also enable you do optimization things in terms of the process. And we see so many vendors, some of which started off with ambitions, let's say, to be kind of general purpose platforms, realizing that manufacturing is a really hot area and really narrowing down zeroing in on the manufacturing opportunity. And a lot of it is around that, how do we optimize the process by being able to have cameras looking at everything that's happening and figuring out ways of making this more efficient and safer.

Eric Hanselman

Well, it is such a rich data source. Again, just sort of -- that sort of mind blowing thing for me. It's like, "All right." And it sounds like what happens is you wind up getting these iterative processes in which you do it for this one specific thing and you say, "Oh, well, I've actually got this data and we could also use it for this. And in fact, if we keep an eye on this aspect, to your point, the human factors, the safety issues, yes, all sorts of wild stuff."

Nick Patience

Yes, totally.

Eric Hanselman

Thinking about what the impacts of all of this data is, what should enterprises be considering for the year ahead for their AI and ML projects?

Nick Patience

Well, as I mentioned earlier, this is -- we don't pretend that AI machine learning is easy for organizations to develop and deploy themselves, but a large number of them are doing it. And it may surprise people quite how many organizations are choosing the build route for at least part of what they're doing versus just relying on applications with embedded machine learning. But it is hard to scale. So it's 1 thing having 2 or 3 models in production and being able to monitor them and manage them once you got 10,000 of those or 100,000 or millions of the models then you can see where the problem quickly comes.

And I think as we said before on this podcast, it's almost organic. So it's not that -- this is not like rules-based systems beforehand, which will carry on doing exactly the same thing forever and ever and ever until you change the rules. As new data flows through the model and the predictions change, the model often needs to be retrained. This is -- it's almost -- it's not quite a live, but it's pretty close to it. And so it's quite a different beast and organizations will find that challenging. And this is where -- but help is at hand as it were. So both the questions and the reports we're doing from the back of the survey and the panel we're doing at 451 Nexus is focused on ML Ops and how ML Ops can come to the rescue essentially and enable organizations to build, deploy, manage and monitor and retrain large volumes of models and do so at scale.

And I think ML Ops itself will evolve as a practice. And I think it's interesting to think as a technology analyst company, when we look at these, we look, obviously, as a company, we track thousands of vendors, large, medium, small and in very early, very early stages. And I think the question is right to ask, should I buy ML Ops tools from a specialist ML Ops vendor? Or should I just rely on the big vendors to take care of everything? And as I alluded earlier, the large vendors are quite keen on getting the message across the ML Ops essentially the entire process, not just this post-deployment stuff.

Yes, if you're a large organization, you're a bank or an energy company and you're going to deploy thousands and thousands of models, chances are you're not going to deploy them all in 1 data center, all in 1 cloud or anything like that, whether it be edge or data centers or anything between. You're going to be deploying them in various places for various reasons of optimization, but also just more mundane things like M&A has happened and you bought a company that uses different systems.

And so I think it's interesting to think about is there a market in 3 to 5 years' time for stand-alone ML Ops vendors? And I think there is. And the reason is because I think you're going to want some kind of neutral approach to monitoring and retraining and managing your models regardless of where they are rather than relying on cloud Vendor A and its ML Ops tools to manage all the other models in all the other places. It's a bit like the application performance management market that sprung up maybe 10, 15 years ago and there were standalone vendors that survived out of that for a similar reason. And I think because of the nature of machine learning models and the fact they need this constant monitoring and retraining, it does provide, I think, an opportunity for standalone ML Ops vendors.

Eric Hanselman

Well, I mean we've got hybrid infrastructure and that hybrid infrastructure is supporting these ML and AI environments. So you're going to need the ability to be able to exist across them. And hey, maybe some specialized applications may be able to go all in on one. But as you pointed out, you're only 1 inorganic event away, 1 merger, acquisition or whatever, from all of a sudden, creating a hybrid environment. So presumably, we need to have the same sort of infrastructure spanning tools across our ML environments that we do for any of the other infrastructure management capabilities we've got.

Nick Patience

Yes. A couple of other things I'd just like to point out, we do have in the survey. I'm not going to go into details here, but we do ask people about return on investment, what they expect to get, when they expect to get it, and we have lots of interesting data for people to dig into on that. Also on explainability and bias, which are related to -- very closely related to ML Ops. In fact, it probably could be argued they're part of the ML Ops process. And we also ask people various questions around their attitudes to various technologies that are driven or at least driven partly by AI, such as facial recognition, touchless supermarkets and all those kinds of things.

And then finally, it wouldn't be an S&P Global survey without ESG. And so we have asked some questions about the environment and the impact of executing, building, training ML models, other things such as tasks being automated or -- and also government regulation of AI initiatives. And so there's lots more in there for people to dig into more or less regardless of what your interest level is in AI and machine learning.

Eric Hanselman

Well, and the explainability and bias identification is a key part of the ESG perspectives more broadly. That was, again, one of the things that Giorgio and Kieran had pointed out. Especially in regulated environments, you've got to be able to explain to a regulator how did you get there? other minor issues.

Nick Patience

Yes. Yes. Exactly.

Eric Hanselman

Well, this has been great, Nick. Thank you very much. Clearly, there's a lot more that audience can check out in the results of the study itself. But thanks for all the perspectives.

Nick Patience

Thank you. Thanks, Eric, and thanks very much for having me.

Eric Hanselman

Well, great to have you back. And that is it for this episode of Next in Tech. Thanks to our audience for staying with us, and thanks to our production team that includes Caroline Wright, Caterina Iacoviello, Ethan Zimman on the Marketing and Events teams and our studio lead, Kyle Cangialosi.

I hope that you will join us for our next episode where we'll be talking a connected cars, both some of the connectivity pieces, some of the infrastructure pieces, some of the security aspects. I hope you'll join us then because there is always something Next in Tech.