Riding the Wave: 670 Days of Generative AI

OpenAI released its first version of ChatGPT on November 22, 2022, setting the Generative AI (GenAI) movement into action. In the approximately 670 days since, there’s been continued innovation from OpenAI, including the first GPT-3.5 model release via API in March 2023 allowing enterprises to bring GenAI to their own applications. During this time, other major technology players such as Anthropic, Google, and Meta have also made their mark. The response to these advances has run the gamut — spanning from hype to heavy speculation, to investment, government hearings, creation of AI boards and the appointment of company AI leaders, and more.

An often-heard comparison of the GenAI revolution is the advent of the Internet. As a reference point, the first web page went live in August 1991, but it took five more years — or ~1800 days —before Google launched. Today Google is the most popular and used website that supports billions of daily visitors.

So where do we stand today? Over the past 18 months, we’ve held hundreds of engagements with AI experts and practitioners, financial and corporate firms of varying sizes, along with start-ups looking to capitalize on momentum. Across these meetings, the sentiment is that the financial industry is beyond the peak of the hype and now facing the realities, along with associated challenges, as we march towards tangible and measurement benefit. Common themes have emerged through these engagements that have shaped the way S&P Global Capital IQ Solutions is approaching its GenAI strategy. These key trends include:

Data Foundations: As the saying goes, ‘garbage in, garbage out’ and that rings true with GenAI. Whether training a model or using Retrieval Augmented Generation (RAG), an AI retrieval method that pulls data for model context used to improve the quality of output, inaccurate data will only power the model to provide inaccurate answers. Simply understanding and organizing an enterprise’s data estate continues to be a challenge for many firms.

A critical element to the data organization, or cataloging, is understanding the data format (e.g., unstructured vs. structured) and associated metadata. Both are critical in successful utilization within GenAI models — unstructured data (e.g., textual based data such as earnings call transcripts), when combined with proper metadata tagging, is prime for Large Language Model (LLM) usage. On the other hand, structured data (e.g., financial data structured in columns and rows) can pose more challenges and require additional shaping and contextualization work to garner the best results from LLMs. Finally, understanding data rights when utilizing licensed data is hampering legal departments and slowing the pace of implementation across the industry. Digital transformation and the rise of cloud adoption has aided in these challenges, however, many organizations, especially larger firms, are still in a hybrid state of on-premise and cloud.

Accuracy: The financial industry requires accuracy and precision as everyday decisions are made that move markets, manage risk, impact supply chains, and drive global economies. Concerns of content created from GenAI solutions, and the associated regulatory requirements, has the industry on high alert.

To combat inaccuracies, or hallucinations, it’s important to spend time on model selection, and in understanding data sources along with guardrails and finetuning methods. In GenAI applications knowing what data sources are powering the model can provide confidence — particularly if quality data is at the foundation. It is now becoming an expectation that traceability, or sourcing, is provided within GenAI tools. Model selection can play a key role in accuracy as certain models are better at specific tasks, such as mathematical reasoning or visualization creation, so taking a multi-model approach can help to provide more accurate results pending the use case. Finetuning and guardrails are necessary in training the model on what and how it operates and creates content — having appropriate subject matter expertise is critical in this process.

From Proof of Concept to Business Value: Approximately 90%¹ of GenAI proof of concepts (POCs) have not, or will not, move to production. This low conversion rate is driven by a few core factors including non- or mis- defined use cases, talent and/or technology gaps, lack of funding and caution of return on investment, and lackluster results from POCs. The excitement and rate at which the technology is evolving has many building solutions in search of a problem, instead of first starting with find a problem that needs to be solved.

Firms with higher success rates first start by pressure testing their use cases by mapping out their workflow(s), determining major pain points, and then evaluating if GenAI is the proper technology to solve the problem. Following use case identification, ensuring connectivity between machine learning engineers or data scientists, developers, subject matter experts, and end users is critical in ensuring solutions are being built towards production value. Building these cross functional teams pairs machine learning talent with subject matter experts to ensure appropriate model finetuning, prompt engineering, and guardrails are instilled throughout the product development process.

Talent, Technology & Costs: GenAI has heightened the war for machine learning and data science talent. At the same time, organizations are appointing AI heads — a June 2024 Gartner poll surveying 1,800 executive leaders found that 54% of organizations have appointed a head of AI². This conflux, coupled with the change management, education, and training necessary to succeed with GenAI, is posing challenges for organizations.

Slower economic growth and uncertainty, pending elections around the world, increased regulation, and industry headwinds, have resulted in tightened budgets for organizations. These factors, combined with AI talent wars, soaring cloud costs associated with GenAI development, the necessary rethinking of technology stacks, and the fear of falling behind competitors, creates competing realities for organizations regarding funding. Additionally, many firms are balancing how they bring GenAI into their organizations and drive value when their revenue generating, core competencies is not AI based. This is resulting in many organizations evaluating “build vs. buy” and seeking trusted partners to support them on their GenAI journey. Finally, as investment is provided, executive teams are seeking a return on the investment. As many current GenAI use cases revolve around operational efficiency, but still require a human-in-the-loop, it can pose challenging in measuring positive financial impact.

While at a recent financial industry conference, a statement was made that “the (GenAI) models we are using today will be the worst we ever use”. A grounding statement when considering the focus across the industry, investment being poured into the space, and expectations of the solutions. Ironically, as I fact checked data for this blog, I first asked ChatGPT and then used traditional Google searching for additional sources to confirm accuracy. What does that say about where we are on the GenAI wave? It tells me we have a lot more to ride.

¹ Source: Forbes, Reasons Why Generative AI Pilots Fail To Move Into Production, January 8, 2024.

² Source: Gartner, Inc., Gartner Poll Finds 55% of Organizations Have an AI Board, June 26, 2024.