There is no exaggeration in saying that gen AI catalyzed AI adoption in enterprises. Deloitte’s State of Generative AI survey found that business leaders are still excited about generative AI. On the other hand, there was also a dip in the numbers. Companies are pouring investments in generative AI, but most initiatives are going nowhere due to challenges in scaling the gen AI solutions. As the initial excitement fades away and businesses start to see the realities, there are five key points you as a CIO should know.
Five points for CIOs to scale gen AI solutions
Set priorities right for pilots
Gartner’s research found that more than half of organizations were either piloting generative AI or already had in production.
On the other hand, a recent McKinsey’s survey found that only 15% of companies experienced meaningful impact of generative AI. By meaningful impact, they meant that at least five percent of their EBIT could be attributed to generative AI usage.
Considering both stats together, we can infer that most pilots are not scaling into production. Why is it happening?
Generative AI has widespread applications across every industry. That said, generative AI is not the solution for every business problem. For some use cases, there could be better technologies. For some other use cases, gen AI could be the best choice but not matured enough yet to implement at scale.
CIOs should work closely with their business colleagues to determine the problem-solution fit and plan their pilots accordingly.
The problem is that businesses are getting excited about the hype around generative AI and spreading their resources and focus thinly across dozens of pilots. Resulting observations are a mix bag of signals and noise, impairing your ability to decide.
To identify the problem-solution fit, CIOs and businesses should collaborate to understand the business impact of generative AI – value creation, business readiness, and change management, and the technical feasibility – infrastructure readiness, data readiness, scalability.
Quality data is the key
As far as generative AI is concerned, the fundamental truth remains same. Garbage in, garbage out. Most CIOs assume that gen AI picks up relevant data from all the sources and generates tailored responses using the data.
Truth is, high-quality data is the bedrock for high-quality enterprise generative AI applications. That’s the catch for almost three-fourths of businesses.
Creating clean data requires careful effort. For example, finance institutions can leverage generative AI in fraud detection. However, to create effective custom generative AI models, finance companies should ensure the accuracy, consistency, and completeness of the customer transaction data. If you build or fine-tune gen AI models on poor-quality data sets, you will experience false alarms and inaccurate risk modeling.
Perfect data is an unattainable ideal. What you need is the right data to control AI hallucinations and generate accurate responses. As you create fresh sets of data every day, your generative AI models should generate responses based on the latest data. CIOs should invest in building a robust data infrastructure with established quality standards.
Control proliferation with reusability
Business users can be hasty to build tailored generative AI solutions for their specific use cases. But the proliferation would also bring some critical challenges. Data privacy and security would be at stake. Not to mention the misuse of gen AI models and high infrastructure costs. Building a gen AI solution from the scratch for each use cases would cost you a fortune.
As a CIO, you are the captain of the ship. You need to plan and build gen AI solutions that can serve multiple purposes, without sinking the opportunities to scale. To that end, you need to set up a cross-functional team to develop a platform with reusable components to build generative AI solutions to cater to many business needs.
For example, the NLP component in a generative AI solution can be used across several departments in an e-commerce business. The marketing team can use to generate product descriptions, the sales team can write personalized emails, and the customer service department can leverage the component to enhance customer experience with faster, more accurate, and personalized responses.
A McKinsey study found that reusable components can accelerate gen AI solutions by at least 30 percent. You need to review all the potential use cases for generative AI in your business and identify use cases that require common assets. Based on this analysis, you can assign resources to build the common assets.
Tune in to experts from Microsoft and Saxon AI in this insightful discussion.
Choosing the provider of the tools and infrastructure required to build the gen AI solutions comes as a common roadblocker. There are some aspects that do not require much analysis of decisions. For example, large language models are continuously evolving. Before you have another round of discussion on choosing an LLM provider, you will have another provider with more advanced model in the race.
In other cases, you wouldn’t have many choices. If you already have a cloud service provider (CSP) who understands your business needs better, you can go with the same CSP’s gen AI offering. Major cloud providers, such as Microsoft and Amazon, rolled out their own gen AI services to help you build viable gen AI solutions to maximize business value. The scale of the results totally depends on the maturity of your digital infrastructure and how well you use the services.
Holistic approach, not piecemeal
Generative AI applications include a variety of components such as LLMs, APIs, etc. In most cases, business leaders extensively discuss each of the individual components. But what’s challenging is bringing all the components together make a workable gen AI solution. This integration is indeed the major road blocker to scaling gen AI solutions for more organizations.
The tech stack for a generative AI solution is a complex set of integrations. For example, you need to manage different applications and databases across different cloud platforms and on-premises systems. You should also handle other components such as latency, resilience, and governance protocols. Any additional component to the solution would make each component in the mix more complex.
The key here is end-to-end automation. Successful gen AI initiatives implemented end-to-end automation in their orchestration process, from data cleaning to pipeline construction and model monitoring. Leveraging a modern MLOps (Machine learning operations) platform would help you automate changes to the core gen AI infrastructure with more speed and safety.
Rein in costs before they blow out
CIOs should look at gen AI costs beyond the obvious and understand the nuances. Model costs are obvious and seems to be the major expense. In reality, they only cost about 15 percent of the total cost. Generative AI landscape is complex and interwoven with underlying elements such as model training, fine tuning, governance, etc.
Let us take the example of fine tuning. OpenAI calculates fine-tuning costs by multiplying the base cost per 1000 tokens by the total number of tokens in the input file, and then multiplying the result by the number of epochs trained. Suppose you want to fine-tune a GPT-3.5 Turbo model over 3 epochs. Fine-tuning this model with a 100K token file would cost around three dollars.
If you ignore the nuances, your generative AI costs would go out of control in no time and result in negative ROI. If that’s the case, you might have to abort the gen AI initiatives, much less scaling them.
Change management is another important aspect you should consider. Typically, businesses are spending about $1 for change management for every $1 spent on development. Right now, businesses are spending $3 for change management for every $1 spent on development. However, there is a good scope for you to bring down the ratio. You need to train your people and build a performance management infrastructure.
Generative AI is not a one-time investment. In fact, running generative AI applications is more expensive than building the applications. This is due to the labor costs for maintaining data pipeline and the model. Risk and compliance management is another key driver of the run costs.
However, you can drive down the model costs using cost-reduction capabilities and tools. This is an on-going process but when done right, it can significantly reduce the costs for a query.
Generative AI is here to stay. Businesses across industries have understood the potential of generative AI. As gen AI evolves, pulling the plug when you hit blockers is no longer an option for you. As a CIO, you should be ready to pivot and scale generative AI solutions to drive your business forward.
