2024 Predictions in AI and Natural Language Processing (NLP)

By on
Read more about co-authors Jeff Catlin and Paul Barba.

While we were right at the dawn of generative AI this time last year, we didn’t predict quite the profound impact and seismic shift it would create around the world with the introduction of ChatGPT. In our set of 2023 predictions, we did note the potential effect of LLMs, with research showing their ability to self-improve, and said, “We predict that while … this won’t drive us into a singularity moment, it will be the hot research topic of 2023 and by the end of the year will be a standard technique in all state-of-the-art, natural language processing results.” That certainly has borne itself out.

Looking at where things have come in the last year, we wanted to try our hand again at forecasting where we see the market heading in 2024 in AI and natural language processing (NLP), including how it relates to our focus on the customer experience (CX). 

Jeff Catlin, EVP of AI Products at InMoment:

ChatGPT Will No Longer Be the Prevailing Technology for the Enterprise by 2025

Like most first movers in technology, ChatGPT will become less and less relevant as the year progresses. Local LLMs like Llama2 (and whatever comes next) will become the engines of corporate AI. There are many reasons for this, but data security and the ability to influence the results by augmenting a local LLM with industry-specific content are likely to be the two that drive this change.

LLMs Will Be Integrated to Solve More Challenging Problems

Technologies like LangChain, which allow users to feed the results of one LLM into another LLM, will become much more important for corporate users than the next, all-knowing LLM. Imagine using an LLM that measures the anger of a caller in a call center (furious), and that anger is fed into a follow-on model that combines the anger with the fundamental issue being addressed in the call to predict the likelihood of that caller canceling their service, or buying a competing product. Combinational AI is the next big step for corporate AI, be it in customer support, buyer purchase behavior, or any other fundamental business problem.

NLP Will Become More Relevant as LLMs Lead to Surge in Unstructured Data Volumes

LLMs are a trigger that encourages companies to utilize all of the unstructured data that they typically ignore because it’s hard to work with. LLMs are a gateway to this content, but powerful NLP that can tear apart unstructured and semi-structured content by speakers, regions, or problem areas will bring the diagnostic abilities of LLMs to the next level.

Paul Barba, Chief Scientist at InMoment:

OpenAI Drama Will Continue to Fill 2024

The ousting and rehiring of Sam Altman to OpenAI created news cycles jam-packed with gossip and hot takes, and I suspect OpenAI stories will continue to fill headlines all next year. The underlying catalysts – the unique non-profit/for-profit hybrid structure, the massive costs, the risks and promises of AI –  haven’t changed, and with the speed this field has been advancing, there’s ample opportunity for these forces to come to a head again and again next year.

The First AI Export Controls Are Most Likely Not the Last

The U.S. government has already placed export controls on selling China the advanced chips used to power AI research. Paired with the regulatory controversy around open-source models that bring advanced AI tools to everyone, I think we will see a reprisal of the software encryption export control fights of the ’80s and ’90s, when foundational web technologies like public key encryption were classified as “munitions” and forbidden for general export.

AI Marketplaces Will Take Off

Tech companies all seemed to have their “model marketplaces” in the machine learning era where enterprising individuals could put a trained model up for rent, and businesses could just pick and choose needed functionality. This never took off, as models were too inflexible, and the effort to evaluate choices was too high. LLMs promise easier integration and advances in AI make it feasible for constructing a solution out of many pre-built blocks to be largely automated.

As we see it, the gradual decline of ChatGPT as the predominant technology for enterprises by 2025 underscores the dynamic nature of the field, where localized language models (LLMs) like Llama2 will rise in importance. The integration of LLMs to address complex issues, facilitated by technologies like LangChain, signals a shift towards combinational AI. Moreover, the surge in unstructured data volumes, driven by LLMs, accentuates the growing relevance of NLP in enhancing diagnostic capabilities. Amidst these technological advancements, the ongoing drama at OpenAI and the emergence of AI export controls suggest a complex regulatory landscape and potential geopolitical challenges. On a positive note, the rise of AI marketplaces, fueled by more flexible LLMs, promises a transformative era where businesses can seamlessly integrate pre-built AI blocks to address diverse needs. As we look ahead, the AI landscape appears dynamic, marked by technological innovation, regulatory considerations, and the continual evolution of market dynamics.