Large Language Models: Helping Manage Data

Generative AI and large language models (LLMs) are the future, and promise a revolution in data management. However, development of LLMs is still very costly and inaccessible to smaller organisations. This will change as the years go by, and AI becomes more commonplace.

AI technology is changing the way the world does business. Generative artificial intelligence (generative AI) refers to the use of large language models (LLMs) to create new content, like text, images, music, audio, and videos.

LLMs are generative AI models that use deep learning techniques known as transformers. These models excel at natural language processing (NLP) tasks, including language translation, text classification, sentiment analysis, text generation, and question-answering. LLMs are trained with vast data sets from various sources, sometimes boasting hundreds of billions of parameters. They could fundamentally transform how we handle, interact with and master data.

Prominent examples of large language models include OpenAI’s GPT3, Google’s BERT, and XLNet, based on a whopping 175 billion parameters.

Industry adoption of large language models

Generative AI is primed to make an increasingly strong impact on enterprises over the next five years.

The generative AI-based LLMs market is poised for remarkable growth, with estimations pointing towards a staggering valuation of $188.62 billion by the year 2032. - Brainy Insights

The world’s total stock of usable text data is between 4.6 trillion and 17.2 trillion tokens. This includes all the world’s books, all scientific papers, all news articles, all of Wikipedia, all publicly available code, and much of the rest of the internet, filtered for quality (e.g., web pages, blogs, social media). Recent estimates place the total figure at 3.2 trillion tokens. One of today’s leading LLMs was trained on 1.4 trillion tokens. – Forbes