BY GABRIEL MATTYS
FEBRUARY 15, 2023
Now that GPT-3 has laid the foundation for advancements in natural language processing, other major players in the industry are also developing their own transformer-based models to create efficient and powerful chatbots. Take Meta, BigScience, EleutherAI, Google, and more: they have all released versions with up to 10 times more parameters than GPT-3 to unlock a deeper understanding of language tasks. Today, I want to grant you a list of natural language processing (NLP) models as alternatives to GPT-3. Also, we will discuss why we need to look for different options to OpenAI’s model.
The recent release of GPT-3 has generated a lot of excitement in the NLP community. This powerful black box language model has been lauded for its potential in automation and usability—but it is closed source and offers limited access to users.
Fortunately, open-source alternatives of similar capability are beginning to enter the market offering far greater transparency and accountability than their commercial counterparts. One huge advantage of using open-source options is the freedom to review the source code which can give users greater insights into processes as well as better control over data compared with proprietary solutions. In general, having alternatives to OpenAI’s GPT-3 will increase the pace of improvement in the industry.
So if you’re looking for advanced NLP capabilities without sacrificing transparency, open-source models may be the best choice for you. Below are some popular OpenAI GPT-3 competitors.
Bloom has taken the world of Large Language Models by storm in 2022 with 176B parameters. It’s a result of the collaboration of BigScience, Hugging Face, and hundreds of researchers and institutions from around the globe.
With 1.6TB of text training data at its core and access to industrial-grade computational resources – Bloom is an open-source alternative to GPT-3 freely available for research and enterprise purposes. What sets it apart though is its dedication to exploring lesser-known languages away from English, making it a truly inclusive model accessible even to those with native tongues that have been historically underrepresented in the digital space. For example, BLOOM was trained on a dataset of 46 natural languages and 13 programming languages.
Also, Bloom is focused specifically on the task of reading comprehension and is not as versatile as GPT-3. Therefore, it can be used in a variety of applications aimed at customer support, creating chatbots, or any educational platform. Due to its multilingualism, it can be used in language translation too.
GPT-J and GPT-NeoX are two language models created by EleutherAI, an independent research collective established in July 2020. While GPT-J is a six-billion-parameter model trained with the company’s 800-gigabyte “The Pile” language dataset and matches the performance of GPT-3’s Curie model, GPT-NeoX is more expansive, boasting 20 billion parameters. In addition to this variant, EleutherAI also released smaller GPT-Neo models with 1.3 billion and 2.7 billion parameters in March 2021. As when it comes to performance, GPT-NeoX has proved to outmatch its rivals – the Curie model of GPT-3 included – by a few notable percentage points according to EleutherAI’s benchmarks results. Also, GPT-J and GPT-NeoX are open-source Natural Language Processing models and can be tested for the model’s capabilities.
GPT-J and GPT-NeoX are the most popular open-source alternatives to GPT-3 today. But, GPT-3 is trained on data with more parameters than EleutherAI models. For example, GPT-NeoX performs better than OpenAI’s smallest versions Ada and Babbage. But, it still can’t outperform Davinci.
LaMDA (Language Models for Dialog Applications) represents the family of Transformer-based neural language models specialized for dialog-based conversations developed by Google. It has the same transformer-based architecture as OpenAI’s GPT-3 and its own BERT, but LaMDA is able to comprehend nuanced questions and conversations covering a variety of subjects. And, that’s why Google’s LaMDA has changed the game for natural language processing.
By utilizing up to 137 billion parameters and pre-training on 1.56 trillion words of public dialog data and web text, LaMDA showcases groundbreaking improvements in understanding conversational content. Since its May 2021 release, two generations of LaMDA have been released by Google, with the second iteration unveiled past May being more finely tuned than the originally-released version — now capable of providing users with valuable recommendations based on their queries. LaMDA2 was trained on Google’s Pathways Language Model (PaLM), which contains 540 billion parameters in its own right. OpenAI’s ChatGPT spurred the development of Bard, a conversational AI chatbot powered by LaMDA and its impressive capabilities — showcasing just how powerful Google’s language model is!
BERT (Bidirectional Encoder Representation from Transformers) is an open-source machine learning framework for various natural language processing tasks. Developed in 2018 by Google researchers, BERT was trained on a whopping 3.3 billion words from both Wikipedia and Google’s BooksCorpus, equipping it with an advanced ability to grasp the context of each word in a sentence. This ultra-smart system has found its way into many industries, particularly healthcare, and finance that requires precision when interpreting the text.
BERT is called bidirectional due to its ability to read the text in both directions at once. It is a remarkable feature because originally language models can read the input only from left-to-right or right-to-left at the same time. This bidirectional functionality was pre-trained on two methods: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). In the first case, BERT was needed to find the hidden word in a sentence considering the word’s context. In the second case, the program tries to predict whether two given sentences are logically related or simply random.
The power and sophistication of BERT is exemplified by its two popular architectures – BERT Base with 12 layers of transformer blocks, 12 attention heads, and 110 million parameters; and BERT Large: 24 layers of transformer blocks, 16 attention heads, and 340 million parameters. This technology is now being used at Google to optimize the interpretation of user search queries.
BERT and GPT-3 differ primarily in their architecture and versatility. With GPT-3, access to more data may be beneficial to specific tasks like summarization and translation, since it was trained on a larger dataset than BERT.
Megatron-Turing NLG (Natural Language Generation) is among the largest language models so far with an impressive 530B parameters. It was introduced in October 2021. It’s a result of a joint effort between Microsoft and NVIDIA and is built upon its two predecessors Turing NLG (17 billion parameters) and Megatron-LM (8 billion parameters). The training of the model was conducted using a Pile Dataset and the powerful NVIDIA DGX SuperPOD-based Selene supercomputer.
This language model can do various tasks related to natural languages such as completion prediction, reading comprehension, common sense reasoning, natural language inferences, and word sense disambiguation.
Also, on the website, there is an invitation for organizations that want to collaborate with NVIDIA. The company tempts to work on research and managing problems such as toxicity, biases, and responsible AI usage.
In May of 2022, Meta released Open Pretrained Transformer (OPT), a solid open-source GPT-3 alternative. This model contains 175B parameters and was trained on both The Pile and BookCorpus datasets. What makes OPT stand out from the rest is that it allows researchers to access the pre-trained models as well as the source code for using or training them. Hopefully, this will lead to a better understanding of the technology and ethics surrounding its use. Although OPT is only available for research purposes at this time, Meta intends to make it available to educational institutes, governmental authorities, civil services, and industry research labs under a non-commercial license.
The possibilities arising from artificial intelligence and natural processing are developing with unprecedented speed. Having more options than just Open AI’s GPT-3 platform is advantageous not just for its affordability, but for advancing the field as a whole. With open-source language generators, researchers, companies, and other organizations have access to plenty of resources for carrying out NLP tasks. Furthermore, introducing competition in the market has proven an immensely helpful way to improve different technologies; it pushes developers and organizations who use them to strive to be better. All signs point to the fact that if we have more alternatives than GPT-3, we will achieve tremendous breakthroughs much faster. This is why EleutherAI, META, BigScince, and Hugging Face are doing such important work by creating models and making them available so that all can use and benefit from them.
Exciting times for tech lovers!