OpenAI's GPT-3: Yet Another Big Leap For NLP
With 175 Billion Machine Learning Parameters, OpenAI's newest language model is ten times as powerful as the last one.
We all know the smart keyboard on our phones can be very handy, save for some “Oh No, Autocorrect!” moments. However, it’s only getting smarter and more refined. NLP or Natural Language Processing is the secret behind all of this magic: speech recognition, language translation, auto-subtitles, predictive text and autocorrect. OpenAI, which includes Elon Musk as one of its founders, has pushed the boundaries of NLP even further. An autoregressive (i.e output depending linearly on its previous inputs) language model, Generative Pre-trained Transformer 3 (GPT-3) uses deep learning to produce human-like text. With 175 billion machine learning parameters, GPT-3 incorporates ten times more parameters than Microsoft's Turing NLG, which was just four months old at the time of GPT3’s release.
Prevention of malicious use was heavily emphasized while releasing the last model (GPT-2) and this one too. The developers chose to not release the source code of the model (as was the case with the previous GPTs) but released it in an API or Application Programming Interface form. This will make it easier for companies using the technology to just include the pre-trained model without any user-side training (“zero-shot”), or quickly train it with a few application-specific examples (“one-shot training”) and deploy it. Many tech giants like Microsoft and Reddit among other companies are already using and evaluating the technology. Individual users too can sign up to evaluate, use and suggest changes. If you already have an OpenAI API key, you can try the model here.
As the model is still in its infancy it suffers from various biases that could be potentially harmful and were cited as a reason to keep the code closed-sourced. Facebook’s AI Lab head Jerome Pesenti, has stated that his study of the model indicated racial, religious, and gender biases. But if anything, it represents a step forward in designing algorithms that become more inclusive, more aware, more sensitive and more human, as they mature and form a bridge to the future where machine-understanding humans, make machines understand humanity.
About the Author: Amogh Raut is a Third Year Electronics and Telecommunications Engineering student at RAIT.