OpenAI's GPT-3: Yet Another Big Leap For NLP

Amogh Raut

With 175 Billion Machine Learning Parameters, OpenAI's newest language model is ten times as powerful as the last one.

We all know the smart keyboard on our phones can be very handy, save for some “Oh No, Autocorrect!” moments. However, it’s only getting smarter and more refined. NLP or Natural Language Processing is the secret behind all of this magic: speech recognition, language translation, auto-subtitles, predictive text and autocorrect. OpenAI, which includes Elon Musk as one of its founders, has pushed the boundaries of NLP even further. An autoregressive (i.e output depending linearly on its previous inputs) language model, Generative Pre-trained Transformer 3 (GPT-3) uses deep learning to produce human-like text. With 175 billion machine learning parameters, GPT-3 incorporates ten times more parameters than Microsoft's Turing NLG, which was just four months old at the time of GPT3’s release. 

Along with conventional language processing, the model can also generate, with minimal description, stories, poetry, MS Excel documents, letters, and even computer code. It demonstrated the generation of code for a TODO app in a test. It can also generate code in programming languages such as Python, JavaScript, and CSS among others. In another test, it convinced human evaluators that the articles and poems it generated were written by a real person. This is both great and scary as it is now a possibility that malicious content, especially fake news, could be spread with compelling levels of authenticity. 

NewsBytesExplainer: GPT-3, the powerful AI that generates stories, poems,  code | NewsBytes

Prevention of malicious use was heavily emphasized while releasing the last model (GPT-2) and this one too. The developers chose to not release the source code of the model (as was the case with the previous GPTs) but released it in an API or Application Programming Interface form. This will make it easier for companies using the technology to just include the pre-trained model without any user-side training (“zero-shot”), or quickly train it with a few application-specific examples (“one-shot training”) and deploy it. Many tech giants like Microsoft and Reddit among other companies are already using and evaluating the technology. Individual users too can sign up to evaluate, use and suggest changes. If you already have an OpenAI API key, you can try the model here.

As the model is still in its infancy it suffers from various biases that could be potentially harmful and were cited as a reason to keep the code closed-sourced. Facebook’s AI Lab head Jerome Pesenti, has stated that his study of the model indicated racial, religious, and gender biases. But if anything, it represents a step forward in designing algorithms that become more inclusive, more aware, more sensitive and more human, as they mature and form a bridge to the future where machine-understanding humans, make machines understand humanity. 


About the Author: Amogh Raut is a Third Year Electronics and Telecommunications Engineering student at RAIT.