The release of ChatGPT on November 30 marked a new milestone in the development of language processing technologies. It is in any case the first time that an AI system, directly resulting from research, has aroused such enthusiasm: numerous articles have appeared in the specialized but also general press. ChatGPT (or OpenAI, the company that develops ChatGPT, or both) are consistently trending on Twitter even today.

Why such an echo? Are ChatGPT’s capabilities a departure from previous technologies capable of generating text?

ChatGPT can chat

Clearly, ChatGPT has allowed a qualitative leap: it is able to answer, in an often relevant way, almost any query in natural language. GPT2, then GPT3 – the previous language models developed by OpenAI – were already very good at generating text from a few words or sentences given as a primer (what is called the “prompt”): we often spoke of ” stochastic parrots” , as proposed by Emily Bender , and many other critical researchers of these techniques. Indeed, one could say that these systems produced text from bits of sentences drawn from the immense corpora at their disposal… even if this image of the parrot must be qualified: the systems do not repeat extracts word for word, but produce in general an original text by taking up, paraphrasing, combining elements seen in various corpora.

ChatGPT takes up these characteristics, but the “chat” part (dialogue) adds a completely different dimension, and often seemingly stunning.

The system no longer produces just a paragraph from a sentence given as input. He is now able to answer both simple and complex questions correctly, he can provide elements of reasoning, speak different languages, analyze or produce computer code, among other things.

For example, if asked if two characters were able to meet, the system is able to determine the dates corresponding to their existence, compare these dates and deduce an answer . This is trivial for a human, but no system until now could answer this type of question so precisely without being highly specialized. Beyond the language, ChatGPT can also provide elements of mathematical reasoning (but it is often wrong ) and analyze computer code in particular.

ChatGPT, a closed system

Regarding the operation of the system, difficult to say more, because OpenAI, despite its name, develops closed systems. That is, the computer code (source code) used is not available, and ChatGPT-related research remains largely a trade secret for now – although, of course, other teams are working on similar systems , also based on deep learning. So we have an idea of ​​what is used by OpenAI.

Other companies have kept a more open model, for example Meta with the work carried out at FAIR , generally open and published in the main conferences of the field. But, more generally, we are increasingly seeing a closure of research. For example, while Apple has always had a private and top-secret development model , Deepmind had an open model and is arguably a little less so now that the company is under the control of Google .

The ChatGPT system itself could become commercial: OpenAI is funded by Microsoft , which could decide to close access to the system one day soon, if it is in its interest.

By testing ChatGPT, you contribute (for free) to improve it

In the meantime, OpenAI benefits from the enormous publicity that its tool brings to it, and also from all the user interactions with it. If a user reports that an answer is not good, or asks the tool to reformulate his answer taking into account an additional element, this is as much information as the system stores and can reuse to refine its answer the next time, on the request asked or on a similar request. By testing ChatGPT, we work for OpenAI for free  !

To come back to the system itself, the dialogue part is therefore what makes the strength and particularity of ChatGPT (compared to GPT3 for example). It is no longer a “simple” language model capable of generating text “by the kilometer”, but a real dialogue system. It has probably benefited from millions or billions of examples evaluated by humans , and the current phase – where hundreds of thousands of users “play” with the system almost for free – still allows it to be continuously improved, since all interactions are obviously recorded and used for this.

It is now quite simple to access corpora of billions of words to develop a “GPT” type language model, at least for languages ​​that are widespread on the Internet.

But the data that allowed the development of ChatGPT (dialogues, interactions with humans) are not public, and this is what gives OpenAI a significant advantage over the competition.

For example, Google has different data, but no doubt also usable for this type of system – especially since Google has been developing a knowledge graph for several years which could make it possible to generate answers with better reliability than ChatGPT. In particular, the analysis of query sequences from Google’s search engine could provide valuable information to guide interaction with the user… But, in the meantime, it’s OpenAI that has ChatGPT, and not Google: OpenAi has the advantage.

Beyond ChatGPT, what applications in the future?

In fact, even if it is possible to circumvent the limits of ChatGPT , the system refuses to explain how to create a bomb, to produce erotic tales or to give her feelings (ChatGPT frequently answers that she is a machine, that she has no feelings or personality ). OpenAI has obviously taken care of its communication. The company has also taken extreme care to “armor” the system, to allow it to thwart most of the pitfalls that can ruin this type of application in a few hours, as often happens for systems open to the general public.

We can cite for example Meta, which last November had to withdraw its system called “Galactica” two days after putting it online . Galactica had been trained in the scientific field and designed to offer services to researchers. It was first touted as being able to write scientific papers automatically from a prompt …before the company clarified that it was obviously just a writing aid. But this unfortunate communication strategy sparked a controversy that forced Meta to quickly unplug Galactica.

Conversely, ChatGPT is still online, and still arouses as much passion among a large audience. Questions remain, however: what impact will ChatGPT have? What applications will be derived from it? And what economic model is the OpenAI company (and Microsoft, its main investor) aiming for?

Author Bio: Thierry Poibeau is DR CNRS at Ecole Normale Supérieure (ENS) – PSL