The impact of a new ‘writing robot’: ‘I had ChatGPT take an exam. It got a nine!’

ChatGPT has set a lot of hearts racing. The new “writing robot,” which writes a poem as easily as a content article, exceeds all expectations. What can we expect from this system? Univers asked Chris van der Lee, who recently received a PhD in computational linguistics and Professor Emiel Krahmer of the Department of Cognition and Communication at the Tilburg School of Humanities and Digital Sciences.

The media has been full of reports about ChatGPT (Generative Pre-trained Transformer). But what exactly is it?

Chris van der Lee: “It’s actually not a robot writing at all, it doesn’t have arms or a body. It’s a language model, a piece of software, which is trained on gigantic amounts of text (written by humans) based on so-called neural networks and can predict very well what the next word is, based on statistical calculations. That sounds fairly simple conceptually, but the potential applications could have a big impact. Similar to Google’s search engine, for example, because it uses all the knowledge on the Internet and in other databases.”

Emiel Krahmer: “I wouldn’t call it a chatbot because those are systems meant for conversation, usually conversations about a defined topic. ChatGPT is a self-learning system and can answer random questions. It’s basically a question-answer system, and the new thing about it is that it can answer a seemingly infinite number of questions.”

‘It may have the benefit of students turning in better-written theses’

Van der Lee: “It gives factual information. And the beauty of it is that not only can it cover countless topics, but it can cover them in different styles. For example, it seems to recognize sarcasm. And it can write poems as well as notices of objection.”

Despite all the jubilation, we must also remain critical, both researchers warn. Krahmer: “The system is capable of a great deal, but the answers do not always appear to be correct. It is not an oracle; it sometimes misses the mark.”

Van der Lee: “In doing so, it uses data from sophisticated language models, with the major languages having a statistical advantage. Languages like English and Spanish are much more widespread than Dutch, and so the information stored in those languages is more likely to be reflected in a response.”

Chris van der Lee, you recently successfully defended your thesis on the use of language models for automatic language generation. Can ChatGPT also write a thesis?

Van der Lee: “Ha-ha, no, that’s unlikely. The system assumes existing knowledge from existing databases. In PhD research, on the contrary, you want to acquire new knowledge by drawing up new hypotheses and collecting new data. This system cannot do that. It can summarize existing texts well, though, and that is very useful, for example, at the beginning of a PhD research, when you have to do a literature review and summarize it.”

Krahmer: “It’s a tool, just like the calculator and spell checker are—but of a very high and general level. And it can write introductory texts very well, which tend to be more general.”

What is the potential impact of ChatGPT in academia?

Krahmer: “In education, it will have a big impact. Students can have the system write an essay, for example. And I’ve already had it take an exam. ChatGPT got a nine! But we shouldn’t overestimate it either, because it makes mistakes quite often, so you can’t rely on it blindly. It’s just like Googling, with that we also tell students to keep paying attention.”

Van der Lee: “It may have the advantage of students submitting better-written theses because ChatGPT can rephrase pieces of text. The language and sentence structure of some students now is sometimes of a lousy level.”

But how can a lecturer tell if something was written by ChatGPT or by a student themselves?

Van der Lee: “That remains tricky.”

Can a lecturer use ChatGPT to check that?

Krahmer: “That could be the case. Only time will tell. We actually have no insight into the model. Maybe it could add a watermark in the text, so you can always see what the source of the text is.”

‘ChatGPT may just disappear behind a paywall’

Van der Lee: “There will be some form of control built in. Because the creators are a commercial party, and they don’t want to suffer reputational damage due to the system producing fake news on a large scale, for example.”

But it’s an open-source system, right?

Krahmer: “No, it is a commercial product that is free to use for now. I think it is also important to add that we have no insight into exactly how the model is put together and what exactly it is trained on. They do that very cleverly because if everyone discovered it and started using it, it could just disappear behind a paywall.”

I am a journalist. Will I soon lose my job because of ChatGPT?

Van der Lee: “You don’t have to be afraid of that. As long as you add depth to existing texts, you keep adding value. The system can’t match that yet.”

So ChatGPT could not do this interview?

Krahmer: “No, Google didn’t cost jobs either. The system will find new applications that will create new jobs.”

Van der Lee: “When the printing press was invented the monks just went and did something other than copy books, then they started brewing beer.”

Translated by Language Center, Riet Bettonviel

_Advertentie.