Visual linguist Neil Cohn: ‘Comics might seem simple, but there is so much more complexity to it’

What defines a language? As a scientist who draws on both the cognitive field and linguistics, visual linguist Neil Cohn is particularly interested in answering this question. Combining this with his deep fascination for drawing and comics, Cohn claims the structure of sequentially drawn images is similar to that of spoken and sign language. In doing so, he introduces a third type of language called visual language.

Already at a young age, Neil Cohn entered the world of comics by selling his self-made comics. Since then, his captivation with comics and drawing has never ceased. Besides creating the illustrations of his own books, he occasionally creates and publishes comics himself. Yet, Cohn does not stick to a purely artistic approach. From the moment he embarked on his studies in linguistics, he also began to look at his hobby from a more scientific angle. It was then that he discovered certain patterns in comics that were identical to those in language. This led him to devise his own theory of language.

Neil, what is language according to you?

“Language is made up of a vocabulary and a grammar. Together, these form a system of patterns that is stored in people’s heads. If the patterns in my head overlap sufficiently with the stored patterns in your head, we share a language. At this point we are able to express the ideas in our minds and communicate with each other. During this interview for example, the set of patterns we share is English. Thus, language is all about the degree of shareability.”

What types of language are there?

“We can express ourselves through three forms of language, namely spoken, sign and visual language. Spoken languages are patterns based on sequential sounds coming from our mouth, whereas sign languages originate from the sequential movements of our bodies. We also use our bodies to express meanings that don’t rise to full languages, like in gestures or “body language”.

“Spoken language has many thousand varieties of which English and Dutch are some. Sign language too has a lot of variations like for example American Sign Language and Dutch Sign Language. As with spoken languages, people won’t be able to understand each other if they both use a different sign language.

‘In its essence, writing is drawing sounds’

“I believe sequential images that are drawn can also be classified as a language, because they rely on the same sort of structure as spoken and sign language. This is what I call visual language. Just like we speak in a spoken language like Chinese, we draw in a visual language.

“Visual language is predominantly used in comics and each comic is drawn in its own visual language. Instances of visual language are the style used in Japanese manga or the American superhero style. Europe too has its own characteristic genre with famous comics as Suske & Wiske and Tintin.”

What about writing?

“As opposed to writing, these languages are intrinsically natural to us. They are part of our cognitive system and as a species we have used these capacities from the very beginning. People, for example, have always drawn. You can see this in cave drawings, which decorate the walls with symbols and illustrations.

“Writing, however, is completely unnatural. In some cultures written language is not present, whereas in others it had to be invented. Writing forms a bridge between the spoken and visual realm. It is the conversion of the spoken way of expressing ourselves into the graphic or virtual way of expressing ourselves. One could say that, in its essence, writing is drawing sounds.”

If the ability to use spoken, sign and visual language is natural to humans, does that also mean we don’t have to learn them?

“No, all languages require an acquisition process in which people – either by passive imitation or by active instruction – get familiar with the structure of a language. They have to learn its vocabulary and grammar in order to make full use of it.

“Still, one is able to use basic components of a language without practice. Think about hand gestures; we all know some without having had to actively practice them. Yet this does not mean that we know sign language.

“The same goes for visual language; if you want to acquire the ability to draw like Japanese manga you have to put some serious time into it. Otherwise, you will not get beyond the basic figures of drawing, such as a simple smiley face, which is the visual equivalent of a gesture.”

Are you saying that drawing is more a matter of practice than of talent?

“Absolutely, everyone can learn to draw. Like any other language, we just have to acquire its vocabulary. We do this by copying others and by imitating our environment. The more visual vocabulary we know, the better we draw.

“Every language has its own vocabulary, which is always linked to meaning. In sign language, the vocabulary is made up of bodily postures and motions, whereas in spoken language it comprises strings of sounds. These signals become associated with different meanings. An open fist and a closed fist can each have different associations within sign language, just as the sound ‘bear’ in spoken language can mean either furry animal or naked.

‘The more visual vocabulary we know, the better we draw’

“In visual language, meaning links to graphics. As in the other languages, this is not always a one-to-one correspondence since an image can have multiple meanings. Consider the eggplant emoji, which has a literal meaning as well as a figurative one.”

But in order for multiple drawings to become a full visual language they need more than vocabulary, right?

“Indeed, this is where grammar – the other prominent element of language – comes into play. To classify as a language, the sequence of images needs to be governed by some sort of sequential structure. In graphics, this is a narrative structure, which functions as a grammar and puts all the images together in such a way that it packages meaning.

“Though all forms of language rely on the same underlying structure, the characteristics of this sequential structure are different for all of them. In spoken language, the grammar consists of nouns and verbs. In visual languages, pictures often convey more information than words, so they instead use a narrative structure.

“The narrative structure of visual language can most easily be found in comics. Here you can see how the sequential images constantly link and refer to each other, causing a fixed pattern of rules that operates as a grammar.

“Emojis, however, lack this form of grammar and complexity. Despite having an extensive vocabulary that indicates meaning, there is no overarching structure that guides the meaning across a sequence of emojis. There is no grammar. Therefore, they do not qualify as a visual language.”

A considerable part of your work involves cognitive neuroscience. What did you find about language and our brain while doing this particular research?

“The most important thing we found is that even though drawing, speaking and signing on the surface look very distinct from each other, under the hood they are all part of one and the same cognitive system.

“You can see this from the way they substitute for each other, as in the famous sentence ‘I heart Amsterdam’. They also work complementary to each other, which is one of the main reasons why we use emoji’s next to written text. It demonstrates that they are integrated systems. If they weren’t we wouldn’t be able to swap them back and forth.

‘The meaning of an image differs per culture’

“Also, when we measure the brain activity of people while manipulating the narrative structure of comics, it elicits similar brain responses as when we would manipulate the structure of, for example, written sentences.

“Other research has found some overlap here with brain reactions induced by violating music. This suggests that language and music have something in common as well.”

In your latest book ‘Who Understands Comics?’ you pose the question of whether the ability to understand a sequence of images is universal. How would you answer this question?

“I would say that is a myth. It is innate to humans to make meaning out of graphics and to be aware that an image links to a meaning, but what this meaning is, differs per culture. In addition, children only begin to develop an ability to understand sequential images between age four and six. And even beyond this age there is no guarantee that images are identically perceived.

“Indeed, most people do understand a sequence of visuals in the same way, however, that is because they have already been exposed to the visual language of that culture. So, understanding a sequence of images requires being familiar with them, like in comics or picture books.”

_Advertentie.