Mennatallah El-Assady was seven years old when she began learning the flute. At first, she found it hard to read all the black dots dancing about on the sheet of music. Her solution was to mark each individual note with a different colour. “That was a real breakthrough: suddenly I could read music,” she recalls. Only later would she discover that she had a form of dyslexia.
Today, aged 31, she develops smart software. Her speciality is interactive information visualisation, a skill she has brought to the AI Center of ETH Zurich, where she was recently appointed a Pioneer Fellow. Her graphics help picture the mathematical data behind AI applications. This enables people without any knowledge of programming to understand how algorithms work, and it lets them influence the decisions those algorithms take.
It’s been a while since El-Assady played the flute on a regular basis, but there’s something about the ability – or not – to read music that still intrigues her. Recently, she co-developed a web app that uses colour, as well as musical notation, to represent a melody. This can display any piece of music whatsoever, using her colour-augmented notation. In addition, the app boasts intuitive features that help users find their way around all the intricacies of a particular composition.
This can involve using the mouse to draw a sequence of high and low notes in a box. Search algorithms then hunt down the corresponding melody and highlight it in colour. “These are visual metaphors that help people interpret a song even if they can’t read music,” El-Assady explains.
An early fascination for all things AI
The music app is just one of many projects she was involved in during her doctoral studies – half of which she completed at the University of Konstanz and half at Ontario Tech. “It was really just a side project, something I developed more for fun at first,” she says. The idea came about, she adds, because of the many similarities between music and language.
El-Assady has been working in the field of computer-based language analysis for almost nine years now. As a student, she was fascinated by the idea of AI-based human support – in the form of search engines, voice assistants and chatbots. It wasn’t long, however, before she realised that human input is also vital – especially when algorithms are being trained to perform a specific task in everyday life.
By way of example, El-Assady cites a language model designed specifically to identify key terms and topics in a text database. Such an AI program could be useful for political scientists, she explains, because it would help them analyse long transcripts of political debates more quickly and easily. The trouble is that there are numerous ways of defining the category for each topic.
An interactive program requesting human input
“Linguistic models don’t look at a text in the same way a person would,” El-Assady explains. Rather, they combine text blocks that frequently occur together and assign them to a common category. “By contrast, a human reader also takes into account the wider context and brings along a broad range of general knowledge.”
And that’s precisely why El-Assady is so interested in the collaboration between human and artificial intelligence: because they are different, and because each person brings along their own perspective. “Together, humans and computers perform better than they would on their own.” That said, the problem with conventional AI programs is that it’s complicated to tailor algorithms so that they accommodate individual preferences. As a rule, this is a job for the machine learning developer.
El-Assady has long been fired by the idea of being able to overcome this barrier, empowering humans. Using AI-based text analysis, she has developed a series of interactive visual interfaces to refine linguistic models. These depict in real time how an algorithm assigns text blocks to new categories during the learning process.
If the AI model is unsure about which category to select, it provides the user with a range of options and asks them to help decide. The more feedback the model gets from a particular person, the more it learns about their preferences and adjusts the options accordingly.
Big interest prompted by analysis of US presidential debate
But, as El-Assady explains, there’s more to this than simply tailoring AI to individuals: “This approach will also enables us to develop computer models based on input from people with different backgrounds.” And that’s especially vital in language analysis, she says, because you need as many different perspectives as possible in order to root out bias and discrimination.
Another of her visualisation tools offers a deep dive into political debates. This uses coloured animated bubbles to illustrate how much attention certain topics receive and which person is leading the conversation. El-Assady’s interactive visualization also shows the patterns of argumentation followed by the speakers.
