Multimodal neural networks are advanced AI-models that can understand and integrate multiple types of information simultaneously, such as text, images and sound, and thereby create a more accurate representation of reality. This makes the technology highly interesting for military use, but its use also raises many ethical questions.

Developments in artificial intelligence (AI) are accelerating rapidly, and the concept of multimodal models, a form of deep neural networks, has become increasingly prominent in both commercial and military contexts. To provide an overview of the latest developments in the research, researchers at FOI issued the report Introduction to Multimodal Models.

The interest in modal neural network models stems from their ability to understand and interact with multiple types of information simultaneously. For example, such a network can understand that an image of a tank corresponds to the description of a tank in text, explains Edward Tjörnhammar, researcher at FOI and one of the authors of the report.

Advertisement

“The strength of multimodal models lies in their ability to simultaneously evaluate different forms of input data. It allows them to create a more accurate representation of reality by combining different forms of contextual data,” he says. This in turn gives multimodal models an increased capacity to understand and interact with complex situations, where different senses or information flows are involved.

Humans function multimodally

There are already commercial AI services to create text, music or art based on a text-based instruction. By operating multimodally, you can say that the AI system, or robot, becomes more like a human.

– We humans function multimodally by instinctively combining different senses, such as vision, hearing, touch and balance. Today, for example, there are multimodal industrial robots that perform advanced and autonomous tasks in a controlled environment, but from there it is difficult to take the step to military purposes, says Edward Tjörnhammar.

As an example, he takes the American robotics company Boston Dynamics which, among other things, develops robot dogs that are used for various purposes in industry and the construction industry, for example to move objects in environments that are dangerous for people to be in. Two robot dogs have also been purchased by the New York Fire Department. They are intended to function even in more complex environments, although the robot dogs do not use multimodal or deep neural network models.

– For example, if you want to make something multimodal, similar to the robot dogs, able to travel through unknown terrain on a battlefield, it immediately becomes much more complicated. The robot must be able to understand the terrain, the topology and the environment in order to be able to make independent decisions based on this external context. This could be done by a multimodal model, but today they are still too large and resource-intensive.

From commercial to military use

It is not unusual for the Armed Forces to look at commercial solutions and then adapt them for military purposes. Edward Tjörnhammar points to several areas where military applications of AI based on multimodal models are already being used or will soon be able to be used.

– It can be used, for example, to analyze satellite images, interpret sounds from the battlefield or understand geolocation, and integrate this data to make decisions in real time, he says, adding:

– But what happens when we get weapons systems that make their own decisions based on the situation? Or when information systems integrate more and more autonomous decision points? Ultimately, this is about life-changing moral decisions. More autonomous decisions in the decision chain means fewer moral decisions are made by humans.

One of the biggest concerns with AI, especially when it comes to military applications, is the risk of misuse.

– One can imagine a future where a post on social media could trigger a missile attack on a residential building.

Research at a breathtaking pace

This image brings to mind the targeted targeting Israel is carrying out in Gaza, Palestine. Edward Tjörnhammar confirms the image that Israel is at the forefront of military applications of the latest AI technology and multimodal models. For example, their AI systems Lavender and Gospel are used for automatic targeting to produce targeted attack lists, so-called “kill lists”.

– But we don't know today whether Gospel or Lavender use multimodal models because we don't know exactly what their capabilities are. But research is being driven forward at a breathtaking pace, not least by large technology companies investing massively in AI, but also by the defense forces of various countries.

The report points out that multimodal models, as part of what is commonly known as AI, will likely have a significant impact both on our everyday lives and on what future defense will look like.

– As long as the democratic conversation works, AI can benefit us all, but we must be vigilant. AI's potential in defense is great, but at the same time we do not want to reduce the number of moral and ethical decisions that should be made throughout the entire military organization, says Edward Tjörnhammar.

Read the report

Introduction to Multimodal Models