The Moral Machine Experiment

This is the third article from the AI Localization Think Tank Series “AI Research explained”.

In this series, we break down important AI research into simple terms and discuss what it means for our industry. 

Today's paper tackles one of the most uncomfortable questions in AI:

How should a machine decide when every decision causes harm?

The paper is called The Moral Machine Experiment and was published by researchers from MIT and collaborators in the journal Nature in 2018. It describes the largest global experiment ever conducted on human moral preferences in the age of autonomous machines

You can read about it below or watch the recoding.

https://www.youtube.com/watch?v=7b6WuHLi1wk

When machines face moral dilemmas

Imagine an autonomous car driving down the street.

Suddenly, the brakes fail. The car has only two possible outcomes:

• If it stays on course, it hits one group of people.

• If it swerves, it hits another.

There is no option where everyone survives.

When human drivers face such dilemmas, they react in a split second. They have no time to think, but act on instinct.

However, a machine cannot rely on instinct. It only has code. Which means someone must decide in advance how the car should behave in such scenarios. And that leads to a deeper question:

Whose moral values should that code represent?

To understand the problem researchers were trying to study, we need to start with a famous thought experiment from philosophy. It’s called The Trolley Problem.

The trolley problem

Imagine a runaway train racing down a track. Ahead of it are five workers who cannot move out of the way in time. On a sidetrack, there is a single person, who cannot move out of the way either.

An observer is standing next to a switch that can divert the train onto the track where the single person is standing.

If the observer does nothing, five people will die. If the observer pulls the switch, the train will be diverted to the sidetrack and only one person will die instead.

What should the observer do?

This dilemma contrasts two classic ethical frameworks:

• Utilitarian ethics: here the idea is that the morally right action is the one that produces the greatest overall good. In this case, sacrificing one life to spare five maximizes the overall good because a greater number of people are saved.

• Deontological ethics: here the focus falls on moral rules and duties. From this perspective, moral rules must be obeyed even if the outcomes are worse. So, actively causing someone’s death, even if the goal is to save others, is considered morally wrong.

For decades, philosophers debated these dilemmas mostly as abstract thought experiments. But now, with autonomous vehicles and AI systems making real-world decisions, questions like this are no longer just philosophical puzzles. They’re becoming engineering problems. And that means someone has to decide in advance what kind of moral logic gets written into that code.

Who decides? Engineers? Policymakers? Companies? Regulators? Society? And how do we find out what people actually want machines to do? This is exactly what the Moral Machine experiment set out to explore.

The world’s largest moral survey

To measure the public moral preferences at scale, the researchers built an online platform called The Moral Machine. It was essentially a multilingual interactive game. It is still up and running and you can access it at https://www.moralmachine.net/.

Participants in the game were shown two unavoidable crash scenarios involving a self-driving car and had to decide which outcome they considered less bad.

For example:

• Should the car stay on course and sacrifice the lives of 3 passengers?

• Or swerve and sacrifice one cat life?

Or:

• Should the car stay on course and hit three jaywalking pedestrians?

• Or swerve and sacrifice a single passenger?

Or:

• Should the car save a group of adults

• Or a smaller group of children?

The scenarios were not random at all. The experiment systematically varied nine moral dimensions.

Specifically, participants were asked whether to spare:

• humans over pets

• more lives over fewer lives

• young people over the elderly

• passengers over pedestrians

• women over men

• fit individuals over those that are not in good shape

• people with high social status over those with lower one

• law-abiding pedestrians over jaywalkers

Additionally, participants had to choose whether the car should stay on course or swerve, that whether they prefer for action over inaction.

The scenarios also included characters like doctors, executives, homeless people, athletes, children, and elderly people to see how those identities affected choices.

In total, the experiment explored nine different ethical factors influencing decisions.

Worldwide participation

The response was enormous. More than 2.3 million people participated. They represented 233 countries and territories. And contributed with almost 40 million decisions in ten languages.

That scale made it possible to observe global patterns in humanity’s moral reasoning.

What humans around the world agree on

Despite cultural differences, the data revealed a few universal moral preferences that appeared consistently across the world. Three stood out clearly:

Humans over animals

First and foremost, humans prioritize their own species across the board. Overwhelmingly, people chose to save humans over animals. This was one of the strongest global preferences observed in the dataset, and it gives us a pretty clear starting point for any kind of ethical framework.

Save more lives

Second, people tend to think in terms of numbers. The participants strongly favored saving a greater number of lives over a fewer lives.

This aligns with the classic utilitarian principle to maximize overall well-being.

Protect the young

Finally, humans (and probably not only humans) have a powerful instinct to protect the young.The data clearly showed that people consistently choose to spare children and young adults over the elderly. In fact, the characters most likely to be spared were babies, children, and pregnant women.

These three preferences — humans over animals, more lives over fewer, and young over elderly — emerged as possible building blocks for machine ethics.

However, while some preferences were nearly universal, others varied significantly across cultures and regions.

Where humanity’s moral intuitions diverge

When the researchers analyzed the collected responses by country, they discovered three major clusters of moral preferences.

Western cluster

This group included North America and many European countries. These societies showed strong preferences for:

• saving more lives

• prioritizing younger individuals

Eastern cluster

Countries in East and parts of South Asia showed weaker preferences for sacrificing the elderly. Respect for older members of society appeared to influence decisions.

Southern cluster

France, former French colonies, and many Latin American countries fell into this cluster. Participants from these countries showed:

• stronger tendencies to spare women

• stronger tendencies to spare physically fit individuals

• weaker preferences for saving humans over animals.

These patterns suggest that moral judgments are influenced by deep cultural values, not just individual opinion.

Culture, economics, and ethics

The researchers found that variations in moral preferences correlate with broader social factors. For example, individualistic societies, which emphasize personal autonomy, showed stronger preferences for saving the greater number of lives. Collectivist cultures, which often emphasize respect for elders, showed weaker preferences for sacrificing older individuals.

Economic and institutional factors also mattered. Participants from countries with strong rule-of-law institutions were less tolerant of pedestrians who violated traffic rules, such as jaywalking.

In other words, people's culture, religion, traditions and social environments influence their opinion on how machines should behave.

These findings raise an important challenge.

If moral preferences differ across cultures, what ethical standard should global technologies follow?

Machine ethics in a global world

The Moral Machine experiment shows that ethical expectations are not fully universal. They are influenced by culture, traditions, religion, institutions, and social norms. So, if an AI product is deployed globally, one ethical design may not feel legitimate everywhere.

And for localization professionals, that question sounds very familiar.

In Localization, we talk about adapting language, tone, imagery, and user experience to local cultures. But what happens when AI systems do not just translate content, but act based on culturally defined moral values embedded in their design? How do we adapt AI systems to local cultural expectations?

For example:

• Should an autonomous vehicle behave differently in different countries?

• Should AI safety policies be localized the way language and cultural content are?

• And who gets to decide the ethical framework behind these systems?

The Moral Machine experiment does not give us the answers. It only shows how hard the problem really is.

If AI systems cross borders, should their moral logic be localized?

That is not just a question for car manufacturers. It is a question for everyone who builds AI for the global world.