This past June, the U.S. National Highway Traffic Safety Administration announced a probe into Tesla’s autopilot software. Data gathered from 16 crashes raised concerns over the possibility that Tesla’s AI may be programmed to quit when a crash is imminent. This way, the car’s driver, not the manufacturer, would be legally liable at the moment of impact.
It echoes the revelation that Uber’s self-driving car, which hit and killed a woman, detected her six seconds before impact. But the AI was not programmed to recognize pedestrians outside of designated crosswalks. Why? Because jaywalkers are not legally there.
Some believe these stories are proof that our concept of liability needs to change. To them, unimpeded continuous innovation and widespread adoption of AI is what our society needs most, which means protecting innovative corporations from lawsuits. But what if, in fact, it’s our understanding of competition that needs to evolve instead?
If AI is central to our future, we need to pay careful attention to the assumptions around harms and benefits programmed into these products. As it stands, there is a perverse incentive to design AI that is artificially innocent.
A better approach would involve a more extensive harm-reduction strategy. Maybe we should be encouraging industry-wide collaboration on certain classes of life-saving algorithms, designing them for optimal performance rather than proprietary advantage.
Every fix creates a new problem
Some of the loudest and most powerful corporate voices want us to trust machines to solve complex societal problems. AI is hailed as a potential solution for the problems of cross-cultural communication, health care and even crime and social unrest.
Corporations want us to forget that AI innovations reflect the biases of the programmer. There is a false belief that as long as the product design pitch passes through internal legal and policy constraints, the resulting technology is unlikely to be harmful. But harms emerge in all sorts of unexpected ways, as Uber’s design team learned when their vehicle encountered a jaywalker for the first time.
What happens when the nefarious implications of an AI are not immediately recognized? Or when it is too difficult to take the AI offline when necessary? Which is what happened when Boeing hesitated to ground the 737 Max jets after a programming glitch was found to cause crashes — and 346 people died as a result.
We must constantly reframe technological discussions in moral terms. The work of technology demands discrete, explicit instructions. Wherever there is no specific moral consensus, individuals simply doing their job will make a call, often without taking the time to consider the full consequences of their actions.
