Pope Leo XIV sounded the alarm on AI in an open letter published on Monday and, later that day, Anthropic co-founder Chris Olah thanked him for doing so. According to the computer scientist, AI is in fact already returning “unsettling” findings and should probably be kept in check.
In a response posted to Anthropic’s website, which Olah also got to present in the Vatican City, he began by immediately conceding that all AI labs, including his own, are not solely operating to help people. He even noted that it might sound “strange” for him to admit as much.
“Every frontier AI lab — including Anthropic — operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing,” he said. “The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing — and I believe many of us do — we will always be influenced by those incentives.”
So, Olah argued, it is the responsibility of others, like the Pope, to openly criticize these companies. And that’s why Olah was “grateful” to Pope Leo for calling out the technology.
He then explained how AI systems are built and added that even the people who are building them are still learning, sharing, “And I will be honest: we keep finding things that are mysterious, even unsettling.”
Olah then presented three issues that others, not those who are creating AI, are supposed to handle. Among them, “the need for moral imagination and ambition regarding human flourishing” and “our duty to the global poor.”
The Anthropic co-founder did not present any potential solutions, but noted that parents are worried for their kids’ minds thanks to AI and that creatives are worried about their work. He further conceded that workers will be displaced by AI and that “supporting those displaced will be a moral imperative of historic proportions.”
“We need informed critics who will tell the labs when we are failing,” he finished. “We need moral voices that the incentives cannot bend.”
