Computer says No: How the EU's AI laws cause new injustice
Big Tech companies pushing for self-regulation, while their algorithms create new injustice in European public services (Photo: Sacha Fernandez)
By Alina Yanchur, Camille Schyns, Greta Rosén Fondahn, Sarah Pilz
When Roger Derikx applied for childcare benefits, he thought he would be receiving money - not losing it.
Derikx, 46 years old, was asked to repay the Dutch government €68,000. The problem: he was never told why he had to return the benefits, to which he was entitled. Authorities repossessed his car, and took a 40 percent cut of his salary for years.
"You have two little children, and you want to give them everything," Derikx said, "but every time they ask something, you have to say 'no' - that's hard."
Derikx was one of 26,000 parents accused of making fraudulent benefit claims by the Dutch tax authority, which used an algorithm for fraud detection. The system was "unlawful and discriminatory," an official investigation found, eventually leading Dutch prime minister Mark Rutte to resign earlier this year.
For Derikx, the investigation came too late. The financial hardship took such a toll on his marriage that he ended up getting a divorce. "That's a big price," Derikx said.
EU proposes AI regulation
The child benefits scandal in the Netherlands is not unique. Reports of abuse have surfaced across Europe.
Danish authorities also used artificial intelligence to identify children in "vulnerable families" that disproportionately targeted parents with foreign origins.
And the Netherlands, one of Europe's frontrunners in this field, used algorithms to create risk profiles of children under the age of 12.
Critics have long advocated against such uses of AI. In January, 61 advocacy organisations sent an open letter to the EU demanding "red lines" on AI applications that threatened fundamental rights.
Meanwhile, industry representatives argued that regulating AI would stifle innovation.
Upon taking office, European Commission president Ursula von der Leyen had made regulating artificial intelligence a political priority, and the EU claimed to seek compromise between addressing the risks and protecting innovation.
By April 2021, the EU Commission announced a long-awaited proposal to regulate artificial intelligence.
Corporate ascendency
Data compiled for this investigation shows that commission officials logged 152 official lobby meetings on AI over the time period from December 2019 until August 2021.
Corporations and business associations accounted for over 100, or two-thirds, of those meetings; less than one third was with civil society, academics, and trade unions.
Most organisations only got to meet the EU once. Google topped the list with nine meetings, the first of which occurred just three days after the Commission took office.
Alongside Google were Microsoft and Big Tech-funded business associations such as DigitalEurope. In total, 10 of the most active organisations had business interests.
European consumer organisation Beuc and European Digital Rights, an advocacy organisation, were the only non-profits near the top of the list.
"This is about democracy," said Margarida Silva from Corporate Observatory Europe, a Brussels-based lobby watchdog. "And we are giving more of a voice to those that have the most money."
An EU spokesperson declined to comment on why industry accounted for the majority of meetings.
And these numbers tell just a part of the story. Only high-ranking officials are obliged to disclose their meetings, but much of the lobbying occurs at lower levels.
DG Connect, the EU commission department responsible for drafting the proposal, stated by email that they had held almost 1,000 meetings on AI policy with "external stakeholders."
'Ethics-washing'
Lobby papers submitted to the EU, and seen by EUobserver's correspondents, detailed how tech firms pushed for more AI in the public sector.
Google encouraged "governments to promote public-sector adoption of AI to help create a thriving ecosystem."
Microsoft wrote that "the public sector could benefit enormously from the adoption of AI solutions." And DigitalEurope, the tech lobby that enjoys close relations with the European Commission, told the EU to "focus on [the] public sector for trust."
Tech firms also played up their capacity to self-regulate, minutes of meetings between tech firms and the commission, obtained through a freedom of information request, showed.
Facebook underlined it spends $3.7bn on "integrity." Google and Microsoft highlighted their corporate AI principles. And tech lobby DigitalEurope demanded that the EU adopted a "voluntary process-based certification scheme" - a practice whereby companies would themselves assess that their AI products lived up to the EU requirements before putting them on the market.
Critics have called this practice "ethics-washing": relying on claims of ethical corporate behaviour to stave off political oversight.
In fact, the European Consumer Organisation told the EU in late 2019 exactly that: "Avoid a purely 'ethics' narrative, which implies self-regulation."
"Public authorities often do not have the capacity to develop their own AI systems," said Ursula Pachl of Beuc, "which means that we are again dependent on these companies."
Public-sector AI
Regardless, the EU proposal conceded to key industry demands on self-regulation and the promotion of AI in the public sector.
When the Commission released its draft AI regulation there was much attention for the proposed ban on social scoring - ranking people for their trustworthiness - and some uses of facial recognition techniques.
But many public-sector uses of AI would still be permitted, even though they were classified as high-risk.
This included the use of algorithms in policing, the judicial system, and approval of social benefits.
Producers would only need to do a self-assessment to make sure their "high-risk" system complied with rules on transparency, risk management, and human oversight.
Presenting the proposal, Margrethe Vestager, EU commissioner for the digital portfolio, said the "legal framework shapes the trust we must build if we want people and businesses to embrace AI solutions."
The commissioner said she was worried about the uptake of artificial intelligence. She feared, for instance, that municipalities would not want to take the risk of using AI to support social workers in finding solutions for vulnerable citizens. "I think that would be a pity," Vestager said.
The draft AI regulation was accompanied by a commitment to invest €1bn per year in AI and making "the public sector a trailblazer for AI."
What had happened to Derikx in the Dutch child-benefit case was a clear illustration of the risks the EU intended to prevent with the regulation, an EU official with close insight into the policy process admitted.
Nonetheless, "the point is not that we shouldn't use AI," the official added. "We should use trustworthy AI that is properly developed. It's in the interest of everybody, including industry."
But concerns remain over whether the proposed certification system and self-assessment through tech providers themselves offers sufficient protection for EU citizens.
A recently published response by members of the EU expert group on AI drew a clear conclusion: "The proposal does not provide adequate fundamental rights protection, nor does it provide sufficient protection to maintain the rule of law and democracy, and thus fails to secure Legally Trustworthy AI."
Invisible injustice
The harm of algorithms is notoriously difficult to demonstrate. Derikx spent years in court to prove his innocence. Others lost custody of their children or committed suicide.
"Most of the cases will go undetected," said Sarah Chander, policy lead at European Digital Rights. "People will not even know that the decisions about their life are being made by an automated system."
"The burden is still on the people who were affected," adds Daniel Leufer of Access Now, one of the rights-advocacy groups.
In the Netherlands, victims were promised €30,000 in compensation, but the Dutch government has struggled to keep its commitment.
"Parents still need to fight for every penny," said Derikx, who has teamed up with a group of victims in their fight for compensation - and recognition. "It's an ugly, nasty process. But I won't let it go."
Alina Yanchur, Camille Schyns, Greta Rosén Fondahn and Sarah Pilz are freelance investigative journalists.
This is the first of a two-part investigation into the lobbying on the landmark EU proposal to regulate artificial intelligence, which leaves the door open to widespread use of the technology in the public sector.
Missing: Paper trail of EU meetings with Big Tech
Podcast: 'Big Tech' and the threat to democracy
'Redacted' - what Google and Microsoft told Mogherini on AI
How Big Tech dominates EU's AI ethics group
New EU rules to make companies liable for their AI failures
Coerced into pleading guilty: the rise of trial-waivers in Europe
EU commissioner Vera Jourova acknowledges minutes should have been produced. "I don't know why we didn't record what we spoke about," she said.
Do tech giants like Google, Apple or Facebook have too much power? Is their growing power a threat to democracy? These topics are highlighted in the latest Nordic Talks podcast, featuring three specialists - among them EU commissioner Margrethe Vestager.
6. Dec 2018, 16:46
Is Mogherini's 'Global Tech Panel' just a talking shop, or does it contribute to EU policy? New documents suggest the latter, but give little clue how technology leaders from Microsoft and Google influence new AI strategy.
Despite the responsibility for drafting the EU's artificial intelligence ethics guidelines, few of the expert group members were ethicists. In fact, 26 experts – nearly half of the group's 56 members – represented business interests.
As more-and-more AI-operated tools surround us — think of package-delivery drones, self-driving lorries, cleaning robots — the EU executive set out a bill to alleviate the burden of proof for consumers.
Almost half of prosecuted cases in Europe are processed outside the framework of a trial - and new research shows that many people will have been coerced into pleading guilty, in some cases for crimes they did not commit.
Europe continues to finance Russia's war in Ukraine with lucrative fossil fuel trades
Sanctions targeting Russia's fossil fuel earnings are now in place but as exports by European-controlled tankers persist, revenues still flow into the Kremlin purse.
Belarus dictator's family loves EU luxuries, flight data show
Lavish birthday trips to Michelin-starred restaurants in France and sunbathing in Italy — leaked flight data show how the family of ruthless Belarusian president Aleksander Lukashenko lived la dolce vita in Europe.