Over 100 use cases of automated decision-making systems in 16 European countries have been compiled by a research network for the 2020 edition of the Automating Society report by Bertelsmann Stiftung and AlgorithmWatch. The report shows: Even though algorithmic systems are increasingly being used by public administration and private companies, there is still a lack of transparency, oversight and competence.

In August this year, hundreds of students came together in London to protest against an algorithm: Their final grades had been determined by a software after the exams could not be held due to the Corona pandemic. In 40 percent of the cases, the algorithmic system had given the students worse marks than expected. Particularly affected were already disadvantaged students. Thus, anger about automated discrimination and a lack of explainability at such an important crossroads in life filled the streets of London.

Not all the use cases compiled in the Automating Society Report 2020 cast such a negative light on our increasingly digitized society. Since the last Automating Society Report from early 2019 automated decision-making (ADM) systems have been developed that have a positive impact on the lives of individuals and society as a whole: For example, the use of an ADM system in Spain has helped to better assess the risk in cases of domestic violence; similar efforts are being made in the German cities of Singen and Weimar with the DyRiAS system. In Portugal, the implementation of an automated system reduced medical prescription fraud by 80 percent in a single year.

It’s all about implementation­­­

The situation is, however, completely different with facial recognition technology: Since the first Automating Society Report in early 2019, the deployment of this technology has vastly increased and can now be found in European schools, airports, casinos and soccer stadiums. The accuracy of the systems is often doubtful, the risk of discrimination high and its legalization was sometimes granted only afterwards – as in the case of police operations in Slovenia. Since the danger of mass surveillance and the infringement of fundamental rights is particularly high with this technology, civil society actors from various countries demand to ban the technology until further notice.

However, only in few cases the question whether an ADM system should be used at all is relevant. The comprehensive research rather shows that it is decisive how ADM systems are implemented. “If systems are designed transparently, if there is a functioning oversight and if the population has the competences to interact with ADM systems in an informed way, they can contribute positively to our society,” says Sarah Fischer, Project Manager at the Bertelsmann Stiftung. But so far this has been the exception. The Automating Society Report 2020 therefore formulates clear and practicable recommendations for a public-interest oriented use of algorithmic systems.

More transparency concerning ADM systems is needed

One of the core findings of the Automating Society Report 2020 is the need for greater transparency of algorithmic systems. The general public needs to be informed about the use and the functionality of the technology – only then can there be an informed debate on the underlying value trade-offs, can wrongdoings be uncovered and denounced. Knowledge on the use of algorithms is the necessary prerequisite for any constructive debate. For this reason, the report recommends establishing public registers for ADM systems used within the public sector. The registers already introduced in Amsterdam and Helsinki could serve as a model for this.

The report shows that a lack of transparency is currently a problem in public and private applications in all 16 countries and that there are often legal hurdles. In Poland persons who disclose information about an algorithmic system used for detecting illegal banking activities can be punished with up to five years in prison. In Spain, on the other hand, quality control and supervision of ADM systems used in public administration are required by law. In practice, however, the authorities rarely share detailed information about the automated systems they use. This shows that transparency rules must not only be implemented but also enforced.

Ensuring effective oversight

Compared to the beginning of 2019, the regulation of automated systems, especially so-called “artificial intelligence (AI)”, has moved high up on the European agenda. The European Commission is committed to realizing “trustworthy AI” that puts people first. But here, too, details matter: Currently, the discourse is mainly driven by commercial and geopolitical imperatives with respect to the United States and China. The Automating Society Report therefore voices criticism regarding the lack of courage in shaping the regulatory framework at EU level and calls for a better legal basis as well as the implementation of oversight mechanisms for ADM deployment.

The report clearly shows the current lack of effective oversight and suggests concrete approaches for effectively auditing algorithmic systems: There should a systematic, independent and documented process in order to obtain objective findings. A legal framework that safeguards fundamental rights and focuses on the needs of the population can only be developed if it is created in a broad participatory process, especially with the involvement of civil society.

Informed civil society as the backbone

The case of automated grading in the United Kingdom shows how important the voice of civil society is: the public debate and the protests following the use of the ADM system led the government to retract the algorithmic grading system and accept the teachers’ evaluations for university admissions. In the Netherlands, civil rights activists have ensured that an opaque ADM system called SyRI, which was designed to detect welfare fraud, was banned by the court for breaches of fundamental rights. Often it is watchdog organizations such as AlgorithmWatch that draw attention to violations of the law in the first place and help raise public awareness of the risks and challenges of ADM systems.

How our automated society develops therefore also depends heavily on an informed civil society that has a basic understanding of ADM systems and wants to participate in the public debate. With the Automating Society Report 2020, which compiles the insights of more than 40 researchers, we would like to contribute to this debate. The report consists of 16 country chapters and one on the European Union, which provide an insight and overview of the automated society in 2020 through journalistic stories and catalogs of relevant ADM systems.

The country-specific reports can also be accessed separately in the local language. They are a translated excerpt of the Automating Society Report 2020.

This text is licensed under a Creative Commons Attribution 4.0 International License