Health protection is non-negotiable in the AI Act negotiations

A health-centric approach to the Artificial Intelligence (AI) Act is essential for the protection of health and fundamental rights of European citizens, write Hannah van Kolfschooten and Janneke van Oirschot. 

Hannah van Kolfschooten, LL.M. is a PhD Researcher at the Law Centre for Health and Life, University of Amsterdam, working on health AI and patients’ rights. 

Janneke van Oirschot, M.Sc. is a Research Officer working on AI and medicines at independent non-profit organisation Health Action International (HAI).

The European Commission’s proposal for an Artificial Intelligence (AI) Act has been the topic of a heated debate since its publication in April 2021. Civil society organisations believe the proposal falls short on fundamental rights protection, industry is worried it will stifle innovation, and governments fear consequences for national security. We critique the AI Act for neglecting the risks health AI pose to patients’ health and fundamental rights.

The 3,000 amendments to the Act tabled by political groups in the European Parliament say a lot about how controversial regulation of AI really is. This summer, the Parliament’s co-rapporteurs start the negotiation process with compromise amendments. Our message to MEPs, who will need to vote on the amendments is the following: Make health non-negotiable. A health-centric approach to the AI Act is essential for the protection of health and fundamental rights of European citizens, in particular the rights to access to healthcare, non-discrimination and privacy.  

AI is the simulation of human intelligence by machines. AI systems are software-based technologies that use certain data-driven approaches to solve specific problems. What all AI systems have in common, is that they recognise patterns in enormous amounts of data.

AI in the health sector is not like AI in any other sector and deserves special consideration because (1) people’s health is at stake, (2) people are in a vulnerable position when in need of healthcare, (3) the collection of health data has dramatically increased in recent times and (4) health data is historically littered with bias. Because of these characteristics, health AI faces unique risks that need to be specifically addressed in the AI Act. 

Take disease outbreak surveillance as an example. Many people with flu-like symptoms use Google for self-diagnosis. AI can use this data to monitor and predict the spread of infectious diseases. This can be useful for public health officials to make decisions about infectious disease control and how to distribute care resources.

But how accurate are these AI systems when the model is based on subjective user data? Limited regulation of the quality of health AI will lead to distrust in public health and healthcare, breeding hesitancy in access to healthcare. What is more, increased use and sharing of health data threatens privacy and data protection rights.

Another example is the use of AI for medical diagnostics. AI can be used to identify skin cancer in images of skin lesions, after being trained on the basis of thousands of images of “healthy” and cancerous skin lesions. But what happens when image datasets were non-representative, incomplete or of low-quality?

Biases in the training data can lead to discrimination and individual injury or even death. Especially racial bias may lead to incorrect diagnoses and deepen existing socio-economic inequality, something that is not taken into account in current regulation on medical technology. Additionally, lack of transparency and explainability threatens patients’ rights to information and informed consent to medical treatment.

These are just a couple of illustrations of the risks of AI usage for health, one of the most popular sectors for AI deployment in the European Union. Yet, the AI Act does not specifically address health AI and does not provide solutions for its key risks. It can’t be stressed enough that health must be prioritised when MEPs negotiate their amendments over the coming months, with some tabled amendments that deserve particular support. 

Foremost, given its extensive risk, important AI uses in health and healthcare should be marked as high-risk, which will ensure more stringent regulatory requirements. 

Second, high-risk AI should undergo a fundamental rights impact assessment, which takes into account risks to human health. Also technical documentation of health AI should include an assessment of its risks for health, safety and fundamental rights. 

Finally, AI systems that disadvantage groups based on health status should be prohibited completely. 

Similarly, we call on MEPs to strongly oppose amendments that remove health AI from the current list of ‘high-risk AI uses’ or add extra requirements for AI systems to be marked high-risk. 

It’s high time to take on a health-centric approach to the AI Act. It’s worth reiterating: health protection is non-negotiable in the AI Act negotiations.