#algorythmes

Ethnic Profiling

Whistleblower reveals Netherlands’ use of secret and potentially illegal algorithm to score visa applicants

In 2022, a report (▻https://www.rijksoverheid.nl/actueel/nieuws/2022/12/12/confronterend-rapport-racisme-bij-buitenlandse-zaken) commissioned by the Dutch Ministry of Foreign Affairs concluded that the agency’s internal culture was riddled with structural racism. Some employees recounted being described as “monkeys” while others were continually asked to disavow terrorist attacks. In response, the foreign minister Wopke Hoekstra promised reform. “This is not who we want to be,” he said.

The Ministry of Foreign Affairs is not alone one among Dutch institutions that have been in the spotlight for structural racism. The Netherlands prides itself on automated decision making to reduce bias. But in the last two years of reporting from Lighthouse Reports have revealed wide-scale use of algorithmic risk profiling systems across the Netherlands (▻https://www.lighthousereports.com/investigation/suspicion-machines).

New, leaked documents obtained by Lighthouse Reports and NRC reveal that at the same time Hoekstra was promising change, officials were sounding the alarm over a secretive algorithm that ethnically profiles visa applicants. They show the agency’s own data protection officer — the person tasked with ensuring its use of data is legal — warning of potential ethnic discrimination. Despite these warnings, the ministry has continued to use the system.

Unknown to the public, the Ministry of Foreign Affairs has been using a profiling system to calculate the risk score of short-stay visa applicants applying to enter the Netherlands and Schengen area since 2015.

An investigation by Lighthouse and NRC reveals that the ministry’s algorithm, referred to internally as Informatie Ondersteund Beslissen (IOB), has profiled millions of visa applicants using variables like nationality, gender and age. Applicants scored as ‘high risk’ are automatically moved to an “intensive track” that can involve extensive investigation and delay.

“Family members of Dutch citizens with a migration background are prevented in all kinds of ways by the Ministry of Foreign Affairs from getting a visa for short stays.” said Kati Piri, a Dutch MP. Regardless of the ministry claiming efficiency over the use of the IOB system, “from countries like Morocco and Suriname, it is incredibly difficult to get a visa,” Piri added.
METHODS

In February 2023 Lighthouse obtained previously unpublished, crucial documents pointing to heated internal discussion around the Ministry of Foreign Affairs’ use of nationality and gender in its algorithmic risk profiling.

According to documents, the internal Data Protection Officer advised the ministry to “immediately stop profiling visa applicants to distinguish them partly on the basis of nationality and then treating them unequally on the basis of that distinction.”

Examples of so-called “risk profiles” used by the algorithm include Surinamese men aged between 26-40 who applied from Paramaribo and unmarried Nepalese men aged around 35-40 who applied for a tourist visa. Officials claim that the risk profiles are also based on data from third parties to see if a group of individuals from the same nationalities attempted to apply for asylum.

Despite the Data Protection Officer’s serious and continuous warnings from at least 2021, the documents suggest that Hoekstra delayed making a decision about the system. With reporting partner NRC, we spoke with internal sources who confirmed it is still active and uses nationality as a variable.
STORYLINES

An internal watchdog at the Ministry of Foreign Affairs has pressed them since midway through 2022 to immediately halt algorithmic profiling of visa applicants based on their nationality. Despite pushback from officials, the DPO continued to insist that the algorithmic assessment system is potentially discriminatory.

With the national newspaper NRC, we chronicled how the ministry turned to algorithmic profiling in the midst of a larger drive to cut costs. Ministry officials maintain that the algorithm helps to centralise workloads and remove bias of caseworkers. They also claim that three consulted experts have concluded that the use of nationality in the algorithmic profiling system was proportional and not discriminatory.

Statistics from the ministry suggest that being flagged as a high risk can carry serious consequences. As of March 2023, 33 percent of applications in the intensive, ‘high risk’ track were rejected, whereas only 3.5 percent of applications in the normal, ‘low risk’ track were rejected.

Being flagged for the intensive track can come with months of delays and consequences in an already difficult bureaucratic process. With NRC we spoke to Saadia Ourhris, a Moroccan-Dutch mother who recounted a relative of hers receiving constant automated rejections when attempting to visit her in the Netherlands.

Reacting to our findings, Dutch MP Kati Piri described the use of algorithmic visa profiling as “downright shocking.”

▻https://www.lighthousereports.com/investigation/ethnic-profiling

#profilage_racial #racial_profiling #profilage_ethnique #visa #migrations #Pays-Bas #racisme #racisme_structurel #algorythmes

ping @cede

Tecnologie per il controllo delle frontiere in Italia: identificazione, riconoscimento facciale e finanziamenti europei

https://i.imgur.com/e9KBWCb.png

L’utilizzo documentato di un sistema di riconoscimento facciale da parte dell’amministrazione comunale di Como o da parte della Polizia di Stato italiana ha aperto anche in Italia il dibattito su una tecnologia che all’estero, già da tempo, si critica per la sua inaccuratezza algoritmica e per la poca trasparenza. Un tema sicuramente preoccupante per tutti ma che certamente assume caratteristiche ancor più pericolose quando interessa gruppi o individui particolarmente vulnerabili come migranti, rifugiati e richiedenti asilo. In questo caso i dati e le informazioni sono processati da parte di agenzie governative a fini di sorveglianza e di controllo, con tutele assai minori rispetto ai cittadini europei ed italiani. Ciò comporta un grande rischio per queste persone poiché le procedure di identificazione al loro arrivo in Italia, effettuate all’interno degli hotspot, rischiano di essere un’arma a doppio taglio per la loro permanenza nel nostro Paese (o in Europa), determinando uno stato di sorveglianza continuativa a causa della loro condizione. Ancora una volta alcune categorie di persone sono costrette ad essere “banco di prova” per la sperimentazione di dispositivi di controllo e sorveglianza, a dimostrazione che esistono e si reiterano rapporti di potere anche attraverso la tecnologia, portando alla creazione di due categorie distinte: chi sorveglia e chi è sorvegliato.

Da questa ricerca emerge che le procedure di identificazione e categorizzazione dei migranti, rifugiati o richiedenti asilo fanno ampio utilizzo di dati biometrici—la polizia italiana raccoglie sia le impronte digitali che la foto del loro volto—ma non è sempre facile comprendere in che modo vengano applicate. Nel momento in cui viene effettuata l’identificazione, le categorie sopra citate hanno ben poche possibilità di conoscere appieno il percorso che faranno i loro dati personali e biometrici, nonché di opporsi al peso che poi questo flusso di informazioni avrà sulla loro condizione in Italia e in tutta l’Unione Europea. Quest’ultima, infatti, promuove da alcuni anni la necessità di favorire l’identificazione dei migranti, stranieri e richiedenti asilo attraverso un massiccio utilizzo di tecnologie: a partire dal mare, pattugliato con navi e velivoli a pilotaggio remoto che “scannerizzano” i migranti in arrivo; fino all’approdo sulla terraferma, dove oltre all’imposizione dell’identificazione e del fotosegnalamento i migranti hanno rischiato di vedersi puntata addosso una videocamera “intelligente”.

Ampio spazio è lasciato alla trattazione di come lo stato italiano utilizzi la tecnologia del riconoscimento facciale già da alcuni anni, senza che organizzazioni indipendenti o professionisti possano controllare il suo operato. Oltre alla mancata trasparenza degli algoritmi che lo fanno funzionare, infatti, non sono disponibili informazioni chiare sul numero di persone effettivamente comprese all’interno del database che viene utilizzato proprio per realizzare le corrispondenze tra volti, AFIS (acronimo di Automated Fingerprint Identification System).

Nelle intenzioni della polizia italiana, infatti, c’era l’impiego di un sistema di riconoscimento facciale, SARI Real-Time, per riconoscere in tempo reale l’identità delle persone a bordo di un’imbarcazione durante le fasi di sbarco sulle coste italiane. Il sistema SARI Real-Time, acquistato originariamente per l’utilizzo durante manifestazioni ed eventi pubblici, è stato reso inutilizzabile a seguito della pronuncia negativa del Garante della Privacy: rischierebbe di introdurre una sorveglianza di massa ingiustificata. La decisione del Garante tutela quindi non solo coloro che vivono nel nostro paese ma anche chi, in una situazione di estrema vulnerabilità, arriva sulle nostre coste dopo un viaggio interminabile e si vede sottoposto a un controllo sproporzionato ancor prima di ricevere supporto medico e valutazione dello status legale.

Come Centro Hermes per la Trasparenza e i Diritti Umani Digitali dal 2011 ci interroghiamo sul funzionamento e sullo scopo delle innovazioni in campo tecnologico, analizzandole non solo da un punto di vista tecnico ma anche attraverso la lente dei diritti umani digitali. Negli ultimi anni la datificazione della società attraverso la raccolta indiscriminata di dati personali e l’estrazione di informazioni (e di valore) relative al comportamento e alle attività svolte da ognuno di noi sono il tema centrale di ricerca, analisi e advocacy dell’associazione. Siamo convinti infatti che vada messa in dubbio non solo la tecnologia digitale creata al presunto scopo di favorire il progresso o di dare una risposta oggettiva a fenomeni sociali complessi, ma anche il concetto di tecnologia come neutra e con pressoché simili ripercussioni su tutti gli individui della società. È importante a nostro parere che qualunque discorso sulla tecnologia racchiuda in sé una più ampia riflessione politica e sociologica, che cerchi di cogliere la differenza tra chi agisce la tecnologia e chi la subisce.

Principali risultati:

https://i.imgur.com/u6aDLCO.png https://i.imgur.com/feYAR79.png

▻https://protecht.hermescenter.org
#rapport #Hermes #frontières #Italie #reconnaissance_faciale #réfugiés #asile #migrations #contrôles_frontaliers #identification #financements_européens #technologie #complexe_militaro-industriel #Côme #surveillance #biométrie #données_biométriques #catégorisation #photos #empreintes_digitales #AFIS #algorythmes #Automated_Fingerprint_Identification_System #SARI_Real-Time #database #base_de_données

sur la mise en place de reconnaissance faciale à Côme:
▻https://seenthis.net/messages/859963

ping @etraces

Don’t assume technology is racially neutral

Without adequate and effective safeguards, the increasing reliance on technology in law enforcement risks reinforcing existing prejudices against racialised communities, writes Karen Taylor.

https://www.theparliamentmagazine.eu/sites/www.theparliamentmagazine.eu/files/styles/original_-_local_copy/entityshare/33329%3Fitok%3DXJY_Dja6#.jpg

Within the European Union, police and law enforcement are increasingly using new technologies to support their work. Yet little consideration is given to the potential misuse of these technologies and their impact on racialised communities.

When the everyday experience of racialised policing and ethnic profiling is already causing significant physical, emotional and social harm, how much will these new developments further harm people of colour in Europe?

With racialised communities already over-policed and under-protected, resorting to data-driven policing may further entrench existing discriminatory practices, such as racial profiling and the construction of ‘suspicious’ communities.

This was highlighted in a new report published by the European Network Against Racism (ENAR) and the Open Society Justice Initiative.

Using systems to profile, survey and provide a logic for discrimination is not new; what is new is the sense of neutrality afforded to data-driven policing.

The ENAR report shows that law enforcement agencies present technology as ‘race’ neutral and independent of bias. However, such claims overlook the evidence of discriminatory policing against racialised minority and migrant communities throughout Europe.

European criminal justice systems police minority groups according to the myths and stereotypes about the level of ‘risk’ they pose rather than the reality.

This means racialised communities will feel a disproportionate impact from new technologies used for identification, surveillance and analysis – such as crime analytics, the use of mobile fingerprinting scanners, social media monitoring and mobile phone extraction - as they are already overpoliced.

For example, in the UK, social media is used to track ‘gang-associated individuals’ within the ‘Gangs Matrix’. If a person shares content on social media that references a gang name or certain colours, flags or attire linked to a gang, they may be added to this database, according to research by Amnesty International.

Given the racialisation of gangs, it is likely that such technology will be deployed for use against racialised people and groups.

Another technology, automatic number plate recognition (ANPR) cameras, leads to concerns that cars can be ‘marked’, leading to increased stop and search.

The Brandenburg police in Germany used the example of looking for “motorhomes or caravans with Polish license plates” in a recent leaked internal evaluation of the system.

Searching for license plates of a particular nationality and looking for ‘motorhomes or caravans’ suggests a discriminatory focus on Travellers or Roma.

Similarly, mobile fingerprint technology enables police to check against existing databases (including immigration records); and disproportionately affects racialised communities, given the racial disparity of those stopped and searched.

Another way in which new technology negatively impacts racialised communities is that many algorithmically-driven identification technologies, such as automated facial recognition, disproportionately mis-identify people from black and other minority ethnic groups – and, in particular, black and brown women.

This means that police are more likely to wrongfully stop, question and potentially arrest them.

Finally, predictive policing systems are likely to present geographic areas and communities with a high proportion of minority ethnic people as ‘risky’ and subsequently make them a focus for police attention.

Research shows that data-driven technologies that inform predictive policing increased levels of arrest for racialised communities by 30 percent. Indeed, place-based predictive tools take data from police records generated by over-policing certain communities.

Forecasting is based on the higher rates of police intervention in those areas, suggesting police should further prioritise those areas.

We often – rightly – discuss the ethical implications of new technologies and the current lack of public scrutiny and accountability. Yet we also urgently need to consider how they affect and target racialised communities.

The European Commission will present a proposal on Artificial Intelligence within 100 days of taking office. This is an opportunity for the European Parliament to put safeguards in place that ensure that the use of AI does not have any harmful and/or discriminatory impact.

In particular, it is important to consider how the use of such technologies will impact racialised communities, so often overlooked in these discussions. MEPs should also ensure that any data-driven technologies are not designed or used in a way that targets racialised communities.

The use of such data has wide-ranging implications for racialised communities, not just in policing but also in counterterrorism and immigration control.

Governments and policymakers need to develop processes for holding law enforcement agencies and technology companies to account for the consequences and effects of technology-driven policing.

This should include implementing safeguards to ensure such technologies do not target racialised as well as other already over-policed communities.

Technology is not neutral or objective; unless safeguards are put in place, it will exacerbate racial, ethnic and religious disparities in European justice systems.

▻https://www.theparliamentmagazine.eu/articles/opinion/don%E2%80%99t-assume-technology-racially-neutral

#neutralité #technologie #discriminations #racisme #xénophobie #police #profilage_ethnique #profilage #données #risques #surveillance #identification #big-data #smartphone #réseaux_sociaux #Gangs_Matrix #automatic_number_plate_recognition (#ANPR) #Système_de_reconnaissance_automatique_des_plaques_minéralogiques #plaque_d'immatriculation #Roms #algorythmes #contrôles_policiers

–--------

Pour télécharger le rapport :

https://i.imgur.com/2NT3jhc.png

ping @cede @karine4 @isskein @etraces @davduf

How white engineers built racist code – and why it’s dangerous for black people

As facial recognition tools plays a bigger role in fighting crime, inbuilt racial biases raise troubling questions about the systems that create them
▻https://www.theguardian.com/technology/2017/dec/04/racist-facial-recognition-white-coders-black-people-police
@etraces @copycult @constant via Marie Lechner.
#algorythmes #biases

Algorithms replace your biases with someone else’s
▻https://thesocietypages.org/socimages/2017/07/05/algorithms-replace-your-biases-with-someone-elses-biases
Weapons of Math Destruction
How big data increases inequality and threatens democracy
▻https://mathbabe.org
▻https://weaponsofmathdestructionbook.com/blog
#data #algorythmes

MUTE VOL. 3, NO. 4 - SLAVE TO THE ALGORITHM
▻http://www.metamute.org/editorial/magazine/mute-vol.-3-no.-4-slave-to-algorithm
As the financial crisis fastens its grip ever tighter around the means of human and natural survival, the age of the algorithm has hit full stride. This phase-shift has been a long time coming of course, and was undoubtedly as much a cause of the crisis as its effect, with self-propelling algorithmic power replacing human labour and judgement and creating event fields far below the threshold of human perception and responsiveness.
#trading #finance #Algorythmes #haute_frequence #alternatives #revue