Rosalie Gillett, Postdoctoral Research Fellow, Centre of Excellence for Automated Decision-Making and Society, Queensland University of Technology, Queensland University of Technology; Kath Albury, Professor of Media and Communication and Associate Investigator, ARC Centre of Excellence for Automated Decision-Making + Society, Swinburne University of Technology, and Zahra Zsuzsanna Stardust, Postdoctoral Research Fellow, Centre of Excellence in Automated Decision-Making and Society, Queensland University of Technology

Dating apps have been under increased scrutiny for their role in facilitating harassment and abuse.

Last year an ABC investigation into Tinder found most users who reported sexual assault offences didn’t receive a response from the platform. Since then, the app has reportedly implemented new features to mitigate abuse and help users feel safe.

In a recent development, New South Wales Police announced they are in conversation with Tinder’s parent company Match Group (which also owns OKCupid, Plenty of Fish and Hinge) regarding a proposal to gain access to a portal of sexual assaults reported on Tinder. The police also suggested using artificial intelligence (AI) to scan users’ conversations for “red flags”.

Tinder already uses automation to monitor users’ instant messages to identify harassment and verify personal photographs. However, increasing surveillance and automated systems doesn’t necessarily make dating apps safer to use.

User safety on dating apps

Research has shown people have differing understandings of “safety” on apps. While many users prefer not to negotiate sexual consent on apps, some do. This can involve disclosure of sexual health (including HIV status) and explicit discussions about sexual tastes and preferences.

If the recent Grindr data breach is anything to go by, there are serious privacy risks whenever users’ sensitive information is collated and archived. As such, some may actually feel less safe if they find out police could be monitoring their chats.

Adding to that, automated features in dating apps (which are supposed to enable identity verification and matching) can actually put certain groups at risk. Trans and non-binary users may be misidentified by automated image and voice recognition systems which are trained to “see” or “hear” gender in binary terms.

Trans people may also be accused of deception if they don’t disclose their trans identity in their profile. And those who do disclose it risk being targeted by transphobic users.

Increasing police surveillance

There’s no evidence to suggest that granting police access to sexual assault reports will increase users’ safety on dating apps, or even help them feel safer. Research has demonstrated users often don’t report harassment and abuse to dating apps or law enforcement.

Consider NSW Police Commissioner Mick Fuller’s misguided “consent app” proposal last month; this is just one of many reasons sexual assault survivors may not want to contact police after an incident. And if police can access personal data, this may deter users from reporting sexual assault.

With high attrition rates, low conviction rates and the prospect of being retraumatised in court, the criminal legal system often fails to deliver justice to sexual assault survivors. Automated referrals to police will only further deny survivors their agency.

Moreover, the proposed partnership with law enforcement sits within a broader project of escalating police surveillance fuelled by platform-verification processes. Tech companies offer police forces a goldmine of data. The needs and experiences of users are rarely the focus of such partnerships.

Match Group and NSW Police have yet to release information about how such a partnership would work and how (or if) users would be notified. Data collected could potentially include usernames, gender, sexuality, identity documents, chat histories, geolocation and sexual health status.

The limits of AI

NSW Police also proposed using AI to scan users’ conversations and identify “red flags” that could indicate potential sexual offenders. This would build on Match Group’s current tools that detect sexual violence in users’ private chats.

While an AI-based system may detect overt abuse, everyday and “ordinary” abuse (which is common in digital dating contexts) may fail to trigger an automated system. Without context, it’s difficult for AI to detect behaviours and language that are harmful to users.

It may detect overt physical threats, but not seemingly innocuous behaviours which are only recognised as abusive by individual users. For instance, repetitive messaging may be welcomed by some, but experienced as harmful by others.

Also, even as automation becomes more sophisticated, users with malicious intent can develop ways to circumvent it.

If data are shared with police, there’s also the risk flawed data on “potential” offenders may be used to train other predictive policing tools.

We know from past research that automated hate-speech detection systems can harbour inherent racial and gender biases (and perpetuate them). At the same time we’ve seen examples of AI trained on prejudicial data making important decisions about people’s lives, such as by giving criminal risk assessment scores that negatively impact marginalised groups.

Dating apps must do a lot more to understand how their users think about safety and harm online. A potential partnership between Tinder and NSW Police takes for granted that the solution to sexual violence simply involves more law enforcement and technological surveillance.

And even so, tech initiatives must always sit alongside well-funded and comprehensive sex education, consent and relationship skill-building, and well-resourced crisis services.

The Conversation was contacted after publication by a Match Group spokesperson who shared the following:

“We recognize we have an important role to play in helping prevent sexual assault and harassment in communities around the world. We are committed to ongoing discussions and collaboration with global partners in law enforcement and with leading sexual assault organizations like RAINN to help make our platforms and communities safer. While members of our safety team are in conversations with police departments and advocacy groups to identify potential collaborative efforts, Match Group and our brands have not agreed to implement the NSW Police proposal.”

This article is republished from The Conversation under a Creative Commons license. Read the original article.