As the world’s leading streaming platform Twitch continues to grow, so do the problems it encounters with various types of bad behaviour. 2022 was probably the most dramatic year in Twitch’s history, and 2023 began with another breed of deeply gross scandal: The popular streamer Atrioc visited a so-called deepfake website live on-stream, which featured explicit and faked imagery of women streamers including Maya Higa and Pokimane.

Atrioc subsequently apologised and stepped back from streaming for a while, but the damage was done and the distress it caused those streamers featured on the website was public and clear to see. Twitch has now outlined how it plans to address “deepfake porn” incidents, both on and off the platform, and starts by addressing the term itself.

The phrase “deepfake porn” has acquired wide currency, but following advice from experts including Danielle Keats Citron (law professor and VP of the Cyber Civil Rights Initiative) and the UK Revenge Porn Helpline, Twitch will refer to this material as “synthetic non-consensual exploitative images” or “synthetic NCEI.” The reasoning goes as follows:

“Calling synthetic NCEI content “pornography” is incorrect. Porn (while prohibited on Twitch) should be consensual and should feature people who know they’re taking part in activities that others are going to see. That’s not what’s happening here, and why it’s important we reinforce unequivocally that this content is created without consent.”

This is incidentally somewhat similar to what’s currently happening with the term “revenge porn,” which experts in the field of abuse have taken issue with: Not least because “revenge” implies that the victim did something to deserve it. In that case the preferred terminology in the field is now “image-based abuse,” more clinical and accurate about the nature of the offense and with less of a victim-blaming implication.

Finally on the matter of terminology, Twitch points out that “deepfake” is something of a contemporary buzzword but not necessarily an accurate description of the methods used to generate content such as this.

This all matters because language is the building block of policy. Synthetic NCEI is prohibited under current Twitch policies and of course will continue to be so, but the wording of two site policies is going to be changed in the light of January’s events:

1. We’re updating our Adult Sexual Violence and Exploitation policy to make it more clear that intentionally promoting, creating, or sharing synthetic NCEI can result in an indefinite suspension on the first offense. 

2. We’re updating our Adult Nudity policy to include synthetic NCEI. Even if that NCEI is shown only briefly, or, for example, shown to express your outrage or disapproval of the content, it will be removed and will result in an enforcement. 

Twitch is now to host a creator camp on March 14 to gather further community and expert perspectives on how to deal with these problems in future. It will be led by Zara Ward, an expert on NCEI and a manager at the Revenge Porn helpline, as well as a streamer herself, and members of the Twitch safety team. Twitch has also assembled a list of resources that should be a first port of call for anyone affected by synthetic NCEI.

These are obviously nascent technologies and can be used for things other than synthetic NCEI, and much of what people create with them is going to be amusing or weird or innocent rather than awful. Twitch is somewhat stepping into the unknown in terms of taking a proactive approach in dealing with it and acknowledges that, outside of the obviously horrible stuff, there are going to be incidents with more nuance: “Not all synthetically-created content is sexual in nature, nor is all of it non-consensual […] But one thing is clear: when it’s used to create synthetic NCEI, that nuance disappears. The creation, promotion, or viewing of this content is not welcome on Twitch.”

The new policies are to be implemented this month. Following the creator camp, Twitch promises further updates about how its approach will evolve. This will be an interesting area to watch, not for the obviously heinous stuff that’s being explicitly addressed here, but the more borderline AI-dependent content on Twitch that can be harder to parse. This year also saw the debut of an AI Seinfeld show that lasted a few days before being suspended for a joke some saw as homophobic and transphobic; while others saw it as a meta-joke taking aim at unfunny rants about trans people and the gay agenda. Welcome to our AI future: It’s a mess.