Ex-Twitter Trust and Safety Head Talks About the Challenges Facing Decentralized Social Platforms

Yoel Roth, former head of Twitter’s Trust and Safety, now affiliated with Match, has voiced his concerns about the future of the open social web and its ability to combat misinformation, spam, and other harmful content, including child sexual abuse material (CSAM). In a recent interview, Roth highlighted the lack of moderation tools available to the fediverse—the open social network that includes platforms like Mastodon, Threads, Pixelfed, and others, alongside other open platforms like Bluesky.

Reflecting on pivotal moments from his time at Twitter’s Trust and Safety, Roth recalled the controversial decision to ban President Trump, the misinformation spread by Russian bot farms, and how even Twitter’s own figures, including CEO Jack Dorsey, fell prey to bots.

During his appearance on the podcast revolution.social with @Rabble, Roth remarked that endeavors to foster more democratically governed online communities within the open social web are often lacking in essential moderation resources.

“When we examined Mastodon and other services based on the ActivityPub [protocol], alongside early iterations of Bluesky and Threads during Meta’s development, it became clear that many platforms emphasizing community governance provided the least technical resources to their communities for enforcing policies,” Roth explained.

He also observed a notable decline in the transparency and legitimacy of decision-making that Twitter previously maintained. While many might disagree with Twitter’s ban on Trump, the company clearly articulated the reasons behind that choice. In contrast, current social media platforms tend to focus so heavily on preventing exploitation by malicious actors that they often fail to explain their decisions.

On numerous open social platforms, users frequently do not receive notifications when their posts are banned; instead, such posts vanish without trace.

“I don’t blame startups for their nascent stage or new software for missing features, but if the primary aim of a project is to enhance democratic governance, and we’ve regressed in that regard, then is it effective?” Roth questioned.

The economics of moderation

He further delved into the economic challenges surrounding moderation, indicating that the federated model has yet to prove its sustainability in this area.

For instance, an organization called IFTAS (Independent Federated Trust & Safety) aimed to create moderation tools for the fediverse, including those addressing CSAM, but ultimately ran out of funding and had to halt many projects early in 2025.

“We foresaw this two years ago. IFTAS anticipated it. Those involved in this sector largely volunteer their time, which has limitations, as people have families and bills to pay, while the costs of computing balloon as machine learning models are employed to identify specific harmful content,” he explained. “It ultimately becomes costly, and the economics of the federated approach to trust and safety never added up, and in my opinion, they still don’t.”

Conversely, Bluesky has chosen to employ moderators and enhance its trust and safety initiatives, although this is restricted to overseeing its own platform. Moreover, they provide tools allowing users to tailor their moderation settings.

“They are executing this work on a broader scale. There is certainly scope for improvement. I would value greater transparency from them. At its core, however, they are taking positive steps,” Roth asserted. As the service becomes more decentralized, Bluesky will face challenges regarding the balance between individual safety and community needs, he noted.

For example, in doxxing scenarios, an individual might not recognize that their personal information is being shared online due to their chosen moderation settings. Nonetheless, there must be accountability for enforcing such protections, even if the user is inactive on the primary Bluesky platform.

Where to draw the line on privacy

Another obstacle for the fediverse is that a strong emphasis on privacy may inhibit moderation efforts. While Twitter aimed to minimize unnecessary data collection, it still retained essential information like users’ IP addresses, access times, device identifiers, and more, which proved advantageous for forensic examinations related to entities such as Russian troll farms.

In contrast, fediverse administrators might overlook collecting crucial logs or refrain from reviewing them due to worries about infringing on user privacy.

However, without such data, it becomes increasingly challenging to identify real bots.

Roth recounted experiences from his Twitter tenure, emphasizing how it became commonplace for users to tag anyone with whom they disagreed as a “bot.” He initially set up alerts and manually reviewed these claims, encountering hundreds of “bot” accusations, all of which were unfounded. Notably, even Twitter co-founder and former CEO Jack Dorsey was misled, retweeting content from a Russian actor posing as Crystal Johnson, a Black woman from New York.

“The CEO interacted with this content, amplified it, and had no way of knowing that Crystal Johnson was actually a Russian troll,” Roth remarked.

The role of AI

A timely topic of discussion revolved around the transformative impact of AI. Roth cited recent research from Stanford, indicating that, in political contexts, large language models (LLMs) can even exceed humans in persuasiveness when appropriately calibrated.

This suggests that solutions based solely on content analysis may not suffice.

Instead, organizations must monitor additional behavioral signals—such as whether an entity creates multiple accounts, employs automation for posting, or posts at unusual hours across different time zones, he proposed.

“These behavioral indicators may still exist even in highly convincing content. That should be the starting point,” Roth advised. “If your focus is solely on the content, you’re engaging in a losing battle against advanced AI models.”