Time and Again, Social Media Giants Get Content Moderation Wrong: Silencing Speech about Al-Aqsa Mosque is Just the Latest Example

Since May 7, Al-Aqsa, one of the holiest sites for Muslims, and the neighborhood of Sheik Jarrah in Jerusalem have been the site of violent attacks against Palestinians, many of whom had come to the mosque to worship during Ramadan. During this violence, people took to Facebook and Instagram to post about what was happening to Palestinians using hashtags that referenced the mosque (#AlAqsa). But Instagram, owned by Facebook, blocked many of the posts through its content-filtering system because it inaccurately identified the hashtag as referencing a terrorist organization. While this may have been an error — though a demeanning and culturally ignorant one —its impact on users’ expression and the flow of information cannot be ignored. Palestinians and their supporters were silenced by one of the most powerful communications platforms in human history at a critical moment.

This wasn’t an isolated incident. In today’s world, a small handful of private corporations make some of the most important decisions regarding our online speech. Companies like Facebook, Twitter, and YouTube set rules for what kinds of content they’ll allow people to publish, defining what constitutes “hate speech,” “praise of terrorism,” and “fake news.” And they rely on algorithms to automatically flag certain words or images that appear to cross the lines. Once posts are removed or accounts are suspended, users — particularly those who don’t make headlines and cannot access backchannels — have too little recourse to appeal or get reinstated.

The major social media companies often get content moderation wrong — both because of their vague and sweeping rules, and because they make mistakes when applying those rules, often through blunt automated detection systems. Perfect content moderation may be impossible, but the major platforms can do better. They should give users more control, and respond to user experiences and reports, rather than rely so heavily on automated detection. They should also provide transparency, clarity, and appeals processes to regular users, and not just those that make headlines or have a personal connection at the company.

Here’s a rundown of some recent examples of content moderation gone wrong.

What Qualifies as Praise of “Terrorist” Groups?

Instagram’s ban on #AlAqsa is not the first time that the social media giants have misapplied their rules regarding praise or support of “terrorist” content. Over the summer, dozens of Tunisian, Syrian, and Palestinian activists and journalists covering human rights abuses and civilian airstrikes complained that Facebook had deactivated their accounts pursuant to its policy against “praise, support, or representation of” terrorist groups. Facebook deleted at least 52 accounts belonging to Palestinian journalists and activists in a single day in May, and more than 60 belonging to Tunisian journalists and activists that same month.

Relatedly, in October 2020, Zoom, Facebook, and YouTube all refused to host San Francisco State University’s roundtable on Palestinian rights, “Whose Narratives? Gender, Justice and Resistance”—featuring Leila Khaled, a member of the Popular Front for the Liberation of Palestine. The companies decided to censor the roundtable after it became a target of a coordinated campaign by pro-Israel groups that disagree with Khaled’s political views. In this instance, the companies pointed to anti-terrorism laws, rather than their own policies, as justification.

But these decisions, too, highlight platforms’ role in curtailing speech in an increasingly online world — not to mention problems with the underlying material support laws. One fundamental problem is that governments—not to mention the social media giants—do not have an agreed upon and transparent definition for terms like “terrorism,” “violent extremism,” or “extremism,” let alone “support” for them. As a result, rules regulating such content can be highly subjective and open the door to biased enforcement.

“Hate” Against Whom?

For years, Facebook moderators have treated the speech of women and people of color — including specifically when describing their experiences with sexual harassment and racism — differently than that of men and white people, pursuant to their community standards regarding hateful speech.

In 2017, when Black women and white people posted the exact same content, Facebook only suspended Black women’s accounts.

Notwithstanding tweaks to their community standards and algorithms, that problem persists. This year, for example, Facebook removed posts in groups created by users as spaces to vent about sexism and racism for posting “anything even remotely negative about men.” Meanwhile, “posts disparaging or even threatening women” stayed up. The company banned phrases like “men are trash,” “men are scum,” and even “I dislike men” — but posts like “women are trash” and “women are scum” were not removed, even if users reported the posts.

Similarly, in June 2020, at the height of protests against the police murder of George Floyd, Facebook’s automated system removed derogatory terms like “white trash” and “cracker” more often than slurs against Jewish, Black, and transgender individuals. Even after Facebook attempted to tweak its algorithmic enforcement to address this, Instagram still removed a post calling on others to “thank a Black woman for saving our country,” pursuant to the company’s “hate speech” guidelines.

As Vice and the Washington Post reported, these policies and applications have forced users to avoid using certain words, swapping out “m3n” for “men” and “wipipo” for “white.”

What is “Sexually Suggestive” and What is Socially Acceptable?

Similar problems have arisen with enforcement of the companies’ policies regarding “sexually suggestive” content and posts regarding “sexual solicitation.”

In 2019, users pushed back after Instagram repeatedly took down a topless photo of Nyome Nicholas-Williams, a fat Black woman, in which her arms covered her breasts. The users noted that nude images of skinny white women were not subjected to the same treatment and less likely to be considered inherently sexual.

More recently, Facebook mislabeled an ad by Unsung Lilly, an independent band whose members are a same-sex couple, as “sexually explicit” because it pictured the two women with their foreheads touching. As an experiment, Unsung Lilly uploaded the same ad with two different photos, one with a “nonromantic” photo of themselves and the other of a heterosexual couple touching foreheads. Both were approved by Facebook.

Along with queer users, sex workers, certified sex educators, and sexual wellness brands are also suffering from Instagram’s new community guidelines regarding “sexual solicitation.” These users report that posts including “flagged words, like ‘sex’ and ‘clitoris,’ have been removed from Instagram’s search function.”

Censorship Decisions are Arbitrary

Social media giants silence speech in arbitrary ways. For example, Facebook recently banned a recent college graduate for three days after he posted a rant criticizing those opposed to loan forgiveness and deeming America the “land of ignorance and greed.” The company also drew seemingly random lines regarding absentee voting, poll-watching, and COVID-19 during the 2020 presidential election, removing such posts as “If you vote by mail, you will get Covid!” and “My friends and I will be doing our own monitoring of the polls to make sure only the right people vote,” but permitting “If you vote by mail, be careful, you might catch Covid-19!” and “I heard people are disrupting going to the polls today. I’m not going to the polling station.”

A Path Forward

These examples highlight ongoing problems with the social media giants’ content moderation policies and practices — specifically, a lack of clarity in community standards, policies that cover too much speech, a disconnect between automated systems and actual user reports, and insufficient access to appeals. These problems have significant impacts in the offline world, altering discourse and limiting access to potentially critical information.

For these reasons, if they are to err, the major social media companies must err on the side of enabling expression. They should establish content policies that are viewpoint neutral, and that favor user control over censorship. And they should ensure that any expansion of content policies comes with a commensurate expansion of due process protections — clear rules, transparent processes, and the option of appeals — for all users. Because of the scale at which these platforms function, errors are inevitable, but there can be fewer and even those that still occur need not be permanent.