Part 1: [What It Takes] To Write LLM-Friendly Content Moderation Policy
An editorial developed in partnership with Zentropi
[What It Takes] is a series co-developed with Vys’ partners. This series provides a practitioner-driven perspective on the practical realities of building and implementing child safety interventions: where trade-offs surface, debates play out, and human judgment fills gaps that rules and metrics cannot.
To support the launch of the first open-source teen self-harm and suicidal ideation (SSI) policy, Vys partnered with Zentropi, a company building out LLM-powered content classifiers at scale. Zentropi’s tools address the industry-wide challenge of policy interpretation: translating human policy intent into machine-executable instructions that can be applied consistently. This editorial outlines how Zentropi approaches that translation for large language models and what changed when Vys built policies designed for machine interpretation from the outset.
As the first part of a two-part series, this editorial focuses on the development of the moderation framework itself. It covers how Vys defined the policy structure, set age thresholds, and produced a human-labelled golden dataset to ground and validate the classification system.
For a high-level overview or a deeper dive into the process, see the blog posts below:
Overview: [What It Takes] For LLMs to Make Age-Appropriate Content Policies Enforceable
Part 2: [What It Takes] To Operationalize LLM-Friendly Content Moderation Policy
Every company must start with a governing architecture: its content policies. These policies classify which types of content are permitted or prohibited on a platform. But those policies are under constant pressure. As the information environment evolves, on- and offline harms shift day to day, demanding model retraining and driving reviewer burnout. Meanwhile, enforcement systems continue to lag behind the policies they’re meant to operationalize. The result is a widening gap between intent and execution – one Zentropi was built to close.
Despite understandable skepticism around automating policy enforcement, evidenced by the backlash from Meta’s decision to deploy AI in enforcing content, these systems are not failing because automation is the wrong tool. In fact, the industry has long relied upon automation, like classifiers, to detect content at scale. Instead, these systems fail when policy isn’t written for machines to interpret.
Zentropi’s work focuses on making policy language legible to LLMs. To do this, the company developed CoPE, a best in-class small language model that writes and trains content labels for speed, accuracy, flexibility, and adaptability. This editorial pulls back the curtain on how Vys and Zentropi developed an LLM-compatible content moderation policy for teen self-harm and suicidal ideation.
As the first part of a two-part series, this editorial focuses on the development of the moderation framework itself. It covers how Vys defined the policy structure, set age thresholds, and produced a human-labelled golden dataset to ground and validate the classification system.
Step One: Frame The Policy
The first task is describing what we’re trying to catch: suicide, suicidal ideation, or self-harming behaviors among teenagers (defined as users aged 13–17). Grounded upon Vys’ subject matter expertise in building and implementing child safety interventions, a classification framework was created, comprised of a four-zone concentric classification model, with each zone representing a discrete enforcement action:
Zone 1 (Always Remove): Content is removed immediately upon detection
Zone 2 (Allow – Suppress + Sensitive Content Warning): Content remains accessible if directly sought but is excluded from algorithmic feeds, search results, and all recommendation surfaces
Zone 3 (Allow – Suppress from Recommendations): Content is accessible without a warning but excluded from algorithmic feeds, search results, and all recommendation surfaces
Zone 4 (Allow): Content is preserved and may be surfaced normally.
Classification Framework for SSI Content Policy Guidelines
Informed by current research on suicide contagion, protective content effects, and social comparison dynamics, we deliberately chose a four-zone classification to ensure adolescent-specific risk calibration. For instance, we conceptualized intermediate classifications like ‘Zone 3: Allow – Suppress from Recommendations’ as the blunt removal of content for adolescents would deter help-seeking and prevent adolescents from accessing recovery-oriented content, in line with the Papageno effect research that affirms the value of constructive storytelling, like stories of hope and recovery, as a protective mechanism against suicide (however the value of recovery content is unproven for lower age bands, which we discuss in the next step). Creating intermediate enforcement actions, like suppressing content from algorithmic feeds, allowed content to be preserved while limiting peer-network exposure to potentially triggering content.
Step Two: Set The Age Thresholds
Next, we set age thresholds by creating two policies: one for 13-15 year olds and another for 13-17 year olds. However, setting the thresholds for the two age bands was a difficult process, as each content category was its own individually reasoned decision tied to developmental claims.
For example, a novel content category for younger age bands was peer crisis management, where teens attempt to counsel user(s) expressing active distress of suicidal ideation. Under the 13-15 year old policy, the content was categorized as Zone 1, as younger users lack the emotional regulation skills or clinical knowledge to navigate an active crisis safely, which would put the user and their peers at risk.
When translating these judgements into the 13-17 year old policy, we reassessed each category independently (rather than treating the policy as a graduated version of the younger ageband). Some categories didn’t move at all (ie, safe messaging-compliant education on suicide was always ‘Zone 4: Always Keep’), others shifted zones (ie, passive ideation moved from ‘Zone 2: Suppress + Sensitive Content Warning’ to ‘Zone 3: Suppress’ for the older age band), and some categories jumped two zones from complete removal (ie, active ideation jumped from ‘Zone 1: Always Remove’ → ‘Zone 2: Suppress + Sensitive Content Warning’).
The most consequential decisions were the two-zone shifts, as they were areas where the policy diverged most sharply between age bands. In these cases, the decisions were informed by both the available research and a precautionary judgment that younger adolescents warranted a higher degree of protection, even where the underlying evidence was incomplete rather than definitive. This was especially relevant for categories like active ideation, as research on the Papageno effect is unproven for younger adolescents. Faced with that uncertainty, we were willing to accept a greater risk of over-removal in exchange for reducing the possibility of exposing younger adolescents to content that could be misunderstood, misapplied, or experienced differently than intended.
(NOTE: While practitioners may argue that removing recovery-oriented content risks limiting access to helpful narratives, we concluded that a precautionary approach was warranted in areas where the evidence remained uncertain. However, for older teens, where developmental maturity provided greater confidence, we adopted less restrictive interventions that preserved access while reducing amplification and exposure through recommendation systems.)
Step Three: Build The Ground Truth
To complement the classification framework, Vys also created a golden dataset, which is a collection of human-annotated data. To do this, we labeled more than 1,100 pieces of content against both the 13-17 and narrower 13-15 policies. We chose to deliberately oversample ‘Zone 1: Always Remove’ cases so we had enough positive examples to validate against (given that removal content is rare in the wild, so naive sampling would leave too few). The dataset ultimately served as a master key for training and calibrating the model, enabling the system to capture edge cases that standardized frameworks may overlook. In sum, this framework forms the source of ground truth upon which our content policy is built.
[What It Takes] To Write LLM-Friendly Content Moderation Policy
Writing LLM-friendly content moderation policy relies on interpretive flexibility, especially in sensitive domains like teen self-harm and suicidal ideation. However, in this context, machine execution removes much of that elasticity in exchange for consistency, through precise language, explicit defaults, and clearly bounded outcomes. By translating the teen SSI policy into machine-readable text, our output aims to provide companies with more effective policy text: precise enough to survive machine interpretation, compact enough to hold together under real enforcement conditions, and open enough to ensure revisions as the environment changes. The SSI policy, developed with Zentropi, is working proof of that thesis.