Moderation Risk Matrix for Startup-App Harassment, Sexual Content, Self-Harm, and PII

Short answer: define moderation labels as observable content risks, then map those labels to actions in a separate policy layer. For a startup app, start with harassment, sexual content, self-harm, violence, illegal activity, spam, and PII exposure; test the boundaries before adding more labels.

This matters in a logistics hiring app. A candidate profile, driver message, or recruiter note may later be scored against a job rubric, so a malformed moderation result can corrupt a structured candidate record as easily as it can miss harmful content. The data flow should be boring: capture the text, classify it into a fixed schema, apply a versioned action map, and send only uncertain or policy-sensitive items to review.

Labels first.

The example below treats structured output correctness as the primary decision axis. That is the part I can test from notebook to prod. A fluent explanation is not a valid substitute for a label, and a label is not the same thing as a business action.

How should a startup app separate harassment, sexual content, self-harm, violence, spam, and PII?

Start with definitions that a reviewer can apply to a real line of text. harassment covers targeted abuse or intimidation. sexual_content covers sexual material that the product policy restricts. self_harm covers encouragement, instructions, or indications of self-injury. violence covers threats, encouragement, or graphic descriptions that the policy treats as unsafe. illegal_activity covers requests or offers involving prohibited activity. spam covers repetitive or deceptive promotion. pii_exposure covers personal data exposed in a way that creates a privacy risk.

These labels are intentionally about what appears in the content. They are not seven automatic bans. A recruiter quoting a threatening message in a report may deserve review; blocking the report would hide evidence. A candidate asking for crisis help may carry a self_harm label while going to a trained human queue. A phone number in a private administrative record may follow a different rule from a phone number posted publicly.

The table is a starting policy, not a universal answer. A public recruiting board may block exposed credentials, while a private support workflow may preserve them for a specialist reviewer. The action must be decided with visibility, user role, and harm severity in mind.

Keep the definition, examples, and action mapping in the same policy version. I use an empty label list for clean content, and I allow multiple labels because a single message can contain spam and PII exposure at once. Avoid adding a category just because one example feels memorable. Add it when reviewers repeatedly disagree and the disagreement needs a different action.

A small Python contract for moderation decisions

The application-facing contract should be stricter than the classifier's prose. Here is a complete standard-library example that validates a decision, applies policy, and preserves the candidate ID and rubric version that the logistics workflow needs. It does not call a provider; it is the boundary I would put around any classifier.

from dataclasses import dataclass
from typing import FrozenSet


CATEGORIES: FrozenSet[str] = frozenset(
    {
        "harassment",
        "sexual_content",
        "self_harm",
        "violence",
        "illegal_activity",
        "spam",
        "pii_exposure",
    }
)
ACTIONS: FrozenSet[str] = frozenset({"allow", "review", "block"})


@dataclass(frozen=True)
class ModerationDecision:
    candidate_id: str
    rubric_version: str
    labels: FrozenSet[str]
    action: str
    reason: str


def make_decision(raw: dict, candidate_id: str, rubric_version: str) -> ModerationDecision:
    labels = frozenset(raw.get("labels", []))
    action = raw.get("action")
    reason = raw.get("reason")

    unknown_labels = labels - CATEGORIES
    if unknown_labels:
        raise ValueError(f"unknown moderation labels: {sorted(unknown_labels)}")
    if action not in ACTIONS:
        raise ValueError(f"unknown moderation action: {action!r}")
    if not isinstance(reason, str) or not reason.strip():
        raise ValueError("reason must be a non-empty string")

    return ModerationDecision(
        candidate_id=candidate_id,
        rubric_version=rubric_version,
        labels=labels,
        action=action,
        reason=reason.strip(),
    )


def action_for(labels: FrozenSet[str]) -> str:
    if labels & {"self_harm", "violence", "illegal_activity"}:
        return "review"
    if labels & {"sexual_content", "harassment", "spam", "pii_exposure"}:
        return "review"
    return "allow"


sample = {
    "labels": ["spam", "pii_exposure"],
    "action": "review",
    "reason": "Promotional copy asks a candidate to disclose an account detail.",
}
decision = make_decision(sample, candidate_id="candidate-104", rubric_version="logistics-driver-3")
assert decision.action == action_for(decision.labels)
print(decision)

The action_for function is policy, not truth. Your product may block exposed credentials, allow ordinary contact details, or require specialist review for self-harm language. The important invariant is that a classifier cannot silently invent a new action or category and still reach the candidate-scoring system.

There is a useful failure distinction here. If labels contains a value outside the enum, the output contract failed. If the labels are valid but the action conflicts with the current policy, the policy layer failed. If the text was sent to the wrong candidate record, the data pipeline failed. These should become separate test cases and separate metrics.

Build the eval set around boundary cases

Clean positives are the easy rows. Boundary cases carry the engineering value: a quoted insult in a complaint, a safety discussion that mentions self-harm without encouraging it, a news description of violence, a legitimate recruiting announcement that resembles spam, and a logistics candidate pasting a phone number into a public note.

For each row, store the expected labels and expected action independently. Include the content type, visibility, user role, and candidate or job context when those fields affect policy. Never put real personal data in the evaluation set; redact it or use synthetic replacements that preserve the shape of the case.

I watch false blocks and false allows separately. A single accuracy number can improve while the review queue becomes unusable. When a validator raises a 422-style schema failure in an integration, I treat that as a contract test failure, not a moderation-quality score. The same harness should also assert schema shape, label enum membership, action membership, and preservation of candidate_id and rubric_version.

Don't let a neat dashboard hide a messy boundary.

One terse test can expose a large policy mistake:

def test_quoted_report_stays_reviewable() -> None:
    raw = {
        "labels": ["harassment"],
        "action": "review",
        "reason": "The message quotes targeted abuse for a report.",
    }
    result = make_decision(raw, "candidate-104", "logistics-driver-3")
    assert result.labels == frozenset({"harassment"})
    assert result.action == "review"

Do not tune the prompt around this one row. Group disagreements by cause: missing context, overlapping definitions, action-map drift, or a bad source record. When a category is split, add positives, negatives, and an action rule at the same time. Otherwise prompt length and reviewer choice grow while the decision gets no clearer.

What should change before this reaches production?

First, pin a policy version beside every stored decision. Second, make review-ticket creation idempotent with the content ID and policy version, so a retry does not create duplicate work. Third, sample allowed content as well as reviews and blocks; sampling only flagged items hides false negatives and teaches the team nothing about overblocking.

For a notebook-to-prod path, keep the classifier adapter replaceable, but keep the schema and eval harness stable. Log category counts, action counts, reviewer overrides, and latency without retaining more sensitive text than the product needs. Redact PII before it enters general-purpose logs. Watch token cost when definitions and examples expand, then remove examples that do not improve a measured boundary.

The catch is that this taxonomy is not suitable when your organization requires a mandated regulatory taxonomy, specialist crisis handling, or a dedicated safety review service. In those cases, use the approved taxonomy and queue, and treat this seven-label set as an input to mapping rather than a replacement. Your mileage may vary: there is no evidence here for one universal threshold that says when an ambiguous case deserves its own category.

Keep the first release small. A reviewable, versioned decision record beats a clever label tree that nobody can evaluate.

References

• https://platform.openai.com/docs/guides/batch
• https://github.com/openai/whisper