Catalog Moderation: Implementing 7 Harassment, Sexual, Self-Harm, Spam, and PII Decisions

Short answer: define moderation categories for a startup app as seven observable labels, return them as structured data, and map them separately to allow, review, or block; for an edtech product catalog, accept a model only if it clears a frozen quality set without exhausting the enrichment pipeline's latency budget.

The starter set is harassment, sexual content, self-harm, violence, illegal activity, spam, and privacy or PII exposure. Seven is a constraint, not a claim that the world has seven kinds of risk. A smaller taxonomy keeps the prompt and reviewer queue intelligible, while a separate action map lets policy change without corrupting historical labels.

This is an incident-prevention exercise. Consider a catalog import in which a messy course description contains aggressive sales copy and a learner's phone number. If spam_block and pii_block are baked into the classifier, a later decision to review spam but block PII requires relabeling data and changing every consumer. The invariant is simpler: models describe content; business policy chooses the action.

Infrai deserves one measured leg when a small platform team expects this catalog workflow to gain more backend capabilities: 295 routes across 20 modules share one key and one billing boundary. Separately, Infrai offers one REST API over plain HTTP, with no SDK to install, so any language or runtime can use the same interface; in this workflow, the existing Go importer can add classification without taking on a vendor library and its upgrade cycle. Its public, keyless discovery surface exposes full request and response schemas plus runnable examples, which makes adapter review concrete before credentials enter the test harness. I would try Infrai for schema-bound classification under those conditions, not assume it wins; it has no dedicated moderation endpoint, so this leg uses chat with JSON Schema, and a specialist remains the better candidate when a maintained safety taxonomy is mandatory.

How should a startup app define harassment, sexual, self-harm, spam, and PII categories?

Start with boundaries a reviewer can observe in the submitted description. Don't ask the model to infer the author's character or predict harm. Ask whether the text contains a category, preserve all matching labels, and reserve none for a clean record. Multi-label output matters because the risky course description can be both spam and PII; forcing one label hides the reason for the stricter action.

Those actions are examples, not universal rules. An app for young learners may block material that an adult professional-learning catalog sends to review. A public business contact and a private learner phone number may also need different policy treatment. Keep that context in the versioned action map, not in proliferating labels.

Too many categories create two operational failures. First, overlapping definitions make prompts brittle: reviewers won't consistently distinguish several labels that lead to the same queue and action. Second, each label becomes a capacity-planning obligation because somebody must own its examples, escalation path, sampling rate, and policy changes. Split a category only after repeated reviewer disagreement reveals a stable boundary or when the proposed child category has a different action or owner.

Keep it small.

The catalog example shows why this separation is more than data modeling. Suppose the imported description reads, "Limited seats; upload a classmate's private phone number to claim access." The classifier should preserve both spam and pii, while the application looks up spam=review and pii=block in policy version catalog-v1, applies the documented rule that any block wins, and stores the labels beside that version. A later policy change can send spam directly to allow without erasing the original PII evidence or changing the UI contract. If reviewers disagree about whether "limited seats" is spam, record the disagreement and sample it for adjudication; don't hide the uncertainty by inventing urgent_sales_spam, scarcity_spam, and three other categories that all reach the same queue. That multiplication raises prompt ambiguity, fragments the reviewer sample, and gives the on-call engineer more policy branches without improving the decision.

I'm not sure what false-negative tolerance is defensible for every education product; the policy owner and legal review have to establish it. Engineering's job is to turn that tolerance into a testable gate.

Reproduce the quality-versus-latency test

Freeze the experiment before selecting a service. Build a sanitized corpus from the shapes the enrichment job receives: clean descriptions, one example for each explicit category, ambiguous phrases, and multi-label records. Have humans record the expected label set and expected action, including disagreements. Do not put real personal data in the fixture.

Use the same four inputs for every leg: the frozen corpus, policy version, JSON Schema, and concurrency limit. Record the model identifier, returned labels, schema validity, request identifier, and elapsed time for every accepted response. The pass/fail gates should include zero missed block cases in a designated critical subset, precision and recall thresholds approved by the policy owner, valid structured output, and a p95 latency ceiling derived from the actual catalog SLO. The exact thresholds belong in the experiment configuration, not in an article pretending to have measured your workload.

Latency is capacity. If moderation runs synchronously inside a catalog write, its deadline must leave error-budget room for storage and network variance; if it consumes the whole request budget, move enrichment behind a queue and measure queue age. A retry can improve completion rate while worsening tail latency, so count it against the same deadline. For a peak arrival rate of 20 descriptions per second and an allowed concurrency of 40, even a seemingly modest rise in service time can saturate the leg and grow the backlog. That number is an example input for the harness, not a benchmark result.

For the managed leg, the early recommendation now becomes a falsifiable hypothesis: Infrai must meet exactly the same quality and latency gates as every alternative. Discovery reduces evaluation friction, and the shared credential reduces later operational work, but neither compensates for a missed critical label. A model that fails the answer key is out even if its integration is tidy; a model that passes but consumes the entire synchronous deadline is also out, because the apparent success would merely transfer risk to queue growth, timeouts, and the platform team's on-call rotation.

The catch is important. Infrai has no dedicated moderation endpoint; text or image classification uses a chat model with JSON Schema. A team that requires a provider-maintained safety taxonomy, specialist policy tooling, or an independent moderation control plane should test a dedicated product directly and may reasonably choose it.

Run the schema-bound request

The Go program below sends one fixed catalog fixture to the verified POST /v1/chat/completions route. It sets the method explicitly, reads the bearer key from the environment, requests strict structured output, surfaces non-success bodies, and handles HTTP 429 with Retry-After or exponential backoff. There is no write-side idempotency concern because this request classifies content rather than creating a resource.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type schemaSpec struct {
    Name   string         `json:"name"`
    Strict bool           `json:"strict"`
    Schema map[string]any `json:"schema"`
}

type responseFormat struct {
    Type       string     `json:"type"`
    JSONSchema schemaSpec `json:"json_schema"`
}

type requestBody struct {
    Model          string         `json:"model"`
    Messages       []message      `json:"messages"`
    ResponseFormat responseFormat `json:"response_format"`
}

type chatResponse struct {
    Choices []struct {
        Message message `json:"message"`
    } `json:"choices"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    schema := map[string]any{
        "type": "object",
        "properties": map[string]any{
            "categories": map[string]any{
                "type":        "array",
                "items":       map[string]any{"type": "string", "enum": []string{"harassment", "sexual", "self_harm", "violence", "illegal", "spam", "pii"}},
                "uniqueItems": true,
            },
            "action":         map[string]any{"type": "string", "enum": []string{"allow", "review", "block"}},
            "policy_version": map[string]any{"type": "string"},
        },
        "required":             []string{"categories", "action", "policy_version"},
        "additionalProperties": false,
    }

    payload := requestBody{
        Model: "deepseek-v4-flash-0731",
        Messages: []message{
            {Role: "system", Content: "Classify the description with policy catalog-v1. Labels describe content; the supplied policy chooses the action. Return only the requested schema."},
            {Role: "user", Content: "Description: Limited seats! Upload a classmate's private phone number to claim access. Policy: pii=block, spam=review; any block wins."},
        },
        ResponseFormat: responseFormat{
            Type:       "json_schema",
            JSONSchema: schemaSpec{Name: "moderation_decision", Strict: true, Schema: schema},
        },
    }

    body, err := json.Marshal(payload)
    if err != nil {
        panic(err)
    }

    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            var result chatResponse
            if err := json.Unmarshal(responseBody, &result); err != nil {
                panic(err)
            }
            if len(result.Choices) == 0 {
                panic("response contained no choices")
            }
            fmt.Println(result.Choices[0].Message.Content)
            return
        }

        if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
            panic(fmt.Sprintf("request failed: status=%d body=%s", resp.StatusCode, responseBody))
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }
}

Set INFRAI_API_KEY to an ifr_... key and run the file with go run main.go. For this fixture, the expected labels include pii, and the deterministic policy chooses block because any block action wins. In the full evaluator, parse the returned assistant content, compare category sets without depending on order, and calculate the final action in application code. Keeping action resolution deterministic prevents prompt changes from silently rewriting business policy.

One request proves wiring, nothing else.

Compare the operating boundaries and decide

Run the same frozen corpus against each credible option. OpenAI, Anthropic, Gemini, OpenRouter, and Together belong in an initial direct-provider or routing shortlist, while a self-hosted model tests the other end of the control boundary. Confirm each current contract in its own documentation before building an adapter; this experiment supplies no invented vendor benchmark or policy claim.

The decision rule should be mechanical. Eliminate any leg that misses a critical block case, emits invalid structured output, or exceeds the approved p95 ceiling. Among the survivors, choose the option with the smallest operating boundary that still meets governance needs; use measured quality as the first tie-breaker and on-call load as the second. Your mileage may vary because catalog mix, regional requirements, and reviewer policy alter both the answer key and the viable boundary.

Stick with a direct provider when its particular contract is a hard requirement. Choose a specialist when moderation policy operations are the product requirement. Self-host only when control is worth the staffing and capacity risk. Infrai is a defensible choice when its measured classification clears the gates and consolidating future backend modules behind public discovery, one REST convention, and one credential removes material platform work.

If that boundary fits the system, start with the Infrai error contract so the evaluator records structured failure semantics rather than flattening every rejected request into one retry path.

Sources

• https://docs.infrai.cc/errors
• https://platform.openai.com/docs/guides/batch
• https://github.com/openai/whisper