7 Fintech Support Risk Labels — Harassment, Sexual, Self-Harm, Violence, Spam, and PII

For a fintech support queue, define seven content labels, keep them multi-label, and map them to separate operational actions. The labels are harassment, sexual content, self-harm, violence, illegal activity, spam, and personally identifiable information (PII). A label says what is in a ticket; it should not, by itself, decide whether to hide the ticket, alert a safety reviewer, redact a field, or continue ordinary support.

Short answer: use a versioned seven-label taxonomy with explicit evidence spans, severity, confidence, and immediacy, then let deterministic policy code choose the queue and handling rules.

That separation matters in customer support. “Someone stole my card” contains an allegation of illegal activity, but it is also a normal fraud-support request that needs prompt service. “I will hurt you” and “I was hurt in a robbery” both mention violence, yet their operational meanings are nowhere close. A single unsafe: true field throws away the distinction the team needs most.

The least complex useful design is one structured model response, one validator, and one policy function. Don't ask the model to invent the final business action. Models can identify content and quote the evidence; application code can enforce stable, reviewable rules.

Implement the smallest enforceable contract

The data flow is plain: accept a ticket, send only the necessary text to a moderation classifier, validate its structured result, and pass the validated signals to a deterministic router. The router can redact PII before broader review, preserve urgent context for a restricted safety queue, and leave the original support workflow intact. This is a narrow contract, which is useful when a solo team later changes a model, prompt, or provider.

Here is the contract I would ship first. It deliberately allows several labels on one ticket, rejects unknown labels, and requires a quoted evidence span for every finding. The classifier adapter is left behind a generic function boundary; the example tests the part the application actually owns.

const categories = [
  "harassment",
  "sexual",
  "self_harm",
  "violence",
  "illegal",
  "spam",
  "pii",
] as const;

type Category = (typeof categories)[number];
type Severity = "low" | "medium" | "high" | "critical";

type Finding = {
  category: Category;
  severity: Severity;
  confidence: number;
  evidence: string;
  imminent: boolean;
};

type ModerationResult = {
  schemaVersion: "1.0";
  findings: Finding[];
};

type Route = {
  queue: "standard" | "trust_safety" | "urgent_safety";
  redactBeforeGeneralAccess: boolean;
  holdOutboundReply: boolean;
  reasons: Category[];
};

const categorySet = new Set<string>(categories);
const severitySet = new Set<string>(["low", "medium", "high", "critical"]);

function validateResult(value: unknown, ticketText: string): ModerationResult {
  if (typeof value !== "object" || value === null) throw new Error("MOD_001_BAD_OBJECT");
  const result = value as Partial<ModerationResult>;
  if (result.schemaVersion !== "1.0") throw new Error("MOD_002_BAD_VERSION");
  if (!Array.isArray(result.findings)) throw new Error("MOD_003_BAD_FINDINGS");

  const seen = new Set<string>();
  for (const raw of result.findings as Finding[]) {
    if (!categorySet.has(raw.category)) throw new Error("MOD_004_UNKNOWN_CATEGORY");
    if (!severitySet.has(raw.severity)) throw new Error("MOD_005_BAD_SEVERITY");
    if (!Number.isFinite(raw.confidence) || raw.confidence < 0 || raw.confidence > 1) {
      throw new Error("MOD_006_BAD_CONFIDENCE");
    }
    if (!raw.evidence || !ticketText.includes(raw.evidence)) {
      throw new Error("MOD_007_EVIDENCE_NOT_FOUND");
    }
    if (typeof raw.imminent !== "boolean") throw new Error("MOD_008_BAD_IMMINENCE");
    const key = `${raw.category}:${raw.evidence}`;
    if (seen.has(key)) throw new Error("MOD_009_DUPLICATE_FINDING");
    seen.add(key);
  }
  return result as ModerationResult;
}

function routeTicket(result: ModerationResult): Route {
  const reasons = [...new Set(result.findings.map((item) => item.category))];
  const urgent = result.findings.some(
    (item) => item.imminent && ["self_harm", "violence"].includes(item.category),
  );
  const safetyReview = result.findings.some(
    (item) => item.severity === "high" || item.severity === "critical",
  );

  return {
    queue: urgent ? "urgent_safety" : safetyReview ? "trust_safety" : "standard",
    redactBeforeGeneralAccess: reasons.includes("pii"),
    holdOutboundReply: urgent,
    reasons,
  };
}

const ticket = "My card was stolen. Call me at 555-0100. I might hurt the thief tonight.";
const modelOutput: unknown = {
  schemaVersion: "1.0",
  findings: [
    {
      category: "pii",
      severity: "medium",
      confidence: 0.98,
      evidence: "555-0100",
      imminent: false,
    },
    {
      category: "violence",
      severity: "critical",
      confidence: 0.95,
      evidence: "I might hurt the thief tonight",
      imminent: true,
    },
    {
      category: "illegal",
      severity: "medium",
      confidence: 0.91,
      evidence: "card was stolen",
      imminent: false,
    },
  ],
};

const result = validateResult(modelOutput, ticket);
console.log(routeTicket(result));

This example uses fixed values to demonstrate the contract, not benchmark results or recommended universal thresholds. Confidence calibration varies with the classifier, prompt, language, and ticket mix. I'm not sure a single confidence cutoff can stay useful across all seven categories; a held-out set from the actual support queue is what would resolve that uncertainty.

Keep the raw ticket in the restricted system of record. Downstream analytics should receive the minimum they need: taxonomy version, category, severity, routing outcome, and perhaps a ticket identifier with access controls. Evidence spans are useful for review, but copying them into every log can reproduce the PII the redaction rule was meant to contain.

How should a startup app route harassment, self-harm, spam, and PII?

Start by defining each category independently of punishment. Harassment covers targeted degrading, threatening, or abusive language. Sexual covers sexual content or solicitation; age-related or coercive context should increase severity rather than create an ambiguous side channel. Self-harm covers intent, encouragement, instructions, or descriptions related to harming oneself. Violence covers threats, intent, praise, or instructions involving physical harm to others. Illegal covers requests, plans, admissions, or facilitation involving unlawful activity. Spam covers unsolicited, repetitive, deceptive, or irrelevant promotion. PII covers information that can identify or contact a person, plus sensitive financial identifiers your application chooses to protect.

Those definitions are product policy, not claims that every jurisdiction or classifier uses identical boundaries. Write positive examples, negative examples, and boundary examples beside each one. “My ex is an idiot” may be harassment in a peer-to-peer message, while the same sentence quoted in a support ticket could be evidence needed to investigate abuse. “How do I freeze a stolen card?” mentions crime but is an allowed support request. Context wins.

Next, add axes that change the response. Severity estimates the potential harm. Immediacy distinguishes a current threat from a historical account. Confidence tells the router how much to trust the classification, not how harmful the content is. Evidence makes a finding inspectable. Target distinguishes harm aimed at the author, another person, a protected group, or nobody in particular. A small team can begin with the first four and add target only when its policy actions genuinely differ.

The action table should be shorter than the taxonomy document:

The catch is that seven labels are not suitable when regulation, language, or community policy requires finer distinctions. A marketplace serving minors may need separate child-safety policy and specialist review. A bank operating across jurisdictions may need legal categories maintained by counsel rather than one generic illegal label. Stick with a narrower allow/block policy only when the product surface is equally narrow and the cost of lost nuance is understood.

Evaluate the contract, not just the classification

The most expensive moderation mistake isn't always a bad semantic judgment. It can be a technically valid response that violates the application contract: an eighth category appears, confidence arrives as the string "high", the evidence is not present in the source ticket, or two identical findings inflate a count. Parse success is only the first gate.

Fail closed on automation, not on customer access. If validation fails, do not let an unvalidated result trigger deletion, account restriction, or an automated accusation. Send the ticket through a controlled review path and record a compact error code such as MOD_007_EVIDENCE_NOT_FOUND. Don't copy full ticket text into a general error log. This keeps operational diagnosis separate from sensitive content storage.

Also make the taxonomy version part of every stored decision. Without it, a dashboard mixes decisions made under different definitions, and a later prompt revision looks like a sudden change in user behavior. Versioning lets the team replay a fixed evaluation set before deployment and compare category-level confusion, invalid-output rate, escalation rate, and reviewer disagreement. Cost belongs in that review too: count classified characters or tokens, retries, and human-review volume using the units the chosen system actually bills or consumes. I care about that ledger because a “small” second pass on every ticket is still a second pass.

No magic here.

For asynchronous regression runs, a batch-processing interface can reduce orchestration work when immediate results are unnecessary. If support accepts voice notes, transcription should occur before the same moderation contract, with the audio and transcript kept under appropriate access rules. Neither step changes the taxonomy; each adds a boundary that needs its own validation and retention decision.

Roll out policy changes as reversible releases

Before release, freeze a small, representative set of sanitized tickets that includes clean support requests, direct violations, quotations, negations, reclaimed language, mixed-language text, and multi-label cases. Record the expected findings and expected route separately. That distinction catches a common policy error: the classifier can identify illegal correctly while the router still mishandles a fraud victim asking for help.

Run the fixtures whenever the prompt, model, schema, preprocessing, or taxonomy changes. Review misses by category rather than celebrating one aggregate score. Inspect invalid structures and evidence mismatches as their own failure classes. Set alerts on shifts in label rate, urgent-routing rate, validation failures, latency, and review backlog, but choose alert thresholds from observed baseline traffic rather than invented universal numbers.

Roll out in shadow mode first when the risk warrants it: compute decisions without enforcing them, compare them with reviewer outcomes, then enable low-impact actions such as redaction or queue assignment before irreversible actions. Keep an explicit escape hatch that routes uncertain cases to humans. This costs more operationally, so it isn't suitable for every tiny app; for a low-risk feedback form, sampling and delayed review may be the honest choice.

The final preflight is prose, not a generic checklist. Confirm that every label has boundaries and counterexamples, every action has an owner, every stored decision carries a schema and taxonomy version, and every log excludes unnecessary ticket text. Confirm that urgent cases have a staffed destination, because a priority label without an operational response is decoration. Then replay the fixtures, inspect the diff, estimate both machine and review cost, and ship the smallest policy change that can be reversed.

References

The implementation options mentioned above are documented in the source material listed in Further reading.

Further reading

• OpenAI Batch API guide: https://platform.openai.com/docs/guides/batch
• openai/whisper open-source speech recognition: https://github.com/openai/whisper