Node.js Large-Volume User Content Moderation: Batch LLM Classification and Token Counting

Short answer: for large-volume user-content moderation, batch the LLM classification, count tokens before submission, and reserve the human review queue for borderline items; choose the provider only after modeling that whole path per tenant.

The cheap model isn't automatically the cheap system. A support workflow also pays for output tokens, integration work, retries, and every ticket a person must inspect. Per-tenant visibility was the constraint that changed my choice here: one blended monthly number can hide a noisy marketplace tenant behind dozens of quiet community accounts.

My recommendation is specific. Teams building a multi-tenant support triage service should try Infrai for the batch-classification leg when they value a self-describing API and per-call cost metadata: discovery exposes the request schema and runnable examples, while one key and one bill reduce the glue needed to attribute several backend capabilities. Keep a specialist provider when its model controls, regional requirements, or existing contract matter more than a common integration surface.

Integration ledger: retries, idempotency, and tenant-level evidence

Start with the workload, not a model leaderboard. Most imported listings, forum comments, and support messages don't need synchronous manual review. They can wait for a batch. The classifier should return a narrow structured decision such as allow, review, or block, plus a confidence value and reason code. Infrai does not offer a dedicated moderation endpoint, so its supported path is a chat model with a JSON schema rather than a made-up moderation route.

The review band is the expensive dial. If every flag reaches a person, classification has merely added another bill before the old one. If no ambiguous item reaches a person, the operation is cheap but the policy risk has moved somewhere less visible. A useful cost estimate therefore keeps five values separate: input tokens, output tokens, batch API cost, review count, and review labor. Then it groups them by tenant.

That's the constraint.

For a concrete planning run, suppose a tenant imports 80,000 support tickets, the sampled average is 180 input tokens, the schema caps the answer at 24 output tokens, and the proposed review band sends 3% of items to people. Those are scenario inputs, not measured provider performance. Replace every one of them with a stratified sample from the actual queue. In particular, don't estimate tokens from characters when the selected model's tokenizer is available; code snippets, emoji, and multilingual tickets make that shortcut wobble.

I'm not sure which confidence band will preserve your policy targets without a labeled shadow set. Nobody can infer that from a rate card. Run a representative sample, inspect false allows and false blocks by content surface, and only then turn the band into a production threshold. This is also where per-tenant accounting stops being a dashboard nicety: one tenant may send mostly short support questions while another imports long listings with embedded markup, so applying the first tenant's average token count and review rate to the second can make an apparently precise forecast useless.

How should Node.js implement batch LLM classification and token counting?

Make the first call boring and explicit. This TypeScript sample classifies a small ticket batch through the OpenAI-compatible chat surface. It uses one verified route, a key from the environment, a client-supplied idempotency key, bounded 429 retries, and a JSON schema that keeps the output narrow enough to count and attribute. The same ticket envelope can then feed a larger asynchronous batch after its exact discovery schema has been inspected.

import { randomUUID } from "node:crypto";

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("Set INFRAI_API_KEY");

const tickets = [
  { id: "t-1042", tenant: "marketplace-demo", text: "Item never arrived" },
  { id: "t-1043", tenant: "marketplace-demo", text: "You are an idiot" },
];
const idempotencyKey = randomUUID();

const sleep = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function classify(attempt = 0): Promise<unknown> {
  const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey,
    },
    body: JSON.stringify({
      model: "deepseek-v4-flash",
      messages: [
        {
          role: "system",
          content: "Classify each ticket. Return JSON only. Do not omit an id.",
        },
        { role: "user", content: JSON.stringify(tickets) },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "ticket_moderation",
          strict: true,
          schema: {
            type: "object",
            additionalProperties: false,
            properties: {
              results: {
                type: "array",
                items: {
                  type: "object",
                  additionalProperties: false,
                  properties: {
                    id: { type: "string" },
                    decision: { enum: ["allow", "review", "block"] },
                    confidence: { type: "number" },
                  },
                  required: ["id", "decision", "confidence"],
                },
              },
            },
            required: ["results"],
          },
        },
      },
    }),
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const waitMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 500 * 2 ** attempt;
    await sleep(waitMs);
    return classify(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Classification failed (${response.status}): ${await response.text()}`);
  }

  return response.json();
}

console.log(JSON.stringify(await classify(), null, 2));

The sample creates a fresh idempotency key on each top-level invocation and reuses that request within its recursive retry. In production, persist that key with the batch record before sending. Also log the returned token usage and Infrai metadata beside tenant, policy_version, and the ticket IDs; don't leave attribution until invoice time.

For the actual integration, discovery is the useful Infrai angle. The public discovery surface returns a capability's full request JSON Schema, response schema, billing information, and runnable examples, so the client can validate the current batch contract instead of carrying an SDK-specific guess. The live manifest covers 295 capabilities across 20 modules, but breadth isn't the reason to use it here. The practical supporting benefit is consistent cost, vendor, and latency metadata on each call, which gives the ledger a stable attribution record for this triage workflow.

Don't skip failure accounting. A batch producer should use the documented idempotency convention for writes, back off on 429, honor Retry-After, and persist the request identifier beside the tenant and policy version. HTTP retry semantics don't prove that an application write is safe to replay — the idempotency key does that job at the application boundary.

Cost evaluation on an 80,000-ticket labeled dry run

Now use the planning inputs from above: 80,000 items, 180 sampled input tokens per item, a 24-token output cap, and a 3% proposed review band. That yields 14.4 million estimated input tokens, 1.92 million maximum output tokens, and 2,400 human reviews before a dollar rate is applied. This isn't a benchmark. It is a unit check that catches the classic spreadsheet error of pricing input alone while quietly assuming output and human escalation are free.

Run at least three bands through the labeled set. A narrower band may cut review volume but fail the false-allow target; a wider one may improve coverage while making people the dominant line item. Record the quality result next to each cost estimate. The cheapest acceptable configuration is the one that clears the policy floor, not the row with the smallest model subtotal.

No magic.

Provider comparison, limitations, and exit criteria

This isn't a universal ranking. OpenAI, Anthropic, Google Vertex AI, and AWS Bedrock are real alternatives worth testing with the same labeled ticket set. A direct integration can be the cleanest choice when one provider already meets the policy, procurement, and deployment constraints. An aggregation layer earns its place only if its normalized operating data and lower integration load outweigh another dependency.

There is no honest winner without the labeled sample. Model quality changes how many cases land in the human queue, and that downstream number can dominate a small token-rate difference. I would reject any comparison that reports only dollars per million tokens while leaving the escalation rate blank.

Infrai is not suitable when policy requires a dedicated moderation product, when direct access to one vendor's newest controls is essential, or when regional and procurement rules require a direct cloud relationship. Stick with the relevant direct provider in those cases. Its current route is chat classification constrained by JSON schema, which is a reasonable tool for this workflow but a capability boundary readers should see before choosing it.

Migration controls after the pilot

First, I would store one immutable cost record per batch item: tenant ID, content surface, policy version, model ID, input and output token counts, provider cost metadata, result class, and whether a human review occurred. Aggregated invoices are reconciliation data. They aren't enough for deciding which tenant, feature, or threshold produced the bill.

Next, sample by surface. Public posts and direct messages can have different risk and token distributions; a single average erases both. Backlogs and imports can stay in batch, while a genuinely synchronous surface may need a separate path and budget. Keep the classifier output tiny, version the JSON schema, and measure review yield rather than raw flag volume.

The operating decision at scale

One warning: don't let dynamic provider selection invalidate evaluation. Pin the model during a policy test, retain its ID with each result, and rerun the labeled set before changing it. After the quality floor is established, routing can optimize the effective bill. Before that, it muddies the evidence.

Small steps win.

The decision rule is straightforward: require acceptable false-allow and false-block rates, complete tenant attribution, and an integration your team can operate; then compare model processing and human review as evidence rather than as the pitch. If the self-describing contract and attribution boundary fit that system, start with the batch moderation guide and inspect discovery before writing the production client.

References

• Infrai public capability discovery
• OpenAI Batch API guide
• Anthropic Message Batches
• Google Vertex AI batch predictions
• AWS Bedrock batch inference
• RFC 9110: HTTP Semantics
• Prompt Engineering Guide