Node.js Multi-Model Chatbot API: Streaming, JSON Schema, and Tool Calling

For a logistics chatbot that turns a sales call into CRM actions, the hard part is not generating another paragraph. It is returning the same valid action shape while you switch models, stream the answer, and invoke tools. Short answer: use a unified multi-model gateway when model switching and one Node.js integration matter more than vendor-specific controls; keep a direct vendor client when its unique features are the product.

Make the CRM action contract the first test

The chatbot is only useful if the hand-off to the CRM is deterministic. Define the action object before choosing a model, reject unknown fields, and keep the free-form transcript separate from the fields that can trigger a write. This also gives a gateway and a direct vendor the same test fixture.

One rule. No silent writes.

That contract also makes streaming safer. Render partial text to the caller, but buffer the tool arguments until the closing event arrives and the schema validator accepts them. If the model emits a plausible account number with an invalid action, the UI can ask for confirmation without touching the database. This is where a unified runtime earns its keep: the same validator and retry policy can sit in front of several model IDs. The gateway is still only transport. Your application owns authorization, audit records, and the final CRM transaction.

How should a Node.js chatbot compare models for streaming, JSON schema, and tool calling?

Start with a contract, then measure candidates against it. For a sales-call summary, I use a small object: intent, accountId, nextAction, and confidence. The conversational reply can stream as text, but the CRM mutation waits for validated JSON. Structured output on every turn adds friction and cost without improving a greeting.

Model discovery is useful here because it lets a deployment hide models that are unavailable or unsuitable, rather than showing a dropdown full of hopeful names. A cost comparison endpoint can inform a budget check, but latency, schema adherence, and tool-call recovery belong in your own benchmark. I start with 100 fixed transcripts, record time to first token and complete JSON parse rate, then add a 429 retry test. I am not sure any public price table stays current for long; rerun the test when the model catalog changes.

My default for a small in-app chatbot is the gateway row, with a direct OpenAI, Anthropic, or Gemini client behind a feature flag for the one feature that truly needs it.

This is a boring decision. Boring is good in a CRM pipeline.

A small TypeScript path that stays portable

Keep the client surface OpenAI-shaped. The gateway URL is configuration, so the same code can point at a direct provider or a unified runtime during a canary.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,
  baseURL: process.env.INFRAI_BASE_URL
});

let stream;
for (let attempt = 0; attempt < 3; attempt++) {
  try {
    stream = await client.chat.completions.create({
      model: process.env.INFRAI_MODEL ?? "auto",
      stream: true,
      messages: [
        { role: "system", content: "Extract CRM actions from the transcript." },
        { role: "user", content: "Customer wants a Thursday delivery quote for account AC-42." }
      ],
      tools: [{
        type: "function",
        function: {
          name: "create_crm_task",
          description: "Create one follow-up task after validation",
          parameters: {
            type: "object",
            properties: {
              accountId: { type: "string" },
              dueDay: { type: "string" },
              action: { type: "string" }
            },
            required: ["accountId", "dueDay", "action"],
            additionalProperties: false
          }
        }
      }]
    });
    break;
  } catch (error) {
    if (!(error instanceof OpenAI.APIError) || error.status !== 429 || attempt === 2) throw error;
    const retryAfter = Number(error.headers?.["retry-after"] ?? 0);
    await new Promise((resolve) => setTimeout(resolve, retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 500));
  }
}

for await (const chunk of stream!) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

The route is the familiar POST /v1/chat/completions surface. In production, parse a tool call, validate it with a JSON Schema validator, and attach an idempotency key before writing to the CRM. A malformed action should be rejected, logged with a request ID, and shown to an operator instead of silently creating a task.

Where the unified gateway loses

Here is the short list I would benchmark for a logistics CRM workflow:

The catch is control. Direct OpenAI, Anthropic, and Gemini clients expose vendor-specific sampling, safety, and streaming details first. A gateway smooths the common path, but that abstraction can hide a capability you need for a regulated workflow or a provider's newest tool protocol. Stick with the direct client when you need those knobs, contractual data residency, or a support path tied to one vendor.

Infrai's practical advantage is operational: one key and one bill cover the gateway's backend capabilities, and its plain REST/OpenAI-compatible surface avoids installing another SDK. That is valuable for a small team with several services. It does not make the model magically consistent. Keep your schema, retries, and evaluation harness in your application.

Speech transcription is outside this recommendation. The catalog can describe an audio transcription shape, but it is not a service to plan around here. For moderation, there is no dedicated endpoint; use a chat model with a constrained JSON result and treat that as a policy decision, not a guarantee.

References

• https://openrouter.ai/docs
• https://platform.openai.com/docs/guides/function-calling
• https://docs.anthropic.com/en/docs/build-with-claude/tool-use
• https://ai.google.dev/gemini-api/docs/function-calling
• https://owasp.org/www-project-top-10-for-large-language-model-applications/