CloudWatch EMF Explained Simply: How to Emit Zero‑Overhead Custom Metrics from Your AI Node.js Service

Most engineers reach for the CloudWatch Metrics API and end up writing extra SDK calls that add latency and cost. Embedded Metrics Format (EMF) lets you ship rich, query‑able metrics by just writing specially‑formatted JSON logs. Learn how to turn your AI inference code into a self‑monitoring powerhouse with no extra overhead.

Why EMF Matters for AI Services

The problem we’re solving

An LLM (large language model) inference endpoint can handle hundreds of requests per second. Each request has a latency, a token count, and sometimes an error. If you log only the raw request/response, you must later run expensive log‑scans to calculate averages, percentiles, or error rates. Adding a second “metrics” call (e.g., PutMetricData) creates extra network hops, adds milliseconds to every inference, and inflates your AWS bill.

The EMF answer

Embedded Metrics Format (EMF) is a way to embed metric data directly inside a CloudWatch Logs event. Think of a log line as a post‑it note that not only tells you what happened but also carries a tiny scoreboard of numbers. CloudWatch reads the scoreboard, extracts the numbers, and stores them as regular CloudWatch metrics—without any separate API call.

In plain English: Write one JSON log line, get both a log entry and a metric for free.

Why AI teams should care

• Zero added latency – no extra HTTP request, just a console.log.
• Cost‑effective – you pay only for the log storage you already have; you avoid per‑metric‑data‑point charges from PutMetricData.
• Richness – a single log line can contain many dimensions (e.g., model name, request ID) that become dimensions on the metric automatically.
• Observability consistency – the same source that you debug with logs is also the source of your dashboards and alarms.

Embedding Metrics in CloudWatch Logs

The format at a glance

An EMF log entry is a JSON object that contains a top‑level @aws key. Inside @aws you must provide:

• Timestamp – epoch milliseconds.
• CloudWatchMetrics – an array describing each metric you’re publishing.
• Namespace – a logical bucket (e.g., MyAIService) that groups related metrics.

All other top‑level keys become dimensions (labels) or measurements (numeric values) for the metric.

{
  "@aws": {
    "Timestamp": 1725067200000,
    "CloudWatchMetrics": [
      {
        "Namespace": "MyAIService",
        "MetricName": "InferenceLatency",
        "Dimensions": [["ModelName", "Endpoint"]]
      }
    ]
  },
  "ModelName": "gpt‑4‑mini",
  "Endpoint": "text‑completion",
  "InferenceLatency": 124,
  "TokenCount": 57,
  "Success": true
}

• InferenceLatency, TokenCount, and Success are the measurements.
• ModelName and Endpoint are the dimensions that let you break down the metric later.

Gotcha #1 – missing @aws fields

If you forget Timestamp or Namespace, CloudWatch silently drops the EMF data. The log still appears in CloudWatch Logs, but no metric shows up.

Tip: Validate your JSON with a small unit test before shipping it to production.

Gotcha #2 – subscription filter

CloudWatch only parses EMF if the log group has a subscription filter that forwards logs to the CloudWatch Metrics processing pipeline. Without it, the EMF line stays a plain log.

aws logs put-subscription-filter \
  --log-group-name /aws/ecs/my-ai-service \
  --filter-name EMFProcessor \
  --filter-pattern "" \
  --destination-arn arn:aws:logs:us-east-1:123456789012:destination:CloudWatchMetrics

In plain English: Think of the subscription filter as a “mailroom clerk” that reads each envelope (log) and extracts the scorecard (metric). If the clerk isn’t hired, the scorecard never leaves the envelope.

Cost considerations for the surrounding ecosystem

• CloudWatch Logs Insights – each query scans the entire log volume at $0.005 per GB. At high traffic, frequent ad‑hoc queries can become pricey.
• Log retention – by default logs never expire. Set a retention policy (e.g., 30 days) to avoid runaway storage costs.
• Metric resolution – 1‑second granularity costs roughly three times more than 1‑minute. For most AI latency monitoring, 1‑minute is sufficient.
• Alarms on missing data – during a deployment, a new version may temporarily stop emitting a metric. Alarms configured to treat “missing data = breaching” will fire unnecessarily. Set “missing data = ignore” or use a “breaching” alarm only after a warm‑up period.

Node.js 22: Using diagnostics_channel for Automatic EMF Emission

Why diagnostics_channel?

Node.js 22 introduced the diagnostics_channel module as a low‑overhead way for libraries to publish structured data about runtime events. It works like a radio station: the service (your inference code) broadcasts a message, and any listener (the EMF formatter) can pick it up without altering the main execution path.

Setting up a channel

// diagnosticsChannel.ts
import { createChannel } from 'node:diagnostics_channel';

// Create a channel named "ai-inference". The name is arbitrary but must be consistent.
export const inferenceChannel = createChannel('ai-inference');

Whenever an inference finishes, we publish a payload to this channel. The payload will later be turned into an EMF log line.

Listener that writes EMF JSON

// emfLogger.ts
import { inferenceChannel } from './diagnosticsChannel.js';
import { CloudWatchLogsClient, PutLogEventsCommand } from '@aws-sdk/client-cloudwatch-logs';
import { format } from 'util';

// Create a CloudWatch Logs client – we only need it once.
const cwLogs = new CloudWatchLogsClient({ region: 'us-east-1' });

// Helper to get the current timestamp in milliseconds.
const now = () => Date.now();

inferenceChannel.subscribe((event) => {
  // Build the EMF payload.
  const emf = {
    '@aws': {
      Timestamp: now(),
      CloudWatchMetrics: [
        {
          Namespace: 'MyAIService',
          MetricName: 'InferenceLatency',
          Dimensions: [['ModelName', 'Endpoint']],
        },
      ],
    },
    ModelName: event.model,
    Endpoint: event.endpoint,
    InferenceLatency: event.latencyMs,
    TokenCount: event.tokenCount,
    Success: event.success,
  };

  // Serialize to JSON – this is the single log line CloudWatch will parse.
  const message = JSON.stringify(emf);

  // In a real container you would write to stdout; CloudWatch Agent picks it up.
  // For demonstration we also push directly via SDK (optional, not required for EMF).
  console.log(message);
});

Explanation of each line

• createChannel – creates a named broadcast pipe.
• subscribe – registers a callback that runs as soon as the channel receives data, with virtually zero overhead.
• @aws block – follows the EMF spec described earlier.
• console.log(message) – writes the JSON to stdout; the AWS CloudWatch Agent (or the container’s logging driver) ships it to CloudWatch Logs.

Key takeaway: By using diagnostics_channel, you separate metric emission from business logic. The inference code just “fires an event”; the EMF formatter does the rest without adding latency.

Putting It All Together: A Real‑World AI Inference Service Example

Service skeleton

// server.ts
import http from 'node:http';
import { inferenceChannel } from './diagnosticsChannel.js';
import { startXRay } from './xraySetup.js';

// Start X‑Ray tracing (see next section for details)
await startXRay();

// Simple HTTP server that pretends to run an LLM inference.
const server = http.createServer(async (req, res) => {
  const start = Date.now();
  const requestId = crypto.randomUUID(); // unique ID for tracing

  // Simulate token counting and latency
  const tokenCount = Math.floor(Math.random() * 200) + 1;
  const latencyMs = Math.random() * 300 + 50; // 50‑350 ms

  // Randomly inject an error 5% of the time
  const success = Math.random() > 0.05;

  // Emit the EMF event via diagnostics_channel
  inferenceChannel.publish({
    model: 'gpt‑4‑mini',
    endpoint: 'text‑completion',
    latencyMs: Math.round(latencyMs),
    tokenCount,
    success,
  });

  // Respond to the caller
  res.writeHead(success ? 200 : 500, { 'Content-Type': 'application/json' });
  res.end(
    JSON.stringify({
      requestId,
      success,
      latencyMs: Math.round(latencyMs),
      tokenCount,
    })
  );
});

server.listen(8080, () => {
  console.log('AI inference service listening on :8080');
});

What this code shows

• The business path (the HTTP handler) does not contain any CloudWatch SDK calls.
• All metric data travels through diagnostics_channel, keeping the request path lean.
• Errors are captured as a boolean (Success) that can be aggregated into an error‑rate metric.
• The service also starts an X‑Ray segment (next section) so each request is traceable end‑to‑end.

Wiring the EMF logger

Remember the emfLogger.ts file from the previous section? Import it once so the subscription is active.

// index.ts
import './emfLogger.js'; // side‑effect import registers the listener
import './server.js';

When the container starts, the listener is registered, and every call to inferenceChannel.publish results in a single JSON log line.

Gotcha #3 – X‑Ray cold‑start overhead

If you enable X‑Ray on a Lambda or a container without adjusting the daemon, the X‑Ray SDK can add 50‑100 ms to cold starts. In our container we start the daemon once at boot time, and we configure the SDK to use the sampling rule that records only 5 % of requests in production.

// xraySetup.ts
import { XRayClient, PutSamplingRulesCommand } from '@aws-sdk/client-xray';
import { config } from 'node:process';

export async function startXRay() {
  // Turn off the default “all‑requests” sampler.
  const xray = new XRayClient({ region: 'us-east-1' });
  const cmd = new PutSamplingRulesCommand({
    SamplingRuleRecords: [
      {
        SamplingRule: {
          RuleName: 'AIInferenceLowSample',
          Priority: 1,
          FixedRate: 0.05, // 5 % of requests
          ReservoirSize: 5,
          ServiceName: '*',
          ServiceType: '*',
          Host: '*',
          HTTPMethod: '*',
          URLPath: '*',
          Version: 1,
          // Optional: limit to 1‑second intervals to avoid bursts.
        },
        // No specific tags needed here.
      },
    ],
  });
  await xray.send(cmd);
  console.log('X‑Ray sampler configured: 5 % of requests will be traced.');
}

Tip: Run a quick benchmark (ab -n 1000 -c 50 http://localhost:8080) before and after adding X‑Ray to see the cold‑start impact.

End‑to‑end flow recap

1. Client makes an HTTP request.
2. Server starts an X‑Ray segment (tracing) and notes the start time.
3. Inference logic runs (simulated here).
4. At the end, the code publishes an event to diagnostics_channel.
5. The EMF listener builds a JSON line that includes metrics and dimensions.
6. The container’s logging driver ships the line to CloudWatch Logs.
7. CloudWatch’s EMF processor extracts the metric and stores it under MyAIService.
8. You can now build dashboards, alarms, or use CloudWatch Application Signals for anomaly detection—all without a separate metrics SDK call.

Monitoring with CloudWatch Application Signals

What are Application Signals?

Application Signals is a newer CloudWatch feature that aggregates latency, error, and request‑count data automatically from multiple sources (including EMF) and runs machine‑learning‑based anomaly detection. It shows you “normal” ranges and flags outliers.

Connecting EMF data

Because our EMF payload already includes InferenceLatency (a latency metric) and Success (an error flag), Application Signals can ingest them directly. You only need to enable the feature on the log group.

aws logs put-subscription-filter \
  --log-group-name /aws/ecs/my-ai-service \
  --filter-name ApplicationSignals \
  --filter-pattern "" \
  --destination-arn arn:aws:logs:us-east-1:123456789012:destination:CloudWatchApplicationSignals

After a few minutes, the CloudWatch console will show a new Application Signals view for the MyAIService namespace, with automatic charts for:

• p50 / p95 latency – derived from InferenceLatency.
• Error rate – calculated from Success = false.
• Token count distribution – useful for capacity planning.

In plain English: Application Signals works like a health‑monitoring smartwatch that reads your EMF “pulse” and alerts you when something feels off.

Gotcha #4 – Missing data during deployments

When you spin up a new container version, the first few seconds may not emit any EMF lines (e.g., if warm‑up logic runs before the first request). Application Signals treats those gaps as “missing data” and can mistakenly flag a spike. Configure the “missing data treatment” in the signal’s alarm settings to “ignore” or “missing = good”.

Cross‑account observability

If your AI service runs in a dev account but you want metrics in a central monitoring account, you need to add an observability access policy to the log group:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountRead",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::987654321098:root" },
      "Action": ["logs:PutLogEvents", "logs:CreateLogStream"],
      "Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/ecs/my-ai-service:*"
    }
  ]
}

Now the central account can subscribe to the same log group and see the EMF metrics without duplicating data pipelines.

The Takeaway

Key points to remember

• EMF turns a single JSON log line into a full CloudWatch metric, removing the need for a separate PutMetricData call.
• The @aws block, correct namespace, and a subscription filter are mandatory; otherwise CloudWatch discards the metric silently.
• diagnostics_channel (Node.js 22) provides a lightweight, zero‑overhead way to broadcast inference results to an EMF formatter.
• A small listener converts those events into JSON and writes them to stdout; the CloudWatch Agent does the rest.
• Application Signals can consume your EMF metrics automatically, giving you anomaly detection and out‑of‑the‑box dashboards.
• Watch out for log‑group retention, Logs Insights query costs, metric resolution pricing, and X‑Ray cold‑start overhead.

Summary bullets

1. Emit once, read twice – one log line becomes both a log entry and a metric.
2. Configure the pipeline – @aws fields + subscription filter = metric materialization.
3. Use diagnostics_channel to keep metric emission out of the request path.
4. Tie in X‑Ray sparingly; set a low sampling rate to avoid cold‑start penalties.
5. Leverage Application Signals for automatic latency/error charts and ML‑driven alerts.
6. Guard against hidden costs – set log retention, choose 1‑minute resolution, and adjust missing‑data alarm behavior.

Now you can instrument your AI inference service with zero added latency, keep your CloudWatch bill in check, and gain instant visibility into latency, token usage, and reliability—all from a single, well‑structured log line. Happy monitoring!

Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-08-31 · Primary focus: CloudWatch

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.