AI-Driven Automated Network Monitoring & Anomaly Detection — Part 1: Setting Up the Monitoring Stack with Prometheus and Node Ex

ItemRecommended Version (2026)Why?


Linux host (Ubuntu 22.04 LTS or Rocky 9)22.04 / 9Stable, long‑term support, native systemd.
Docker Engine24.0+Containerised Prometheus and Node Exporter.
Python3.12Needed for the AI inference side‑car.
Git2.40+For pulling configuration repos.

# Ubuntu example
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

# Add Docker’s official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
    | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Set up the repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Verify
docker version

# Create a dedicated network
docker network create monitoring

# Run Node Exporter (listens on 9100)
docker run -d \
  --name node-exporter \
  --network monitoring \
  --restart unless-stopped \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  -v "/sys:/sys:ro" \
  -v "/proc:/proc:ro" \
  -v "/etc/localtime:/etc/localtime:ro" \
  prom/node-exporter:latest \
  --path.rootfs=/host

curl http://localhost:9100/metrics | head -n 20
# Expected: # HELP node_cpu_seconds_total ...

mkdir -p $HOME/prometheus/{data,conf}

# $HOME/prometheus/conf/prometheus.yml
global:
  scrape_interval: 15s        # Default scrape cadence
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

remote_write:
  - url: "http://ai-infer:9091/write"
    # Optional: add authentication headers if you lock the endpoint down
    # basic_auth:
    #   username: prometheus
    #   password: secret

docker run -d \
  --name prometheus \
  --network monitoring \
  -p 9090:9090 \
  -v $HOME/prometheus/conf/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v $HOME/prometheus/data:/prometheus \
  --restart unless-stopped \
  prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus

python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn prometheus-client scikit-learn pandas

# train_model.py
import pandas as pd
from sklearn.ensemble import IsolationForest
import joblib

# Pull a few minutes of historic node metrics (you can use the Prometheus HTTP API)
# For the demo we generate synthetic data
def synthetic_data(rows=5000):
    import numpy as np
    ts = pd.date_range(end=pd.Timestamp.now(), periods=rows, freq='15s')
    df = pd.DataFrame({
        'cpu_usage': np.random.normal(0.2, 0.05, size=rows),   # 20% avg
        'mem_usage': np.random.normal(0.55, 0.07, size=rows),  # 55% avg
        'net_rx': np.random.exponential(100, size=rows),
        'net_tx': np.random.exponential(80, size=rows),
    }, index=ts)
    return df

df = synthetic_data()
model = IsolationForest(contamination=0.01, random_state=42)
model.fit(df)

joblib.dump(model, 'anomaly_model.joblib')
print('Model saved.')

python train_model.py

# ai_infer_service.py
import joblib
import pandas as pd
from fastapi import FastAPI, Request, HTTPException
from prometheus_client.parser import text_string_to_metric_families
import uvicorn
import logging

app = FastAPI()
model = joblib.load('anomaly_model.joblib')
log = logging.getLogger("uvicorn.error")

def parse_remote_write(body: bytes) -> pd.DataFrame:
    """
    Very simplified parser: Prometheus remote_write sends protobuf-encoded
    WriteRequest messages.  For a tutorial we accept the *text* format that
    the Node Exporter also exposes (via /metrics) and convert it to a DataFrame.
    """
    metrics = {}
    for fam in text_string_to_metric_families(body.decode()):
        for sample in fam.samples:
            name, labels, value = sample
            # Flatten label dict into a single key for demo purposes
            key = f"{name}{{" + ",".join(f"{k}='{v}'" for k, v in labels.items()) + "}}"
            metrics[key] = value
    # Convert to a single‑row DataFrame
    df = pd.DataFrame([metrics])
    # Keep only the features we trained on
    features = ['node_cpu_seconds_total{mode=\'idle\'}',
                'node_memory_MemAvailable_bytes',
                'node_network_receive_bytes_total',
                'node_network_transmit_bytes_total']
    # Fill missing with 0 (real service would handle NaNs more gracefully)
    df = df.reindex(columns=features, fill_value=0)
    return df

@app.post("/write")
async def remote_write(request: Request):
    try:
        payload = await request.body()
        df = parse_remote_write(payload)
        # Predict anomaly (1 = normal, -1 = outlier)
        pred = model.predict(df)[0]
        if pred == -1:
            log.warning(f"⚠️ Anomaly detected: {df.to_dict(orient='records')[0]}")
        return {"status": "ok", "anomaly": pred == -1}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=9091)

# Dockerfile
FROM python:3.12-slim

WORKDIR /app
COPY venv /app/venv
COPY ai_infer_service.py /app/
COPY anomaly_model.joblib /app/

ENV PATH="/app/venv/bin:$PATH"

EXPOSE 9091
CMD ["uvicorn", "ai_infer_service:app", "--host", "0.0.0.0", "--port", "9091"]

# Build
docker build -t ai-infer:latest .

# Run (same monitoring network)
docker run -d \
  --name ai-infer \
  --network monitoring \
  -p 9091:9091 \
  --restart unless-stopped \
  ai-infer:latest

# Simulate CPU load for 30 seconds
stress --cpu 4 --timeout 30
# Wait for the next scrape; you should see a warning in the ai-infer container logs
docker logs -f ai-infer | grep "Anomaly"

⚠️ Anomaly detected: {'node_cpu_seconds_total{mode='idle'}': 12.34, 'node_memory_MemAvailable_bytes': 1.2e+09, 'node_network_receive_bytes_total': 5.6e+07, 'node_network_transmit_bytes_total': 2.1e+07}

CheckWhy it mattersHow to verify


Node Exporter runs as privilegedNeeded for host metrics (e.g., /proc)`docker inspect node-exporter | grep Privileged`
Prometheus scrape interval ≤ 15 sHigher granularity improves anomaly detection latencyCheck `global.scrape_interval` in `prometheus.yml`
Remote write TLS/Basic AuthProtect metric pipeline from eavesdroppingInspect `remote_write` block for `tls_config` or `basic_auth`
Model versioningAI drift can cause false positivesStore `.joblib` files with semantic tags (e.g., `model_v2026.08.01.joblib`)
Alerting rule for anomaly flagPush notifications to Slack/TeamsCreate a Prometheus rule that fires on `ai_anomaly_detected == 1`

from prometheus_client import Gauge, start_http_server

# Define a gauge that will be scraped by Prometheus
anomaly_gauge = Gauge('ai_anomaly_detected', '1 if an anomaly was detected in the last batch')

@app.on_event("startup")
def start_metrics():
    # Expose metrics on port 8000 (different from write endpoint)
    start_http_server(8000)

@app.post("/write")
async def remote_write(request: Request):
    ...
    if pred == -1:
        anomaly_gauge.set(1)
    else:
        anomaly_gauge.set(0)
    ...

  - job_name: 'ai-infer'
    static_configs:
      - targets: ['ai-infer:8000']

groups:
  - name: ai_anomaly_alerts
    rules:
      - alert: NetworkAnomalyDetected
        expr: ai_anomaly_detected == 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Anomaly detected on {{ $labels.instance }}"
          description: "AI model flagged a metric outlier. Check node metrics for root cause."