When I saw the Agents of SigNoz challenge, my timeline was immediately flooded with basic uptime monitors and standard e-commerce tracing setups. But what if we flip the script and use observability as a Blue Team security tool to catch attackers in real time?
Most developers use OpenTelemetry to figure out why a database query is slow or why an API endpoint is failing. As someone who spends time on penetration testing and building custom security tools on Kali Linux, I see server latency entirely differently. A sudden CPU spike is not always bad code; sometimes it is an active attacker attempting resource exhaustion or a Denial of Service.
I wanted to see if I could build a trap. My goal was to create a vulnerable Express.js API that actively detects SQL injection payloads, deliberately stalls the server CPU, and alerts me instantly through the SigNoz dashboard using OpenTelemetry traces and metrics. Here is exactly how I built it, the errors I hit during the local deployment, and how I caught the attack.
The Setup and The Foundry Curveball
My environment of choice is Kali Linux, and I decided to self-host SigNoz using Docker. Like many developers starting out, I went straight to the GitHub repository and attempted to run the legacy bash installation script using a simple wget command.
My terminal immediately threw a <!DOCTYPE html> syntax error. It turned out I was downloading the HTML page of the script rather than the raw file, but even after fixing that, I realized the old install.sh script had been deprecated. SigNoz recently migrated to a much more robust CLI tool called Foundry for their deployments.
I pivoted to the new instructions and installed the Foundry CLI:_
curl -fsSL https://signoz.io/foundry.sh | bash
To tell Foundry that I wanted a local Docker Compose setup rather than a Kubernetes deployment, I created a quick casting.yaml configuration file:
apiVersion: v1alpha1
kind: Installation
metadata:
name: signoz
spec:
deployment:
flavor: compose
mode: docker
I ran foundryctl cast -f casting.yaml and the download process began. Everything was going smoothly until the process reached the massive ClickHouse telemetry store containers. Right at 233.1MB out of 246.3MB, my local internet connection dropped completely. The docker daemon threw a fatal context deadline exceeded (Client.Timeout exceeded while awaiting headers) error, and the entire installation crashed out with a signal: killed exception.
Usually, a broken docker pull means you have to clean up dangling images and start over. However, the Foundry CLI handled the interruption perfectly. Once my connection was restored, I ran docker compose down to clear the partial container states, re-ran the cast command, and it picked up the exact remaining 13 megabytes without missing a beat.
Building the Trap
With the SigNoz backend up and running, I needed an application to monitor. I started by creating a new directory and initializing a blank Node.js project. To make sure SigNoz could automatically capture the traces without me writing hundreds of lines of custom spans, I needed to install the official OpenTelemetry auto-instrumentation packages. If you are following along, you can set up the exact same environment using this command:
npm init -y
npm install express @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-grpc
The auto-instrumentations-node package is the real hero here. It automatically intercepts HTTP requests and Express middleware out of the box, saving a massive amount of setup time.
I wrote a specific login route to act as my trap. If the username field contained common SQL injection characters (like a single quote) or cross-site scripting tags, the server would log a local security alert.
But logging is not enough for true observability. To simulate how a real server reacts to a heavy malicious payload being mitigated by a Web Application Firewall, I added a heavy while loop that deliberately stalls the CPU for a split second before returning a 403 Forbidden error. This meant that an attack would not just cause a status code error, it would cause a highly visible, physical spike in latency.
Here is the exact code I used for app.js:
const express = require('express');
const app = express();
app.use(express.json());
// Dummy login route serving as the trap
app.post('/api/login', (req, res) => {
const { username, password } = req.body;
// Simulate detecting a malicious payload
if (username.includes("'") || username.includes("<script>")) {
console.warn(`[SECURITY ALERT] Malicious payload detected from IP: ${req.ip}`);
// Simulate high CPU and Latency during attack mitigation
let i = 0; while (i < 100000000) i++;
return res.status(403).json({ error: "Access Denied" });
}
if (username === 'admin' && password === 'admin123') {
res.status(200).json({ message: "Success" });
} else {
res.status(401).json({ error: "Invalid credentials" });
}
});
app.listen(8081, () => console.log('Vulnerable API running on port 8081'));
You might notice the application is set to listen on port 8081 instead of the standard Node.js port 8080. This was another hard lesson learned during the build. The new Foundry architecture maps the SigNoz frontend dashboard directly to port 8080. When I initially tried to run my API, I hit an EADDRINUSE error. I used a quick sed -i 's/8080/8081/g' app.js command to swap my application port and avoid the conflict.
To instrument the application without manually rewriting all my HTTP handlers, I used OpenTelemetry's auto-instrumentation package. I exported my local SigNoz endpoint and started the server:
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_RESOURCE_ATTRIBUTES="service.name=sec-target-api"
node --require @opentelemetry/auto-instrumentations-node/register app.js
Executing the Attack
With the API listening and sending telemetry data via gRPC to the local collector, I opened a second terminal window to act as the attacker machine.
I wrote a quick bash script to bombard my new API with fifty consecutive requests. I mixed standard failed logins with blatant SQL injection payloads to see if SigNoz could differentiate the traffic noise from the actual attacks.
for i in {1..50}; do
# Normal failed login attempt
curl -s -X POST http://localhost:8081/api/login -H "Content-Type: application/json" -d '{"username":"test","password":"123"}' > /dev/null
# Attack payload featuring SQLi
curl -s -X POST http://localhost:8081/api/login -H "Content-Type: application/json" -d '{"username":"admin'\'' OR 1=1--","password":"123"}' > /dev/null
done
The moment I executed the loop, my server terminal lit up with warnings. The custom security logic was catching the payloads perfectly.
Catching the Attacker in SigNoz
Local terminal logs are great, but in a production environment, you cannot sit and watch a server console all day. I opened the SigNoz dashboard on localhost:8080 and navigated to the Traces Explorer to see what the telemetry data captured.
The Traces list painted a perfect picture of the attack. Every single malicious request was captured, categorized, and flagged with a red 403 status code. The normal failed logins were also logged, allowing me to easily filter and separate standard user error from malicious intent.
But the most critical piece of observability came from the service metrics. Because I had specifically coded the API to stall the CPU during payload detection, I navigated to the Latency graphs expecting to see the impact.
The P99 latency graph showed an immediate, massive spike pushing past 200ms at the exact timestamp the bash loop was executed.
This visualizes a major concept in cybersecurity. Attackers frequently attempt to exhaust server resources by sending heavy, complex payloads that force the database or backend logic to work overtime. Without a dedicated observability platform like SigNoz, a developer might look at a sluggish server and assume they need to optimize their code or upgrade their hardware. By looking at this dashboard, I could immediately correlate the massive latency spike directly with the influx of 403 errors. The code wasn't slow; it was under attack.
To get even more granular, I clicked into one of the specific red traces from the explorer. SigNoz generated a highly detailed flamegraph representing the exact lifecycle of the HTTP POST request.
The flamegraph visually broke down exactly where the delay was happening. I could see the exact microsecond the request hit the Express middleware, passed through the JSON parser, and finally hit the patched request handler where the CPU stall occurred. If this had been a real application, this trace would tell me exactly which function was vulnerable to resource exhaustion.
What I Learned and Takeaways
Building this project taught me several practical lessons that reading documentation simply cannot cover:
- Foundry is Resilient: Dealing with massive Docker image pulls on an unstable internet connection is usually a nightmare. The Foundry CLI proved incredibly resilient, allowing me to resume a broken ClickHouse installation seamlessly.
- Mind Your Ports: When self-hosting complex microservice architectures, port collisions are inevitable. Knowing that SigNoz claims port 8080 by default saved me a lot of debugging time when setting up my own Node.js backend.
- Metrics Tell a Story: Security is not just about error logs. Monitoring P90 and P99 latency metrics provides a physical representation of how your server reacts to hostile traffic. A sudden latency spike combined with an unusual HTTP status code is often the first indicator of an active breach attempt.
Conclusion
Open-source tools like SigNoz give you total, uncompromised visibility into your system, proving that by combining full-stack development with deep observability, you can catch anomalies before they become full-scale breaches.
