I created this post for the purposes of entering the All Things about Agentic Hackathon (#AllThingsAgenticHackathon).

The problem: alert fatigue is real

Security teams are drowning in alerts. SSH brute-force attempts, anomalous
logins, port scans, privilege escalation attempts — most of it is noise,
but a few alerts every day are genuinely dangerous. Sorting the real
threats from the routine ones by hand burns hours that should go toward
actual incident response.

I wanted to build something that didn't just talk about security
alerts, but actually made a judgment call and acted on them — the way a
junior analyst on triage duty would, but running continuously in the
background.

That's what I built for the Taskmaster track of the All Things
Agentic Hackathon: an autonomous agent that reads incoming security
alerts, decides what they mean, and either handles them or escalates them
— without a human triggering each step.

What the agent actually does

For every alert it receives, the agent:

1. Classifies it as low-risk or high-risk, using Gemini to reason about the specifics — not just keyword matching
2. Auto-remediates clearly benign activity (routine logins from known VPN IPs, self-service password resets completed through normal MFA) by calling a remediate_alert tool
3. Escalates ambiguous or dangerous activity (brute-force patterns, privilege escalation, suspected data exfiltration) by calling an escalate_alert tool — complete with a drafted incident summary and a severity level a human analyst can act on immediately
4. Logs every decision to a persistent audit trail, so nothing is a black box afterward

And the resulting audit trail — one clean record per alert, showing
exactly what the agent decided and why:

{
  "alert_id": "ALT-1001",
  "action": "escalated",
  "severity": "critical",
  "summary": "Potential root compromise via SSH brute force on production database prod-db-01 from external IP 41.203.12.8 (6 failed attempts followed by a successful login within 40 seconds).",
  "logged_at": "2026-08-29T09:50:18.634514+00:00"
}

The stack:

• Gemini, accessed via the Gemini API, does the actual reasoning and classification for each alert
• Google ADK (Agent Development Kit) orchestrates the agent — the model, its instructions, and the two tools it can call
• The audit trail is written through a small storage layer that transparently switches between a local JSON file (for development) and Firestore (for the cloud-deployed version), controlled by a single environment variable
• A separate FastAPI service (cloud_main.py) is built to receive alerts pushed from Pub/Sub, designed to run on Cloud Run so the agent operates asynchronously in the background rather than on a fixed schedule

One design decision I'm genuinely happy with: the agent's core logic in
agent.py and tools.py never changes between local development and the
cloud-deployed version. Only the entry point (a local loop vs. a
Pub/Sub push handler) and the storage backend (local JSON vs.
Firestore) differ — and that's controlled by one environment variable.
That made building toward the cloud version feel like an extension of the
local one, not a rewrite.

A challenge worth mentioning: handling upstream failures gracefully

Partway through testing, I hit intermittent 503 UNAVAILABLE errors from
the Gemini API during periods of high demand. Rather than let one
transient failure crash an entire run, I added a retry layer around each
alert with a short backoff:

for attempt in range(1, MAX_RETRIES + 1):
    try:
        # process the alert
        return
    except ServerError as e:
        if attempt == MAX_RETRIES:
            print(f"FAILED after {MAX_RETRIES} attempts: {e}")
            return
        print(f"  (attempt {attempt}/{MAX_RETRIES}) retrying...")
        await asyncio.sleep(RETRY_DELAY_SECONDS)

During one of my test runs, this happened in real time — the agent hit a
503 on one alert, retried automatically, and completed correctly on the
next attempt, without any manual intervention. For something meant to run
unattended in the background, that resilience matters as much as the
core reasoning does.

What I learned

Building this reinforced something I'd only understood abstractly before:
decoupling an agent's reasoning from its trigger source and its storage
backend is what actually makes "runs in the background" a real property
of the system, rather than just a description of the demo. If the
agent's logic had been tangled up with "read from this local file" or
"write to this specific database," moving toward a genuinely asynchronous,
cloud-native version would have meant a rewrite instead of an extension.

What's next

• Complete a live Cloud Run deployment (the code is ready — see the repo)
• A lightweight dashboard reading directly from the audit trail, for a more visual view of the agent's decisions over time
• Expanding the synthetic alert set and tuning the escalation logic against a broader range of real-world alert types

Built for the Taskmaster track of the All Things Agentic Hackathon.

AllThingsAgenticHackathon