What happens when a web server captures an attack in its access logs, but the underlying host system remains completely blind to the actual commands being run? Setting up a Security Information and Event Management (SIEM) platform is not just about turning on a tool and watching a dashboard. It is about engineering true visibility and discovering where critical telemetry gets lost. In this post, we will look at configuring custom container log ingestion in a Wazuh SIEM environment on play-server, executing live reconnaissance and command injection attacks against Damn Vulnerable Web Application (DVWA), and auditing post-exploitation persistence. We will then see how these configurations perform under real testing and where logging breakdowns occur.
My name is Joseph, and before entering the cybersecurity field, I worked for four years as a residential house painter. Painting taught me that quality always comes down to surface preparation—if you skip sanding and priming, the topcoat fails under stress. That exact principle translates directly into security operations. Plumbing log pipelines, fixing permission blocks, and making sure your sensors actually capture threat activity is the SOC equivalent of prep work. Deploying and tuning this SIEM environment has been one of the most rewarding parts of my learning journey, and I hope this write-up helps anyone looking to build and audit their own detection lab.
To create a practical testing lab, I set up a containerized instance of Damn Vulnerable Web Application (DVWA) running on port 8080 of play-server, monitored by a host-level Wazuh agent.
Standard agent installations only ingest stock system logs (/var/log/syslog, /var/log/auth.log). To capture web application attacks hitting DVWA, I modified the deployment to bind-mount the container’s internal Apache logs directly to the host filesystem at /var/log/dvwa/.

1. Verify Directory and Bind Mount: Created the host-side log directory and confirmed the container logs were mapping: sudo mkdir -p /var/log/dvwa docker run -d -p 8080:80 -v /var/log/dvwa:/var/log/apache2 --name dvwa vulnerables/web-dvwa
2. Update Agent Configuration: Appended a custom localfile collection block to /var/ossec/etc/ossec.conf: sudo nano /var/ossec/etc/ossec.conf
3. Updated the host Wazuh agent configuration located at /var/ossec/etc/ossec.conf by appending a custom localfile collection block: <localfile> <log_format>apache</log_format> <location>/var/log/dvwa/access.log</location> </localfile>
4. Restart the Agent: Applied the new configuration settings: sudo systemctl restart wazuh-agent

Setup Mistake & Solution: The Hidden Permission Gatekeeper

Immediately after adding the configuration, I generated test traffic on DVWA, expecting events to populate in the Wazuh Dashboard right away. However, zero events appeared.

The Mistake:

I created the host directory /var/log/dvwa/ while operating under root privileges, leaving the directory permissions set to a restrictive 700. Because the wazuh agent process runs under its own unprivileged service user account, it was silently denied read access to /var/log/dvwa/access.log.

The Solution:

I inspected directory permissions using ls -la /var/log/dvwa/ and spotted the access boundary issue. I updated the permissions across the directory and log files so the host agent could read the stream: sudo chmod -R 755 /var/log/dvwa/ sudo chmod 644 /var/log/dvwa/*

Within seconds of applying the fix, the Wazuh Manager began receiving live Apache access events from the container.

Experiment time!

With telemetry successfully feeding into the SIEM, I conducted three targeted experiments to evaluate how Wazuh handles reconnaissance, web application attacks, and post-exploitation activity.

Experiment 1: Network Reconnaissance & Web Scanning

Objective: Determine if default Wazuh decoders detect automated reconnaissance and vulnerability scanning targeting the DVWA web instance.
Execution: From an external host (172.16.20.200), I executed an Nmap service scan followed by a Nikto web vulnerability scan: nmap -sV 10.170.0.22 nikto -h http://10.170.0.22:8080

Findings: Apache logs captured rapid request bursts and scanner User-Agent headers, triggering Wazuh Rule 31101 (Web server 400 error codes) and Rule 31108 (Vulnerability scanner activity). The SIEM recorded over 380 event hits in Wazuh Discover categorized under Rule ID 31101 via the web-accesslog decoder. Automated scanners create high-volume, noisy telemetry that standard SIEM rules pick up easily.

Experiment 2: Command Injection & Kernel Process Audit

Objective: Test SIEM visibility against an OS command injection payload submitted through DVWA to evaluate application-level vs. OS-level telemetry correlation.
Execution: Through the DVWA Command Injection web interface, I submitted the following payload: 127.0.0.1; whoami; id; cat /etc/passwd The browser successfully executed the commands, returning system account details for www-data. I monitored raw log output directly on play-server using sudo tail -f /var/log/dvwa/access.log.

Findings: Apache access logs recorded POST /vulnerabilities/exec/ HTTP/1.1 from source IP 172.16.20.200. However, querying system audit logs via sudo ausearch -m execve -ts recent | grep -E "whoami|passwd|cat" yielded zero hits. This highlighted a vital architectural boundary: standard web logs record external HTTP parameters, but container namespace isolation hides internal child processes from host-level audit without dedicated container socket monitoring.

Experiment 3: Post-Exploitation Persistence via Scheduled Tasks

Objective: Test SIEM detection when an adversary establishes persistence by modifying Linux system schedule files (MITRE ATT&CK T1053.003).
Execution: I simulated post-compromise persistence on play-server by appending a scheduled task to /etc/crontab:echo "* * * * * root echo 'SIEM test persistence' >> /tmp/test.log" | sudo tee -a /etc/crontab

Findings: Running sudo ausearch -f /etc/crontab returned "no matches" because no kernel audit watch rule (-w /etc/crontab -p wa) had been explicitly declared in /etc/audit/rules.d/. However, administrative execution telemetry caught the activity, triggering a high-severity alert under Wazuh Rule 5402 (Successful sudo to ROOT executed), which recorded the full execution string data.command: /usr/bin/tee -a /etc/crontab.

Experimentation Mistake: Fighting Dashboard Filters

During Experiment 2, I spent over an hour struggling to find my manual command injection payload in the Wazuh Discover dashboard. I kept filtering for terms like whoami and cat /etc/passwd in the UI search bar but received zero matching results.

The Mistake:

I assumed Wazuh automatically parsed raw HTTP POST request body parameters into custom dashboard fields by default. Additionally, ongoing web scanner traffic was filling the index, leading me to believe the log entry was not being generated at all.

The Solution:

I stepped back from the web UI and opened a direct terminal session on play-server to run sudo tail -f /var/log/dvwa/access.log while submitting the payload in my browser. Seeing the POST /vulnerabilities/exec/ entry populate in real time proved that the log was reaching the host. I adjusted my SIEM query to search for the URI path (/vulnerabilities/exec/) rather than the unindexed POST payload parameters, instantly locating the event.

Conclusion

We looked at the three main experiments in this SIEM deployment and saw how each one generated different levels of telemetry. First, reconnaissance tools like Nikto and Nmap hit web endpoints aggressively, causing standard HTTP error decoders to flag them immediately. Second, web access logs captured external request paths like POST /vulnerabilities/exec/, but host audit daemons like auditd could not see inside container process namespaces without specialized socket integrations. Lastly, while auditd failed to flag /etc/crontab edits due to a missing file watch rule, administrative process monitoring captured the exact sudo tee persistence payload.
To avoid making these same mistakes in your own lab, always verify that host service accounts have proper read access (chmod 644 / 755) to custom log paths before troubleshooting SIEM configuration files. Additionally, use terminal log tailing (tail -f) to verify event generation at the host level before assuming your SIEM dashboard search syntax or agent decoders are broken.

Final thoughts

The coolest thing I learned

The coolest concept I learned during this project is that a SIEM is only as strong as its underlying decoders and file permissions. Seeing raw HTTP requests get transformed into structured alerts taught me that security operations is not just about collecting data but about understanding how decoders dissect raw text strings into actionable fields.

One piece of advice

Master your log paths and permissions before touching detection rules. I lost considerable time early on simply because the wazuh user account could not read /var/log/dvwa/. Before writing custom rules or searching dashboards, confirm your host agent can physically read the target log stream.

My favorite resource

My single favorite external resource throughout this sprint was Atomic Red Team by Red Canary. Having a structured library of executable tests directly mapped to MITRE ATT&CK techniques made simulating attacker activity, such as cron job persistence, simple, clear, and repeatable.

Thank you

I want to give a huge thank-you to two key contributors in the security space:
The Atomic Red Team Project (Red Canary & contributors like Casey Smith): Providing open-source, accessible attack telemetry generation tools allows aspiring analysts to test defenses in real-world scenarios without needing years of penetration testing experience.
The Wazuh Open Source Team: Building a powerful, accessible, open-source SIEM platform gives students and home-lab researchers hands-on experience with enterprise-grade security monitoring.

References

1. Wazuh Ruleset - Apache Log Analysis
   Author: Wazuh Documentation Team (Wazuh, Inc.)
   Published: 2026-01-15
   Accessed: 2026-08-25
   Description: Official documentation detailing how the Wazuh manager decodes Apache access logs, covering default rule mappings for HTTP error codes and web scanner signatures.

2. MITRE ATT&CK T1059.004 - Command and Scripting Interpreter: Unix Shell
   Author: MITRE Corporation
   Published: 2025-10-12
   Accessed: 2026-08-26
   Description: Threat intelligence framework reference outlining how adversaries leverage Unix shell interpreters to execute arbitrary system commands via application exploits.

3. MITRE ATT&CK T1053.003 - Scheduled Task/Job: Cron
   Author: MITRE Corporation
   Published: 2025-09-18
   Accessed: 2026-08-29
   Description: Comprehensive breakdown detailing adversary methods for maintaining persistent system access via Linux cron schedules.

4. Atomic Red Team - T1053.003 Cron Persistence Tests
   Author: Red Canary
   Published: 2025-11-04
   Accessed: 2026-08-29
   Description: An open-source repository of actionable red team automation scripts designed to validate SOC detection capabilities against scheduled task persistence.

5. OWASP Command Injection Defense Cheat Sheet
   Author: OWASP Foundation
   Published: 2024-12-01
   Accessed: 2026-08-26
   Description: Practical security reference detailing the underlying mechanisms of command injection vulnerabilities and strategies for proper input sanitization.