Finding out there is "suspicious activity" in your infrastructure is enough to make any DevOps engineer's heart rate spike. If you’re running containerized workloads, you need a way to see exactly what’s happening inside those isolated environments in real-time.
Falco, the open-source standard for cloud-native runtime security. In this guide, we'll walk through a hands-on scenario: investigating a suspicious Nginx container by detecting unauthorized spawning processes.
A team member reports odd behavior in a specific container. Our goal is to use Falco to monitor the execve system call—which is triggered whenever a new process is started—and log those events to a report for analysis.
Step 1: Create a Custom Falco Rule
Falco uses a flexible YAML-based syntax for defining security rules. We need to create a rule specifically targeting our Nginx container.
- Create a new rules file:
vi nginx-rules.yml - Paste the following configuration:
- rule: spawned_process_in_nginx_container
desc: A process was spawned in the Nginx container.
condition: container.name = "nginx" and evt.type = execve
output: "%evt.time,%proc.name,%user.uid,%container.id,%container.name,%container.image"
priority: WARNING
- Save and exit (
Esc,:wq,Enter).
Breakdown of the Rule:
- Condition: We are filtering for events where the container name is exactly "nginx" and the event type is
execve(process execution).- Output: This defines the format of our log, capturing the timestamp, process name, user ID, and container metadata.
Why is this rule important?
In a secure, production containerized environment, containers should follow the principle of immutability. An Nginx container should only run Nginx.
If a hacker successfully exploits a vulnerability in your Nginx web server, the first thing they will often try to do is open a reverse shell (bash or sh) or run malicious scripts to look around. Because execve catches any new process being spawned, this rule will instantly catch an attacker attempting to run commands inside your container.
Step 2: Run the Analysis
Now, we run Falco using our custom rule. We’ll use the -M flag to run the scan for a set duration (45 seconds) and redirect the output to a log file for further investigation.
Run the following command as root:
sudo falco -r nginx-rules.yml -M 45 > /home/cloud_user/falco-report.log
-M 45: Instructs Falco to run in duration mode for exactly 45 seconds and then gracefully terminate.
Step 3: Audit the Results
Once the run is complete, you can inspect the log file to see every process that was triggered during that window. This is your "paper trail" for the suspicious activity.
cat /home/cloud_user/falco-report.log
Summary of What Happened
Look at the timestamps and the repeating process loop:
05:07:16: sh $\rightarrow$ cat $\rightarrow$ sh $\rightarrow$ sleep
05:07:21: sh $\rightarrow$ cat $\rightarrow$ sh $\rightarrow$ sleep (Exactly 5 seconds later)
05:07:26: sh $\rightarrow$ cat $\rightarrow$ sh $\rightarrow$ sleep (Exactly 5 seconds later)
The Verdict: Inside your container named nginx, there is an automated loop script running every 5 seconds. This script spawns a shell, uses cat to read a file, and then executes sleep 5 before repeating.
Because your Falco rule looks for any new process execution (evt.type = execve), it successfully caught all 4 commands running every 5 seconds. Over the 45-second testing period, this loop ran 9 times, resulting in exactly 36 total security alerts (9 loops Ă— 4 commands = 36 events).
