How to Implement Log Sanitization with Fluent Bit 3.0 and AWS Lambda 2026 for PII Redaction

Introduction

Log data often contains sensitive personally identifiable information (PII) such as email addresses, phone numbers, social security numbers, and credit card details. Unprotected PII in logs poses compliance risks (GDPR, CCPA, HIPAA) and increases the likelihood of data breaches. Log sanitization via PII redaction removes or masks this sensitive data before logs are stored or processed further.

Fluent Bit 3.0, the lightweight log processor and forwarder, pairs seamlessly with AWS Lambda 2026 (the 2026 iteration of AWS’s serverless compute service) to build a scalable, cost-effective log sanitization pipeline. This guide walks through implementing this pipeline end-to-end.

Prerequisites

• Fluent Bit 3.0 installed on your log source (EC2, EKS, on-prem server, or containerized environment)
• Active AWS account with permissions to create Lambda functions, IAM roles, and configure Fluent Bit output plugins
• AWS CLI 2026 version configured locally for testing
• Basic familiarity with regular expressions (regex) and Python (for Lambda function development)

Step 1: Configure Fluent Bit 3.0 to Collect Logs

First, define Fluent Bit’s input configuration to collect logs from your target source. Fluent Bit 3.0 supports 80+ input plugins for files, systemd, container runtimes, and cloud services. Below is a sample configuration for tailing application log files:

[INPUT]
    Name tail
    Path /var/log/myapp/*.log
    Parser docker
    Tag myapp.logs
    Refresh_Interval 5

[INPUT]
    Name systemd
    Tag systemd.logs
    Systemd_Filter _SYSTEMD_UNIT=myapp.service

Fluent Bit 3.0’s updated parser engine supports nested JSON and custom grok patterns out of the box, reducing pre-processing overhead.

Step 2: Develop the AWS Lambda 2026 PII Redaction Function

AWS Lambda 2026 supports Python 3.12+, Node.js 22+, and new managed runtimes with built-in PII detection libraries. We’ll use Python 3.12 for this example, combining regex-based redaction for structured PII and Amazon Comprehend 2026 for unstructured text detection.

Sample Lambda function code:

import re
import boto3
from typing import Dict, List

comprehend = boto3.client('comprehend')

# Regex patterns for common PII
PII_PATTERNS = {
    'email': r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+',
    'phone': r'\+?1?\d{9,15}',
    'ssn': r'\d{3}-\d{2}-\d{4}',
    'credit_card': r'\d{4}-\d{4}-\d{4}-\d{4}'
}

def redact_pii(text: str) -> str:
    # Redact structured PII via regex
    for pii_type, pattern in PII_PATTERNS.items():
        text = re.sub(pattern, f'[REDACTED_{pii_type.upper()}]', text)

    # Detect unstructured PII via Amazon Comprehend 2026
    try:
        response = comprehend.detect_pii_entities(Text=text, LanguageCode='en')
        for entity in response['Entities']:
            start, end = entity['BeginOffset'], entity['EndOffset']
            text = text[:start] + f'[REDACTED_{entity["Type"]}]' + text[end:]
    except Exception as e:
        print(f"Comprehend error: {e}")

    return text

def lambda_handler(event: Dict, context):
    redacted_logs = []
    # Fluent Bit sends logs as a list of records in the event
    for record in event.get('records', []):
        log_data = record['data'] if isinstance(record['data'], str) else json.dumps(record['data'])
        redacted = redact_pii(log_data)
        redacted_logs.append({
            'recordId': record.get('recordId'),
            'data': redacted,
            'result': 'Ok'
        })

    return {'records': redacted_logs}

Note: AWS Lambda 2026 introduces native PII redaction APIs, reducing reliance on external services for basic use cases.

Step 3: Integrate Fluent Bit 3.0 with AWS Lambda 2026

Configure Fluent Bit’s output plugin to forward logs to your Lambda function. Fluent Bit 3.0 includes an updated AWS Lambda output plugin with support for asynchronous invocation and batching:

[OUTPUT]
    Name lambda
    Match myapp.logs
    Function_Name myapp-pii-redaction-lambda
    Region us-east-1
    Batch_Size 100
    Batch_Time 1s
    Retry_Limit 3
    IAM_Role fluent-bit-lambda-invoke-role

Assign the IAM role fluent-bit-lambda-invoke-role to your Fluent Bit host (EC2 instance profile, EKS service account IAM role) with permissions to invoke the Lambda function. The Lambda function itself needs an IAM role with permissions to call Amazon Comprehend and write redacted logs to your final destination (e.g., CloudWatch Logs, S3).

Step 4: Test the Pipeline

Generate sample logs with PII to validate the pipeline:

echo '{"user_email": "test@example.com", "phone": "1234567890", "message": "User logged in"}' >> /var/log/myapp/app.log

Check Fluent Bit logs to confirm it forwards the record to Lambda, then verify the Lambda function’s CloudWatch logs to ensure PII is redacted:

{"user_email": "[REDACTED_EMAIL]", "phone": "[REDACTED_PHONE]", "message": "User logged in"}

Forward redacted logs to your final storage (e.g., CloudWatch, Elasticsearch) by adding a second output plugin to Fluent Bit or configuring the Lambda function to write to the destination directly.

Step 5: Production Optimization

• Enable Fluent Bit 3.0’s built-in buffering and retry logic to handle transient Lambda errors
• Use AWS Lambda 2026’s provisioned concurrency for predictable latency, or leverage new scale-to-zero enhancements for cost savings
• Monitor pipeline health with Fluent Bit’s Prometheus metrics and Lambda CloudWatch metrics (invocation count, error rate, duration)
• Rotate IAM credentials regularly and use AWS Secrets Manager 2026 to store sensitive configuration

Conclusion

Implementing log sanitization with Fluent Bit 3.0 and AWS Lambda 2026 provides a scalable, low-maintenance solution for PII redaction. This pipeline reduces compliance risk, protects user privacy, and integrates seamlessly with existing log infrastructure. As Fluent Bit and Lambda add new features in 2026, you can extend this pipeline to support additional PII types, real-time alerting, and cross-region log replication.