Log Management at Scale: How We Cut Costs 70% Without Losing Signal

High value (always need during incidents):
  Application errors (stack traces)
  Authentication events
  Business transactions
  External API calls with responses
  Health check failures

Medium value (sometimes useful):
  Request/response logs (sampled)
  Performance metrics in logs
  Deployment events
  Configuration changes

Low value (almost never needed):
  Debug/trace level logs
  Health check successes
  Static asset requests
  Heartbeat messages
  Verbose framework logs

import os
import logging

# Log level from environment variable, changeable at runtime
LOG_LEVEL = os.environ.get('LOG_LEVEL', 'WARNING')
logging.basicConfig(level=getattr(logging, LOG_LEVEL))

# Endpoint to change log level without restart
@app.post('/admin/log-level')
async def set_log_level(level: str):
    logging.getLogger().setLevel(getattr(logging, level.upper()))
    return {'status': 'ok', 'level': level}

# Normal operation
kubectl set env deployment/api LOG_LEVEL=WARNING

# During incident
kubectl set env deployment/api LOG_LEVEL=DEBUG

# After incident
kubectl set env deployment/api LOG_LEVEL=WARNING

retention_policy:
  hot_storage:  # Fast search, expensive
    duration: 7 days
    filter: "level >= WARN OR tag:business_event"

  warm_storage:  # Slower search, cheaper
    duration: 30 days
    filter: "level >= INFO"

  cold_storage:  # Archive only, cheapest
    duration: 365 days
    filter: "tag:audit OR tag:compliance"

  drop:  # Don't store at all
    filter: "level = DEBUG OR source:health_check"

# Bad: Unstructured
logger.info(f"User {user_id} purchased {product_id} for ${amount}")
# Parsing this requires regex, which costs compute

# Good: Structured
logger.info("purchase_completed", extra={
    'user_id': user_id,
    'product_id': product_id, 
    'amount': amount,
    'currency': 'USD'
})
# Output: {"message": "purchase_completed", "user_id": "u123", ...}
# Queryable without parsing

import random

def should_log_request(request):
    # Always log errors
    if request.status_code >= 400:
        return True
    # Always log slow requests
    if request.duration_ms > 1000:
        return True
    # Sample 10% of successful requests
    return random.random() < 0.10

Before:
  Daily ingestion: 2 TB
  Monthly cost: $12,000
  Useful data: ~5%

After:
  Daily ingestion: 400 GB
  Monthly cost: $3,600
  Useful data: ~70%