Building a Secure Self-Hosted Observability Pipeline for an AI Evaluation Platform

When building an AI evaluation platform, it is tempting to begin with dashboards, models, and scoring frameworks.

We started with a more fundamental question:

How can a team safely view evaluation telemetry without exposing the VM, its databases, credentials, or internal services to the public internet?

This article documents the pipeline we actually built: a secure, self-hosted trace-observability path for an AI evaluation platform.

It does not claim that a durable multi-tenant evaluation execution system has been completed. The observability and access pipeline is live; durable job execution, real project onboarding, and CI gating remain separate next steps.

The pipeline we built

The implemented path is:

Team member’s browser
        |
        | HTTPS
        v
Cloudflare Access
        |
        | Email one-time passcode
        | Approved company email domain
        v
Cloudflare Tunnel
        |
        | Outbound-only connection from Azure
        v
Azure virtual machine
        |
        v
Caddy reverse proxy
        |
        +----------------------+
        |                      |
        v                      v
Arize Phoenix UI         Evaluation API
                               |
                               | Safe operational trace metadata
                               v
                       OpenTelemetry Collector
                               |
                               | Redaction rules
                               v
                         Arize Phoenix
                               |
                               v
                     Trace and evaluation visibility

This gives the team a clean browser interface for traces and evaluation metadata while keeping the actual infrastructure private.

What is live today

The deployed pipeline includes:

• A self-hosted Arize Phoenix instance for trace visibility
• A protected browser entrypoint
• Company-email OTP authentication at the edge
• A Cloudflare Tunnel connecting the public edge to the private Azure VM
• A Caddy reverse proxy routing browser and API traffic locally
• An evaluation API that emits safe observability spans
• An OpenTelemetry Collector that applies redaction before telemetry reaches Phoenix
• Private container image distribution through Azure Container Registry
• Azure Key Vault-backed secret retrieval
• Azure managed identity for workload access
• A synthetic, non-sensitive validation trace visible in Phoenix

The live implementation is an observability pipeline, not yet a finished evaluation-job processing platform.

What is not yet live

The following are planned next steps and should not be confused with the deployed pipeline:

• Durable evaluation job queueing
• Persistent evaluation job state
• DeepEval worker execution for submitted jobs
• Real production project onboarding
• Per-project quotas and token budget enforcement
• Entra workload identity checks for agent-to-platform API calls
• Project-level UI access separation
• Automated raw-data retention and backup expiry
• CI advisory-to-gate workflow promotion

Being explicit about this boundary is important. A trace platform can prove that telemetry moves safely through the system; it does not by itself prove that evaluation jobs are being executed, persisted, or governed correctly.

Why we used a tunnel instead of a public VM

The Azure VM has no public IP address.

This means the following are not exposed to the internet:

• SSH
• Phoenix
• The evaluation API
• PostgreSQL
• OpenTelemetry Collector ports
• Container runtime ports
• Internal reverse-proxy routes

Instead of accepting inbound traffic from the internet, the VM runs a tunnel client that creates an outbound connection to Cloudflare.

Traditional public-server pattern

Internet --> Public IP --> VM --> Application


Implemented pattern

Internet --> Cloudflare Access --> Cloudflare Tunnel <-- Private VM

This removes the most obvious attack surface: a publicly addressable VM.

Even if someone discovers the Azure resource group, VM name, private IP, or application port, they cannot directly reach the workload from the public internet.

Browser access: secure but simple for the team

The platform is protected using Cloudflare Access.

The user flow is:

1. A team member opens the evaluation platform in a browser.
2. Cloudflare intercepts the request before it reaches Azure.
3. The user enters their company email address.
4. Cloudflare sends a one-time passcode.
5. Cloudflare verifies the passcode.
6. Cloudflare allows the authenticated browser session through the tunnel.
7. The user reaches the Phoenix interface.

This gives the team a low-friction experience while preserving a strong outer security boundary.

There is no shared platform password.

There is no public application login endpoint.

There is no need to expose the VM directly to make the UI convenient.

Security layers in the implemented pipeline

The security design uses multiple layers rather than relying on one control.

Layer 1: Cloudflare edge protection
Layer 2: Company-domain email OTP
Layer 3: Cloudflare Tunnel
Layer 4: No public Azure VM IP
Layer 5: Local reverse proxy routing
Layer 6: Private container registry
Layer 7: Managed identity
Layer 8: Key Vault-backed secrets
Layer 9: Telemetry redaction
Layer 10: Separate service containers and databases

If one layer is misconfigured, the other layers still reduce the blast radius.

Layer 1: Cloudflare Access

Cloudflare Access is the first line of defence.

It validates identity before requests are sent to the Azure environment. Only users with an approved company email domain can receive an OTP and establish an authenticated browser session.

This prevents anonymous internet users from reaching the application interface.

Layer 2: No public IP on the VM

The VM does not have a public IP address.

This removes direct exposure for:

• Remote administration ports
• Browser application ports
• Database ports
• Telemetry ports
• Container-host ports

Administrative access must come through approved Azure and network access paths rather than a public SSH endpoint.

Layer 3: Outbound-only tunnel

The tunnel originates from the VM.

That matters because the Azure workload does not need to accept public inbound traffic. It only needs to establish outbound connectivity to Cloudflare.

The tunnel maps the protected public hostname to a local reverse-proxy listener on the VM.

The result is:

Cloudflare knows how to reach the service.
The public internet does not know how to reach the VM.

Layer 4: Caddy as the local traffic boundary

Caddy acts as the local reverse proxy.

It provides one controlled internal entrypoint and routes traffic to the correct service:

/             -> Phoenix UI
/v1           -> Evaluation API
/health       -> Evaluation API health check
/ready        -> Evaluation API readiness check

This avoids exposing each service separately to users.

The browser sees one protected site. Internal services remain behind the local proxy boundary.

Private image distribution

The containers used by the platform are stored in Azure Container Registry.

The registry is configured to avoid public image access. The VM retrieves images using its Azure managed identity.

This provides several advantages:

• No container-registry password is stored in source code.
• The VM does not need broad human credentials.
• Runtime identity permissions can be limited to image pulls.
• Build permissions can be granted temporarily and removed after publishing.
• Images can be pinned by immutable digest.

Why immutable image digests matter

A container tag such as latest or production can change over time.

An immutable digest identifies one exact image build.

That gives us a stronger deployment statement:

This workload ran this exact image.

rather than:

This workload ran whatever image happened to have this tag.

Temporary image-push permission was used only during the build process and revoked afterward. The runtime VM retains image-pull permission only.

Secrets are not stored in code

The platform needs sensitive configuration, including:

• Tunnel credentials
• Database passwords
• Model-provider credentials
• Service configuration secrets

These are stored in Azure Key Vault rather than:

• Source code
• Git history
• Dockerfiles
• Compose files
• Container image layers
• Browser-side configuration

The VM accesses only the required secrets through its managed identity.

Managed identity instead of embedded credentials

The workload uses a dedicated Azure managed identity.

A managed identity gives the VM a controlled Azure identity without placing a long-lived Azure password or client secret on disk.

The identity has scoped permissions for:

• Pulling approved container images
• Reading only the Key Vault secrets required by the platform

It does not receive broad administrative permissions across the subscription.

This follows the principle of least privilege:

The workload receives only the permissions required to operate.

Telemetry without leaking prompts or secrets

Observability is useful only if it does not become a second data leak.

The evaluation API emits safe operational metadata to OpenTelemetry, such as:

• Evaluation job identifier
• Project identifier
• Environment
• Agent identifier
• Suite version
• Evaluation state
• Score
• Correlation identifier
• Execution timing

The telemetry path is designed not to emit:

• Raw prompts
• Raw model responses
• Authorization headers
• API keys
• Submitted secrets
• Browser authentication data

The OpenTelemetry Collector also applies redaction rules before forwarding telemetry to Phoenix.

This means the UI can answer operational questions such as:

• Did a request reach the platform?
• Did a trace complete?
• Which environment produced the trace?
• Which evaluation suite produced the score?
• How long did it take?
• Did the evaluation pass, fail, block, or become unavailable?

Without automatically storing potentially sensitive agent content.

The Phoenix trace pipeline

The implemented telemetry pipeline is:

Evaluation API
      |
      | Creates safe spans
      v
OpenTelemetry Collector
      |
      | Removes protected fields
      v
Phoenix ingestion endpoint
      |
      v
Phoenix trace interface

The first trace sent through this path was intentionally synthetic.

It included only non-sensitive metadata such as:

• Development environment
• Reference project
• Synthetic validation mode
• Passed status
• A fixed score

No customer prompt, user message, model response, or production agent content was used to validate the pipeline.

This was important because it proved the telemetry integration before real data was introduced.

Incident: troubleshooting a tunnel error

During deployment, the public browser endpoint showed a Cloudflare Tunnel error.

The issue was not a public DNS issue and not an exposed-Azure-port issue.

Cloudflare had the hostname route, but it did not see a healthy tunnel connector.

The root cause had two parts:

1. The moved VM did not receive all expected bootstrap setup.
2. The tunnel process could not read its credential file because the file permissions were too restrictive for the service account.

The resolution was:

• Install the missing runtime dependency.
• Correct the secret materialization process.
• Apply restrictive directory permissions.
• Apply readable-only-by-required-service permissions to the tunnel credential.
• Restart the tunnel connector.
• Verify that Cloudflare reported a healthy connection.
• Verify the browser flow reached Cloudflare Access.

This incident reinforced an important operational principle:

In a tunnel-based design, tunnel health is a production dependency.

The critical health path is no longer “is port 443 open?” It becomes:

Is the connector authenticated?
Is the connector healthy?
Can it reach the local proxy?
Can the proxy reach the correct upstream service?

Current security posture

The remaining hardening work

The current system is a strong access and observability foundation, but it is not the final security state.

The highest-priority next steps are:

1. Connect the API to durable evaluation workers and persistent job state.
2. Add project-specific identity and environment policy enforcement for API clients.
3. Add approval-driven onboarding for each project.
4. Enforce project-level quotas and model-token budgets.
5. Implement tested retention expiry for raw content and backups.
6. Decide whether different teams need separate per-project UI access boundaries.
7. Run capacity testing before broad production adoption.
8. Add alerting for tunnel health, proxy health, collector health, and Phoenix availability.

Final takeaway

The most important thing we built was not just a dashboard.

We built a controlled path from an authenticated team browser to a private Azure observability service:

Authenticated user
        ->
Cloudflare Access
        ->
Cloudflare Tunnel
        ->
Private Azure VM
        ->
Reverse proxy
        ->
Phoenix trace UI

The VM is not public. The databases are not public. The internal service ports are not public. Secrets are not stored in code. Image distribution is private. Telemetry is designed to avoid carrying prompts, outputs, credentials, and headers into the observability layer.

The next phase is evaluation execution and project governance. But before connecting real agent workloads, we now have the secure visibility pipeline required to observe them responsibly.