Redshift Spectrum + Lake Formation — Enterprise Governance on NAS Data

TL;DR

In Part 1, Athena provided serverless SQL. In Part 2, Databricks hit boundaries. In Part 3, Snowflake works with config. In Part 4, DuckDB Lambda was cheapest. In Part 5, EMR Spark delivered full ETL. This Part 6 adds enterprise governance: Redshift Spectrum + Lake Formation provides 4-layer authorization on NAS data.

Redshift Serverless (8 RPU) successfully queries FSx for ONTAP data via S3 Access Points using the same Glue Catalog tables as Athena — no additional data registration needed. Add Lake Formation on top for table-level, column-level, and tag-based access control.

~2x slower than Athena for simple scans (Redshift Serverless cold start overhead), but Redshift adds DWH capabilities: federated JOINs with local tables, materialized views, and stored procedures.

Quick Decision Guide:

• Need DWH JOINs with NAS data → Redshift Spectrum (this article)
• Need enterprise governance (table/column/tag) → Add Lake Formation
• Need serverless SQL only (no DWH) → Use Athena (Part 1) — faster and cheaper

GitHub: fsxn-lakehouse-integrations

How to Read This Article

This article is:

• A reproduction-focused validation report
• Evidence from one environment (Redshift Serverless 8 RPU, ap-northeast-1)
• A governance architecture guide for Lake Formation + FSx S3 AP

Read by role:

• DWH engineer: Architecture → Setup → Benchmark Results
• Security / governance reviewer: 4-Layer Authorization → Governance Impact
• Data engineer: When to Use → Comparison with Athena
• Partner / SA: Partner Decision Card → Discovery Questions

Prerequisite Concepts

Before reading this article, it helps to understand:

• Redshift Spectrum — Redshift's ability to query data in S3 via external schemas (Glue Catalog)
• Redshift Serverless — pay-per-query Redshift without cluster management (measured in RPU)
• Lake Formation — AWS's centralized governance layer for data lakes (table/column/tag permissions)
• Glue Catalog — AWS's metadata catalog (shared by Athena, Redshift Spectrum, EMR, Glue)
• External Schema — a Redshift schema that maps to a Glue Catalog database

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  Redshift Serverless (8 RPU)                                     │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  SQL Query                                                │   │
│  │  SELECT * FROM fsxn_spectrum.sensor_readings              │   │
│  │         JOIN local_table ON ...                            │   │
│  └──────────────────────────────────────────────────────────┘   │
│                          │                                       │
│              External Schema (Glue Catalog)                       │
└──────────────────────────┼───────────────────────────────────────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
    Lake Formation    IAM Role    S3 Access Point
    (table/column     (API        (resource
     permissions)      access)     policy)
              │            │            │
              └────────────┼────────────┘
                           │
                           ▼
              FSx for ONTAP Volume (Parquet files)

4-Layer Authorization:

1. Lake Formation — Who can access which tables/columns (fine-grained)
2. IAM — Who can call which AWS APIs
3. S3 Access Point Policy — Which principals can access this access point
4. File System — UNIX permissions on the underlying files

Benchmark Results

Environment: Redshift Serverless 8 RPU, ap-northeast-1. FSx for ONTAP Single-AZ, 128 MB/s.

Performance note: Redshift Serverless has cold start overhead (~2-3s for first query). Warm queries on provisioned Redshift clusters would be faster. For simple scans, Athena is ~2x faster because it has no DWH initialization overhead.

Evidence Matrix

Setup

Step 1: Create External Schema (reuses Glue Catalog)

CREATE EXTERNAL SCHEMA fsxn_spectrum
FROM DATA CATALOG
DATABASE 'fsxn_athena_verification'
IAM_ROLE 'arn:aws:iam::<ACCOUNT_ID>:role/fsxn-redshift-spectrum-role'
REGION 'ap-northeast-1';

Key insight: This uses the same Glue Catalog database that Athena uses. No additional table registration needed — if Athena can query it, Redshift Spectrum can too.

Step 2: Query FSx for ONTAP Data

-- Simple count
SELECT COUNT(*) FROM fsxn_spectrum.sensor_readings;
-- Result: 10000 (3,231ms)

-- Aggregation
SELECT status, COUNT(*), AVG(temperature)
FROM fsxn_spectrum.sensor_readings
GROUP BY status;
-- Result: 3 groups (2,580ms)

-- JOIN with local Redshift table (DWH capability)
SELECT s.device_id, s.temperature, d.location
FROM fsxn_spectrum.sensor_readings s
JOIN device_master d ON s.device_id = d.device_id
WHERE s.temperature > 35;

Step 3: Add Lake Formation Governance

# Set Lake Formation admin
aws lakeformation put-data-lake-settings \
  --data-lake-settings '{"DataLakeAdmins": [{"DataLakePrincipalIdentifier": "arn:aws:iam::<ACCOUNT_ID>:user/<admin>"}]}'

# Grant table-level SELECT to a role
aws lakeformation grant-permissions \
  --principal '{"DataLakePrincipalIdentifier": "arn:aws:iam::<ACCOUNT_ID>:role/fsxn-analyst-role"}' \
  --resource '{"Table": {"DatabaseName": "fsxn_athena_verification", "Name": "sensor_readings"}}' \
  --permissions '["SELECT", "DESCRIBE"]'

Lake Formation Data Permissions: fine-grained table-level SELECT grant for fsxn-athena-glue-role on sensor_readings table.

Lake Formation Governance Value

Fine-Grained Governance — Verified (May 2026)

All three fine-grained Lake Formation capabilities have been validated on FSx for ONTAP S3 AP data:

Governance implication for regulated workloads: Lake Formation on FSx for ONTAP S3 AP data provides the same fine-grained access control as on native S3 data. Column masking, row filtering, and tag-based classification all work without data movement. This is the strongest AWS-native governance path for FSx for ONTAP data.

Iceberg + Lake Formation path: Glue Data Catalog supports Iceberg table registration natively. For transactional workloads requiring ACID guarantees: sync FSx for ONTAP data to S3 via DataSync → write as Iceberg table (EMR Spark) → register in Glue Catalog → query via Redshift Spectrum with full Lake Formation governance (column/row/tag). This provides the best of both worlds: FSx for ONTAP as source of truth + Iceberg ACID + Lake Formation governance.

Enterprise governance use cases:

• Healthcare: Column-level masking of PHI fields (e.g., hide patient_name from analysts)
• Finance: Row-level filtering by business unit (each team sees only their data)
• Public sector: LF-Tag classification enforcement (sensitivity: public/internal/confidential)

Comparison with Other Engines in This Series

Partner Decision Card

Discovery Questions for Partners

When a customer asks about Redshift Spectrum + Lake Formation + FSx for ONTAP S3 AP:

1. Does the customer already have a Redshift cluster or Serverless workgroup? (If yes, adding Spectrum is trivial)
2. Do they need to JOIN NAS data with existing DWH tables? (This is Redshift Spectrum's unique value)
3. Is table/column-level governance required? (Lake Formation adds this layer)
4. Is the workload read-only analytics, or does it need write-back? (Spectrum is read-only from external data)
5. What is the query frequency? (For < 10 queries/day, Athena is cheaper)
6. Is cross-account data sharing needed? (Lake Formation simplifies this)
7. Are there compliance requirements for column-level masking? (Lake Formation provides this)
8. What is the acceptable query latency? (Redshift Serverless has ~3s cold start)

Governance Impact Summary

Key insight: Redshift Spectrum and Athena share the same Glue Catalog and Lake Formation permissions. Governance configured for one automatically applies to the other. This means you can use EMR Spark for write-back, register output in Glue, apply Lake Formation permissions, and query from both Athena and Redshift Spectrum with the same governance.

AI Readiness Score

Scoring methodology: Redshift Spectrum + LF scores highest on Governance (same as Athena + LF) but lower on Cost and Simplicity due to RPU pricing and DWH management overhead. Choose Redshift Spectrum when DWH JOINs are required; choose Athena when serverless SQL is sufficient.

Cost Analysis

Monthly estimate (100 queries/day, avg 5s each):

• 100 queries × 5s × 8 RPU × $0.36/RPU-hour ÷ 3600 = ~$0.40/day = ~$12/month

Compare with:

• Athena (same queries): ~$5/TB × data scanned
• DuckDB Lambda: ~$1.10/month (but no DWH JOINs)

When Redshift Spectrum is cost-justified: When you already have Redshift and need to JOIN NAS data with local tables. The marginal cost of adding Spectrum queries is low.

When to Use (and When Not To)

Use Redshift Spectrum + Lake Formation when:

• Customer already has Redshift (adding Spectrum is trivial)
• Need to JOIN NAS data with DWH tables
• Enterprise governance (table/column/tag) is required
• Cross-account data sharing is needed
• Compliance requires column-level masking

Don't use when:

• Simple serverless SQL is sufficient (use Athena — faster, cheaper)
• Need write-back to FSxN (use EMR Serverless)
• Need sub-second latency (use DuckDB Lambda)
• No existing Redshift investment (Athena is simpler to start)
• Dataset is small and ad-hoc (DuckDB Lambda is cheapest)

Known Failure Signatures

What's Next

• Part 7: Table Format Boundaries — why Delta, Iceberg, and Hudi can't write to FSx S3 AP, and what flat Parquet patterns work instead (critical knowledge for architecture decisions)

Previously in this series:

• Part 1: Athena — Query NAS Data In Place
• Part 2: Databricks — A Layer-by-Layer Validation of Observed Boundaries
• Part 3: Snowflake — From 'Access Denied' to Working External Tables
• Part 4: DuckDB Lambda — Serverless Analytics for $0.00001/Query
• Part 5: EMR Spark — Read-Write ETL on NAS Data

References

• Redshift Spectrum documentation
• AWS Lake Formation documentation
• FSx for ONTAP S3 Access Points
• Redshift Serverless documentation
• GitHub: fsxn-lakehouse-integrations

Key achievement: This validation established that Redshift Spectrum + Lake Formation provides the strongest enterprise governance path for FSx for ONTAP S3 AP data — 4-layer authorization (Lake Formation → IAM → S3 AP → File System), table/column-level access control, and seamless sharing of Glue Catalog with Athena. The same governance configuration applies to both Athena and Redshift Spectrum queries, enabling a unified governance model across query engines.

All benchmarks are from a specific test environment (Redshift Serverless 8 RPU, FSx for ONTAP Single-AZ 128 MB/s, ap-northeast-1). Performance improves with warm queries and provisioned clusters.

Disclaimer: This article is an independent validation report and does not represent AWS or NetApp official guidance. Product behavior and platform capabilities may change. Always validate in your own environment.