Two Clouds, One Identity Crisis: AWS vs. GCP IAM for Data Engineers

AWS IAM and GCP IAM solve the same problem — who can do what, to which resource — and disagree about almost everything else. AWS starts from identity and asks what it's allowed to touch. GCP starts from a resource hierarchy and asks who's standing in the right branch of the tree. Everything downstream of that one design choice, from how you debug a denial to how you structure a team's projects, follows from it.

Both systems have a talent for producing a confident AccessDenied at the worst possible moment, and most of the resulting confusion isn't because either one is badly designed — it's that they're solving the same problem from opposite directions, and the vocabulary doesn't line up. This article is the map: how each cloud actually thinks about identity and access, why the same words mean dangerously different things across them, and what changes in your daily work when your stack is Athena and Redshift on one side and BigQuery on the other.

IAM is nobody's favorite topic. Let's make it a fast one.

1. The Mental Model: A Fenced Yard vs. a Family Tree

Before any diagram makes sense, you need the core intuition, because everything else follows from it.

AWS is identity-centric. An AWS account is a fenced yard. Inside the yard live identities — IAM users, IAM roles, groups — and each identity carries a keychain of policies that say what it may touch. Resources can also have their own opinions: an S3 bucket policy or a KMS key policy can invite or reject principals on its own. Every request is made by an identity, and the answer is computed from the policies attached to everyone involved.

GCP is resource-centric. GCP doesn't really have "users" in IAM at all. Humans are emails from your Google Workspace or Cloud Identity domain, and there are no IAM groups living inside GCP either — groups are managed outside and referenced by email. What GCP does have is a strict resource hierarchy: Organization → Folders → Projects → Resources. Permissions are defined as bindings attached to a node of that tree, and they flow downhill. A binding at the organization level rains down on every project beneath it.

My favorite way to compress it: in AWS you authenticate into the yard and carry keys; in GCP you already are somebody (an email), and the only question is which branches of the tree have your name on them.

In the AWS yard, access is negotiated between the identity's policy and the resource's policy. In the GCP tree, a binding anywhere above you applies to you, and (brace yourself) there is no way to subtract it at a lower level. We'll get to that.

2. The Terminology Trap (or: "Role" Is a Trap)

Here is where most cross-cloud confusion is born, so let's defuse it with a table:

So when a GCP person says "I gave the pipeline the BigQuery role," they mean something closer to "I attached an AWS managed policy for BigQuery access." And when an AWS person says "assume the role," a GCP person would say "impersonate the service account."

One small aesthetic difference you'll hit immediately in scripts and Terraform: AWS permissions look like s3:GetObject — service, colon, verb. GCP permissions look like bigquery.tables.getData — service, dot, resource, dot, verb. You will typo both approximately forever.

3. How a Request Gets Answered

Both clouds ultimately run your request through a little decision engine. The engines rhyme, but the plumbing differs.

AWS: "no" beats "yes", silence means "no"

When your request hits AWS, it gathers every policy that applies: the identity policy of the caller, the resource policy of the target, Service Control Policies above the account, permission boundaries if any. Then:

The golden rule: explicit deny wins over everything, allows accumulate, and if nothing says yes, the answer is no. The union-of-allows part is genuinely nice — you can grant one action from one policy and another action from a second policy, and the identity gets both.

GCP: walk up the tree, add everything up

GCP checks a single permission (like bigquery.tables.getData) for a single principal. It collects every binding from the resource and all its ancestors — table, dataset, project, folders, organization — and unions them together. Deny policies, if your org uses them, are evaluated first and override any allow. IAM Conditions can additionally scope a binding (by resource name prefix, by time of day, whatever your security team dreams up in CEL syntax).

The gotcha that bites AWS people: inheritance is additive and there's no per-resource veto. If someone binds a broad role at the folder level, every project under that folder has it, and you cannot cancel it lower down with a "smaller" binding. Deny policies help, but they cover a defined set of permissions. Meanwhile in AWS, the account is a hard wall — nothing you do in one account affects another.

Practical translation for data folk: in GCP, treat your project structure as your security architecture. Where you would spin up separate AWS accounts for prod and non-prod (a very good habit), in GCP you spin up separate projects — they're nearly free, they're the natural blast radius boundary, and they're the unit everything (billing, APIs, service accounts, quotas, and yes, IAM) hangs off of.

4. Day One: Getting Humans Through the Door

On AWS, the modern answer for humans is IAM Identity Center (the artist formerly known as AWS SSO). You log in to a portal with your corporate identity, pick an account, and land in a role session with a permission set governing what you can do. Long-lived IAM users with passwords are officially in the "please stop" category.

On GCP, there is no portal dance: you log into the console with your Google Workspace account, full stop. Access is simply a question of whether someone added your email — or better, one of your Google Groups — to a binding at the right node of the tree.

Both clouds converge on the same advice, and it's worth internalizing: humans federate, they don't get local credentials, and permissions go to groups, never to named individuals. The person who left the company two years ago should not still be an entry in an ACL, and the auditor who asks about it is always right.

5. Your Pipeline Needs an Identity (and It Is Not a Password)

Here's the part that matters most for us. Data pipelines are workloads, and both clouds landed on the same beautiful pattern for them: no stored credentials at all — short-lived tokens acquired at runtime.

On AWS: the role, the profile, and the metadata service

You create an IAM role with a trust policy saying "compute services may become me" and a permission policy saying what "me" may do. Attach it to the compute — an instance profile on EC2, a task role on ECS, an execution role on Lambda — and the application never sees a secret:

The SDKs do this automatically. Your code just calls boto3.client("s3") and the keychain materializes. The classic AWS horror story — a credentials.csv with a long-term access key committed to a repo — is exactly what this pattern makes unnecessary.

A peek at what the permission side looks like, for a pipeline that reads a raw events bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::acme-raw-events",
        "arn:aws:s3:::acme-raw-events/events/2026/*"
      ]
    }
  ]
}

On GCP: the service account and the metadata server

The GCP equivalent is a service account — an email like pipeline@acme-prod.iam.gserviceaccount.com. You attach it to your VM, your Dataflow job, your Composer (Airflow) environment, and the application fetches tokens from the metadata server:

Granting it access to read a dataset looks like this:

gcloud projects add-iam-policy-binding acme-prod \
  --member "serviceAccount:pipeline@acme-prod.iam.gserviceaccount.com" \
  --role "roles/bigquery.dataViewer"

GCP has its own version of the credentials.csv horror: service account JSON key files. They're technically sometimes necessary, but the default posture is the same as AWS's: if your workload runs in GCP, attach the service account and let the metadata server handle it. And to attach or impersonate a service account, you need iam.serviceAccounts.actAs — a permission that matters more than it sounds, because whoever has it effectively has everything the service account has.

The BigQuery nuance everyone hits once

If your warehouse is BigQuery, here's a rite of passage: you can see the table but the query fails (or vice versa). That's because running a query needs two different grants in two different places:

• roles/bigquery.jobUser on the project — for the right to run a job at all
• roles/bigquery.dataViewer on the dataset — for the right to read the data

Contrast with Redshift, which sits behind IAM for infrastructure access but keeps its own database users and grants inside — IAM gets you to the door, and the database has its own bouncer. BigQuery is IAM: the door is the warehouse. (Athena is IAM-native too, which is why your Athena permissions feel suspiciously like your S3 permissions.)

One more gift both clouds give modern teams: workload identity federation. Point either cloud at an OIDC provider — say, GitHub Actions — and your CI can exchange a GitHub-issued token for cloud credentials. No keys in secrets managers, no rotation ceremonies.

6. Sharing Data Across Teams

So the marketing analysts want your daily_revenue table. Who could say no? (IAM, obviously.)

On AWS, cross-team usually means cross-account, and the dance is role assumption: your account trusts a role ARN in their account (or vice versa), someone assumes it, and temporary credentials are minted for the session. Resource policies — bucket policies granting another account's principals directly — are the other common move. Either way, the account boundary keeps blast radius small, which is why "one account per team/environment" is such durable advice.

On GCP, sharing is just another binding, and the granularity is delightful because BigQuery is IAM-native: grant dataViewer on a single dataset or even a single table to the analysts' group. Two GCP-specific gems worth knowing:

• Authorized views: you can expose a view without granting any access to the underlying tables. Perfect for "here's the aggregated, PII-free version."
• Cross-project queries: reference other-project.dataset.table and query it like it's local — if the binding allows it. No data copying, no export jobs.

7. Columns, Rows, and the PII Question

Eventually someone asks, "exactly who can read the email column?" and you get to find out together.

On AWS, the fine-grained story for data lakes is Lake Formation: a permissions layer sitting on top of S3 and the Glue Data Catalog that understands databases, tables, and columns. This produces my favorite AWS support ticket of all time: you have s3:GetObject on the bucket, IAM says yes... and Athena still denies you. Why? Because the location is registered with Lake Formation and you're opted in to LF enforcement there, so LF wants its own grant. Two doormen, one door, and both must recognize you. (Once you know this, it's fine. Arguably even nice — column-level grants are worth it.)

One caveat worth knowing before you hit it in production: since hybrid access mode reached general availability in late 2023, registering a location with Lake Formation is a choice between two modes, not an automatic escalation — you can opt individual principals into Lake Formation enforcement while everyone else keeps going through IAM and S3 permissions alone, uninterrupted. So the "two doormen" gotcha isn't universal the moment a location touches Lake Formation; it depends on which mode your data lake admin chose at registration, and who's been opted in since. Worth checking both before you assume you've found the culprit.

On GCP, the equivalents are policy tags (attach a tag to a column in Dataplex/Data Catalog, and reading that column requires membership in the tagged access group — column-level security enforced by BigQuery itself), plus row-level security policies and dynamic data masking on tables. The mental shift: instead of a separate permission system layered on the storage, the fine-grained controls live inside the warehouse and speak IAM.

8. The 3 a.m. Toolkit: Debugging a Denial

When (not if) the denial arrives, here's your toolbox.

On AWS:

• CloudTrail — every denied API call is logged with the full context. This is your source of truth for "who, what, when, denied."
• IAM Policy Simulator — "would this role be allowed to call this action on this resource?" without running anything real.
• IAM Access Analyzer — scans for grants that reach outside your zone of trust, and helps you prove least privilege.

On GCP:

• Policy Troubleshooter — genuinely one of the best debugging tools in any cloud. You give it a principal, a resource, and a permission; it walks the hierarchy and shows you exactly which binding allowed or denied it, and why. The first time you use it after years of CloudTrail archaeology, you may get emotional.
• Cloud Audit Logs — the CloudTrail equivalent, queryable, and yes, exportable into BigQuery so you can audit your cloud with your warehouse. Eat your own cooking.
• IAM Recommender — looks at actual usage and suggests shrinking roles nobody fully uses. Least privilege, but automated.

9. What Survives Both Clouds

Strip away the terminology and the two systems are agreeing on a philosophy. Here's the checklist that translates directly:

• No long-lived credentials. Workloads assume roles (AWS) or attach service accounts (GCP); humans federate; CI uses workload identity federation. If it's a key in a file, it's a finding.
• Grant to groups and service accounts, never individuals. Person-based grants are how ghosts end up in your access review.
• Environments are boundaries. AWS accounts or GCP projects — separate them like prod matters, because it does.
• Structure follows the hierarchy. On GCP, folder and project layout is the security architecture; inheritance is additive and unforgiving.
• Least privilege is a process, not a state. Access Analyzer and IAM Recommender exist precisely because humans are bad at guessing needed permissions.
• Everything as code. You already version your DAGs and dbt models; policies that drift via console clicking will eventually deny a pipeline at 3 a.m. Terraform it.

10. Parting Thoughts

AWS IAM and GCP IAM are two dialects of the same language: who can do what to which thing, and can you prove it. AWS speaks it through identities carrying policies, negotiated against resources, walled off by accounts. GCP speaks it through emails in bindings, raining down a resource tree. Neither is harder, exactly — they're differently surprising.

And the next time AccessDenied shows up in your logs, you won't panic. You'll open CloudTrail or Policy Troubleshooter and remind yourself: it's not a bug, it's a policy. Somewhere, in a JSON document or an IAM binding, someone wrote the word "no" — and now you know exactly where to look.

May all your tokens be short-lived.