Google Cloud Data Agent Kit: Build Pipelines from Your IDE Without the Boilerplate

Originally published on DevToolHub.

Writing data pipeline code is repetitive. You pull raw files from Cloud Storage, figure out the schema, decide whether to push through BigQuery or Spark, write the SQL or PySpark, wire up the orchestration, add the governance checks — and repeat this for every new dataset. Most of that work is boilerplate that follows known patterns.

Google Cloud Data Agent Kit is an open-source suite of AI skills, MCP tools, and IDE plugins that drops into VS Code and Claude Code to handle that boilerplate for you. You describe what you want to do with your data. The agents figure out the execution path.

[IMAGE: articles/images/2026-07-05-google-cloud-data-agent-kit-featured.png | alt: "google cloud data agent kit in VS Code IDE showing BigQuery pipeline generation with AI agents"]

What the Google Cloud Data Agent Kit Actually Does

The kit has four distinct layers:

Agentic skills — pre-codified workflows for the tasks you run repeatedly: query optimization, data validation, drift detection, ML model lifecycle management, and governance enforcement. These are the "skills" that agents draw on when you give them a goal.

MCP tools — Model Context Protocol connections that give the agents secure, authenticated access to your cloud data services (BigQuery, AlloyDB, Spanner, GCS) without requiring you to copy credentials into your editor or manually configure connection strings.

IDE plugins — native integrations for VS Code, Claude Code, Gemini CLI, Codex, and Antigravity CLI. The plugin surfaces a Unified Data Hub — a single view of your data estate, including datasets, orchestration pipelines, and active jobs.

Intelligent routing — the agents automatically pick the right compute engine for each task. BigQuery handles SQL-native analytics and ELT. Spark handles custom Python transformations and distributed ML training. You don't configure this — the agent chooses based on what the task requires.

The kit itself is in preview, but the two agents it packages are GA:

• Data Engineering Agent — builds pipeline transformations from natural language descriptions and enforces governance rules
• Data Science Agent — manages the full model lifecycle from data wrangling through training, scaling across BigQuery Dataframes and Serverless Apache Spark

Installing Google Cloud Data Agent Kit in VS Code and Claude Code

VS Code:

1. Open the Extensions panel (Ctrl+Shift+X / Cmd+Shift+X)
2. Search for "Google Cloud Data Agent Kit"
3. Install and authenticate with your Google Cloud account

Setup takes under a minute and configures the MCP connections automatically.

Claude Code:

Install via the Claude Code Plugin system. Once installed, Data Agent Kit's tools appear in Claude Code's available MCP tools, and you can invoke the Data Engineering Agent directly from the CLI.

Both environments give you access to the same agents and MCP connections — the difference is UX. VS Code gives you the Unified Data Hub sidebar for browsing datasets. Claude Code gives you conversational access from the terminal.

⚠️ Note: You need an active Google Cloud project with BigQuery API enabled. The agents authenticate with your existing gcloud auth credentials — no separate service account setup required for initial testing.

Building a Data Pipeline: What It Looks Like End to End

A real example from the kit's starter pack: starting with raw files in Cloud Storage and ending with batch inference results in Cloud Spanner — without leaving the IDE.

The Data Engineering Agent handled:

1. Creating a Spark notebook for initial data ingestion
2. Setting up Iceberg table management in BigQuery for the processed data
3. Generating a dbt project for the transformation layer
4. Wiring up an end-to-end orchestration pipeline

The Data Science Agent then:

1. Trained a XGBoost model on the processed dataset
2. Ran batch inference at scale
3. Pushed results to Cloud Spanner

Each step was triggered through natural language in the IDE — "create a pipeline to ingest these GCS files into BigQuery with Iceberg table management" — not by writing the pipeline code manually.

That doesn't mean zero code is involved. The agents generate code that you review and run. But the scaffolding, connection boilerplate, and orchestration wiring is generated rather than hand-written.

Connecting to BigQuery, GCS, and Spanner via MCP

The MCP layer is what makes the agents actually useful rather than just generating generic code. Because the agents connect directly to your cloud services, they can:

• Inspect your actual dataset schemas before generating transformation code
• Validate that generated SQL runs against your real BigQuery tables
• Read GCS file metadata to determine the right ingestion approach
• Check existing orchestration jobs before creating new pipeline steps

This matters because a generic AI assistant generating pipeline code doesn't know your actual schema or data. The MCP connection gives the agents context about your real data estate. For a deeper look at how MCP works, see what is Model Context Protocol.

For GCP data management compared to AWS and Azure, Data Agent Kit is a BigQuery-native integration — it works best in environments already invested in GCP data services.

[IMAGE: articles/images/2026-07-05-google-cloud-data-agent-kit-diagram.png | alt: "MCP architecture connecting VS Code to BigQuery, GCS, AlloyDB, and Spark via Data Agent Kit"]

When to Use It and When to Write Pipeline Code Yourself

Google Cloud Data Agent Kit is genuinely useful for:

Prototyping new pipelines — instead of spending an hour setting up the scaffold before you can even test your logic, you describe the data flow and get working code in minutes. Then you refine it.

Repeated ad-hoc data exploration — querying GCS files, visualizing dataset distributions, pulling quick stats without leaving your editor.

Standard transformation patterns — dbt project setup, Iceberg table management, standard ELT flows. These are well-understood patterns that the agents handle reliably.

It's less useful for:

Highly custom orchestration — if your pipeline has unusual scheduling requirements, specific failure recovery logic, or dependencies across multiple data systems with complex ordering, the generated scaffolding becomes a starting point rather than a finished product.

Performance-critical workloads — generated Spark code is correct but not always optimally tuned. You'll still need to profile and optimize queries that run at scale with strict SLA requirements.

The honest framing: this is a tool that eliminates boilerplate, not business logic. The parts of pipeline engineering that require understanding of your specific data, compliance requirements, and performance constraints still require human judgment.

Official documentation: Google Cloud Data Agent Kit extension docs and the GCP blog announcement.

Frequently Asked Questions

Q: Does Google Cloud Data Agent Kit work with non-GCP data sources?
A: The current version focuses on GCP services — BigQuery, AlloyDB, Spanner, GCS, and Serverless Apache Spark. The MCP tools are built specifically for GCP authentication and API patterns. Cross-cloud connections aren't supported in the current release.

Q: Is the Data Agent Kit free to use?
A: The kit itself is open-source and free to install (currently in preview). You pay for the underlying GCP services (BigQuery queries, Spark compute, GCS storage) at standard GCP rates. There's no additional charge for the Data Engineering Agent or Data Science Agent.

Q: Can I use it with Gemini CLI instead of VS Code?
A: Yes. The Data Agent Kit supports Gemini CLI, Claude Code, VS Code, Codex, and Antigravity CLI. The agents and MCP connections work the same across environments.

Q: What happens to my data when the agents access it?
A: The MCP tools use your existing Google Cloud credentials and IAM permissions. The agents don't copy data to external systems — they read metadata and execute queries against your GCP services within the permissions of your authenticated account.

Q: Does this replace dbt or Dataform for transformations?
A: No. The Data Engineering Agent can generate dbt projects as part of a pipeline setup, but it doesn't replace dbt as a transformation framework. It uses dbt for the parts where dbt is the right tool — the same way an experienced engineer would.

Quick Summary:

• Google Cloud Data Agent Kit is an open-source suite of AI agents and MCP tools that integrates into VS Code and Claude Code
• Two GA agents: Data Engineering Agent (pipelines, transformations, governance) and Data Science Agent (model training, inference)
• MCP connections give agents real context about your actual schemas, datasets, and existing jobs — not just generic code generation
• Intelligent routing picks BigQuery for SQL analytics and Spark for Python/ML workloads automatically
• Best for prototyping, exploration, and standard transformation patterns — complex orchestration still needs manual refinement
• Free to install; you pay standard GCP rates for the underlying services