Introduction
Database selection guidance often reads like a features checklist:
Firestore for flexible documents and real-time sync, Bigtable for
high-throughput analytical workloads. What that guidance doesn't
always convey is how the wrong choice manifests in practice — as
cost growth, query limitations, or both. This post builds a
high-frequency event-logging workload on Firestore, measures where it
breaks down, and migrates the same workload to Bigtable, including the
row-key design work that Bigtable migrations often get wrong on the
first attempt.
Solution overview
Phase A Phase B
┌──────────┐ ┌──────────┐
│ Event │ ~200 writes/s │ Event │ ~200 writes/s
│ Producer │──────────────────▶│ Producer │──────────────────▶
└──────────┘ └──────────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ Firestore │ │ Bigtable │
│ (documents)│ │(wide-column)│
└──────────┘ └──────────┘
Prerequisites
- A Compute Engine VM to run the load-generation script (isolates network variability from your local machine)
- Familiarity with basic Python
Walkthrough
Phase A: Firestore under sustained write load
gcloud services enable firestore.googleapis.com
gcloud firestore databases create --location=asia-southeast1 --type=firestore-native
from google.cloud import firestore
import time, uuid, random, threading
db = firestore.Client()
EVENT_TYPES = ["click", "view", "purchase", "error", "login"]
def write_event():
db.collection("events").document().set({
"event_id": str(uuid.uuid4()),
"user_id": f"user_{random.randint(1, 1000)}",
"event_type": random.choice(EVENT_TYPES),
"timestamp": firestore.SERVER_TIMESTAMP,
})
def worker():
while True:
write_event()
time.sleep(0.05)
for _ in range(10):
threading.Thread(target=worker).start()
Run this for 15–20 minutes to generate roughly 200,000 documents, then
attempt an aggregate query:
docs = db.collection("events").where("event_type", "==", "click").stream()
count = sum(1 for _ in docs)
Observed limitations:
- Multi-field queries frequently require a manually created composite index, surfaced through an error message with an index-creation link
- Aggregate counts over large result sets incur a read operation per document, which is directly reflected in Billing Reports under Cloud Firestore
Review actual cost in Billing → Reports, filtered to Cloud
Firestore, before proceeding to the migration.
Phase B: Migrating to Bigtable
Row-key design is the primary design decision in this migration.
A row key derived directly from a monotonically increasing value (such
as a raw timestamp) causes writes to concentrate on a single tablet —
a pattern known as hotspotting — because Bigtable stores rows in
lexicographic key order.
A row key that leads with a high-cardinality field, such as
user_id, distributes writes across the keyspace:
row_key = f"{user_id}#{reversed_timestamp}".encode()
Where reversed_timestamp = 9999999999999 - unix_timestamp_ms,
placing the most recent event first within a given user's key range.
gcloud services enable bigtable.googleapis.com bigtableadmin.googleapis.com
gcloud bigtable instances create event-log-bt \
--cluster=event-log-cluster \
--cluster-zone=asia-southeast1-b \
--cluster-num-nodes=1 \
--instance-type=PRODUCTION
cbt -instance=event-log-bt createtable events
cbt -instance=event-log-bt createfamily events cf1
from google.cloud import bigtable
import time, random
client = bigtable.Client(project="<PROJECT_ID>", admin=True)
table = client.instance("event-log-bt").table("events")
def write_event():
user_id = f"user_{random.randint(1, 1000)}"
ts_ms = int(time.time() * 1000)
row_key = f"{user_id}#{9999999999999 - ts_ms}".encode()
r = table.direct_row(row_key)
r.set_cell("cf1", "event_type", random.choice(["click", "view", "purchase"]))
r.commit()
Validating key distribution: Run the same load twice — once with a
naive timestamp-only key, once with the user_id-prefixed key — and
compare per-node CPU utilization in Bigtable → Monitoring. A
poorly distributed key shows a pronounced skew toward a single node;
a well-distributed key shows roughly even utilization across the
cluster.
Cost comparison
Bigtable bills for provisioned node-hours regardless of traffic
volume, while Firestore bills per operation. For short-duration,
moderate-volume workloads like this walkthrough, Bigtable may appear
more expensive in absolute terms — Bigtable's cost efficiency
advantage typically emerges only at sustained high throughput (commonly
millions of operations per day). Record both figures from Billing
Reports for a direct comparison.
Clean up resources
gcloud bigtable instances delete event-log-bt --quiet
Firestore data must be deleted at the collection level, or by removing
the entire test project.
Conclusion
Neither database is universally superior — the right choice depends on
access patterns and scale, not feature checklists alone. This
walkthrough demonstrates two concrete decision points worth
internalizing: composite index requirements as an early signal that
Firestore's query model may not fit the workload, and row-key design
as the single highest-leverage decision in any Bigtable schema.
In Part 5, we deploy to GKE and introduce a bug that only
manifests after the container has already passed its initial health
check.
