Building a GCP-native data platform from zero

Architected and operate a GCP data platform processing millions of events per month, with cross-cloud CDC ingestion and SOC2-aligned governance.

• GCP
• BigQuery
• Dataflow
• Dataform
• Cloud Composer
• Datastream CDC
• Terraform
• Terraform Cloud

Context

In my current role, I own the data platform: a GCP-native analytics stack ingesting operational data from multiple source systems into a unified BigQuery warehouse.

Problem

Building a production-grade data platform from an empty GCP project meant solving, at once: reliable ingestion from systems living in a different cloud, a governance model that satisfies SOC2 without slowing analysts down, and infrastructure that a small team can actually operate.

Approach

• Designed a Medallion architecture (Bronze/Silver/Gold) using BigQuery, Dataflow, and Dataform.
• Deployed Datastream CDC for real-time replication from AWS-hosted PostgreSQL/MariaDB into BigQuery over a private network path.
• Orchestrated ingestion and transformation with Cloud Composer (Airflow).
• Made Terraform (via Terraform Cloud, GitHub-OIDC authenticated — no long-lived cloud credentials) the only way infrastructure changes ship, with CI gates for linting, security scanning, and coverage.
• Built the SOC2/PCI-DSS-aligned governance layer: least-privilege IAM, Zero Trust review, audit-ready logging.

Outcome

A platform that runs the company’s operational analytics with high reliability, passed SOC2 controls review, and cut infrastructure deployment cycles from days to hours.

Lessons

Making Terraform Cloud the only path to production infrastructure changes — with GitHub-OIDC auth and no long-lived credentials — pays off in audit readiness and in how confidently a small team can operate a platform this size.