gcp data engineer Advanced Practice Exam: Hard Questions 2025

A media company is designing a global streaming analytics platform. Events arrive from mobile/TV apps with occasional out-of-order delivery (up to 2 hours late) and must power (1) real-time dashboards with <5s latency, and (2) daily revenue reports that must exactly match finance totals (no double counting). The company needs a single architecture that supports both low-latency and correct historical recomputation when late data arrives. Which design best meets these requirements with minimal operational burden?

A logistics company uses Dataflow streaming to compute per-shipment state from Pub/Sub events. They use a keyed stateful DoFn with timers to emit “shipment delivered” when a sequence is complete. After a pipeline update, they observe duplicate “delivered” outputs and incorrect final states for some keys. Logs show frequent worker restarts and a backlog of unacked messages. The team suspects state inconsistency during retries. Which change most directly addresses correctness under retries and worker restarts?

A financial institution ingests CDC (change data capture) from on-prem databases into Google Cloud. The source produces occasional duplicate events and out-of-order updates for the same primary key. The target is BigQuery, and analysts require a queryable table that always reflects the latest state per key with auditability (full history) and the ability to backfill without downtime. Which ingestion and modeling approach best meets these requirements?

Your team runs a Dataflow streaming job reading Pub/Sub and writing to BigQuery. Suddenly, BigQuery write latency spikes, Dataflow throughput drops, and Pub/Sub backlog grows. BigQuery shows intermittent “quota exceeded” errors for streaming inserts. You must restore near-real-time processing quickly while preserving data (no loss) and minimizing code changes. What is the best remediation?

A healthcare provider must store petabytes of time-series device telemetry with strict data residency (EU only), low-latency point lookups by deviceId+timestamp, and periodic aggregations over large ranges. They also need to delete all data for a patient within 30 days of a request (right-to-erasure), and deletions must not break other devices’ data. Which storage design best balances lookup performance, analytical capability, and deletion requirements?

An ecommerce company uses BigQuery as their enterprise warehouse. They ingest daily snapshots of product catalog data and also stream incremental changes. Analysts complain about inconsistent query results: some queries see partially updated catalog data during ingestion windows. The company needs atomic visibility of each catalog version to downstream queries while keeping the streaming changes. Which approach best provides consistent reads with minimal disruption?

A data science team runs feature engineering in BigQuery for a real-time fraud model. They need point-in-time correct features (no label leakage): for each transaction, compute aggregates over the prior 30 days of customer behavior excluding events after the transaction time. Source events can arrive late, and the training pipeline is re-run frequently. Which solution best enforces point-in-time correctness and supports backfills?

Your organization is migrating from on-prem Hadoop to Google Cloud. They have 5 years of logs stored as compressed files and need: (1) ad-hoc SQL exploration by analysts, (2) repeatable pipelines for curated datasets, and (3) strong governance with column-level security and auditability. The logs are semi-structured (JSON) and schemas evolve frequently. Which approach best satisfies these goals?

A large retailer runs hundreds of BigQuery ETL queries nightly. After a refactor, some pipelines intermittently produce incomplete tables, but there are no query errors. Investigation shows that downstream jobs sometimes start before upstream tables finish loading. You must enforce correct dependencies, add data quality gates (row count and freshness checks), and enable re-runs for a specific date partition without reprocessing everything. What is the best orchestration approach on Google Cloud?

A global IoT platform uses Pub/Sub -> Dataflow -> BigQuery. They require end-to-end exactly-once results for a derived “daily active devices” metric used for billing disputes. They currently compute the metric in a streaming pipeline and write per-device daily records into BigQuery. Occasionally, device records are duplicated due to retries, causing overbilling. They cannot accept any overcount and must be able to prove correctness. What is the best design change?