Google Professional Machine Learning Engineer Question 236
Single answerGoogle Cloud PlatformYou are building a machine learning pipeline to train and evaluate a model for predicting product demand. The dataset contains millions of rows and is stored in BigQuery. You need to preprocess the data, split it into training and evaluation sets, and write the results back to BigQuery for downstream processing. Which approach should you take to ensure scalability and efficiency?
- A
Use BigQuery's SQL capabilities to preprocess the data, split it into training and evaluation sets using SQL queries, and write the results into separate BigQuery tables.
- B
Export the dataset from BigQuery to Cloud Storage, preprocess and split the data locally using Python, and then upload the processed data back to BigQuery.
- C
Use BigQuery ML to preprocess the data and directly train the machine learning model from BigQuery without splitting the data.
- D
Use Apache Beam with Cloud Dataflow to preprocess the data and split it into training and evaluation sets, then write the results back to BigQuery.
Show answer and explanation
Correct answer: A
Explanation
The goal is to preprocess and split the data efficiently for a large dataset stored in BigQuery. BigQuery's native SQL capabilities are optimized for this purpose and avoid the need for data export or additional processing tools. This ensures scalability, efficiency, and simplicity in the pipeline.
- A. Correct.
Correct: BigQuery's SQL capabilities are highly scalable and optimized for large datasets. Using SQL to preprocess and split the data within BigQuery ensures efficiency and avoids unnecessary data movement.
- B. Incorrect.
Incorrect: Exporting the dataset to Cloud Storage and processing it locally introduces unnecessary overhead and is not scalable for large datasets. This method is inefficient and prone to errors when handling millions of rows.
- C. Incorrect.
Incorrect: While BigQuery ML can train models directly within BigQuery, the question specifically asks for preprocessing and splitting the data, which is not the primary use case for BigQuery ML.
- D. Incorrect.
Incorrect: Although Apache Beam with Cloud Dataflow is a scalable option, it adds unnecessary complexity for this use case, as BigQuery's native SQL capabilities can handle the preprocessing and splitting efficiently.