Google Professional Data Engineer Question 236
Select 4Google Cloud PlatformYou are tasked with building a data pipeline using Cloud Composer to orchestrate workflows. The pipeline performs the following steps: 1) Extract data from a Cloud Storage bucket, 2) Transform the data using a custom Python script, and 3) Load the processed data into BigQuery. To achieve this, you need to create a directed acyclic graph (DAG) in Cloud Composer. Which of the following steps are required to correctly define and deploy the DAG?
- A
Define the DAG structure using Python and the Apache Airflow DAG class.
- B
Store the DAG definition file in the /dags folder of the Cloud Composer environment.
- C
Manually compile the DAG into a binary format before uploading it to Cloud Composer.
- D
Use Airflow operators such as PythonOperator and BigQueryOperator to define tasks within the DAG.
- E
Ensure that task dependencies are defined using the 'set_upstream()' or 'set_downstream()' methods.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
To create a DAG in Cloud Composer, you must define the workflow as Python code using the Apache Airflow DAG class, specify tasks using operators like PythonOperator and BigQueryOperator, and define task dependencies. The DAG file must be placed in the /dags folder of the Composer environment. There is no need to manually compile the DAG as Cloud Composer handles this automatically.
- A. Correct.
Correct. Cloud Composer DAGs are defined using Python and the Apache Airflow DAG class. This is the primary way to create and organize your workflows.
- B. Correct.
Correct. All DAG files must be stored in the /dags folder of the Cloud Composer environment for them to be recognized and executed.
- C. Incorrect.
Incorrect. DAGs do not need to be manually compiled into a binary format. Cloud Composer automatically parses and executes Python-based DAG files.
- D. Correct.
Correct. Airflow operators such as PythonOperator and BigQueryOperator are used to define individual tasks in the DAG, ensuring modular and functional workflows.
- E. Correct.
Correct. Task dependencies must be explicitly defined using methods like 'set_upstream()' or 'set_downstream()', or with shorthand operators (e.g., '>>' or '<<') to establish the execution order.