Google Professional Data Engineer Question 237
Select 3Google Cloud PlatformYou are tasked with creating a data processing workflow using Cloud Composer. The workflow involves extracting data from a Cloud Storage bucket, processing it with a Cloud Dataflow job, and storing the results in BigQuery. Which of the following steps are essential to define a valid directed acyclic graph (DAG) in Cloud Composer?
- A
Define a Python file with a unique DAG ID and schedule interval.
- B
Use Airflow operators to define tasks and dependencies between them.
- C
Store the DAG file in the 'dags/' folder of the Cloud Composer environment.
- D
Manually execute each task in the DAG to validate its functionality.
- E
Ensure all tasks are executed in parallel without any dependencies.
Show answer and explanation
Correct answers: A, B, C
Explanation
To create a valid DAG in Cloud Composer, you must define it in a Python file with a unique DAG ID and schedule interval, use Airflow operators to define tasks and dependencies, and store the file in the 'dags/' folder of the Composer environment. Manual execution of tasks and enforcing parallel execution are not essential steps for defining a valid DAG.
- A. Correct.
Correct. A valid DAG in Cloud Composer must have a unique DAG ID and a schedule interval defined in a Python file.
- B. Correct.
Correct. Airflow operators are used to define tasks and set dependencies between them, which form the structure of the DAG.
- C. Correct.
Correct. For the DAG to be recognized and executed by Cloud Composer, it must be stored in the 'dags/' folder of the Composer environment.
- D. Incorrect.
Incorrect. DAGs are scheduled to run automatically based on the schedule interval. Manual execution is not a requirement for defining a valid DAG.
- E. Incorrect.
Incorrect. Tasks in a DAG can have dependencies and do not need to run in parallel. In fact, defining dependencies is a key part of creating a valid DAG.