Databricks Data Engineer Associate Question 364
Select 3You are tasked with setting up a new Delta Live Tables (DLT) pipeline to process streaming data in Databricks. Which of the following components are necessary to create and configure the pipeline?
- A
A JSON or YAML configuration to define the pipeline settings, such as input source and target
- B
A cluster policy to enforce compute resource limits
- C
A notebook or Python script defining the transformation logic using DLT syntax
- D
A database to store the pipeline's metadata and results
- E
A storage location to store the pipeline's checkpoints and logs
- F
A custom library installed in the cluster to process the data
Show answer and explanation
Correct answers: A, C, E
Explanation
To create a DLT pipeline, you need to define the pipeline settings using a JSON or YAML configuration, provide a notebook or Python script with DLT transformations, and specify a storage location for checkpoints and logs. These components ensure the pipeline can process and monitor data effectively. Other options, like cluster policies or custom libraries, are not mandatory for DLT pipeline creation.
- A. Correct.
A JSON or YAML configuration is required to define the pipeline's settings, such as data sources, targets, and mode of execution.
- B. Incorrect.
Cluster policies are not required for creating a DLT pipeline, although they can be used to manage resource allocation in Databricks.
- C. Correct.
A notebook or Python script that uses DLT syntax is essential to define the transformation logic and data flow.
- D. Incorrect.
While the pipeline may write results to a database, the database itself is not a required component for creating the pipeline.
- E. Correct.
A storage location is necessary to store the pipeline's checkpoints and logs for recovery and monitoring purposes.
- F. Incorrect.
Custom libraries are not required; DLT provides built-in functionality for processing data.