Databricks Data Engineer Associate Question 366
Select 2You are tasked with creating a new Databricks Delta Live Tables (DLT) pipeline. Which of the following components are necessary to successfully define and deploy the pipeline?
- A
A storage location for the pipeline's metadata and logs
- B
A cluster definition to execute the pipeline
- C
The pipeline's JSON configuration file defining input tables and transformations
- D
A notebook or Python script containing data transformation logic
- E
A Databricks SQL query to monitor the pipeline's output
Show answer and explanation
Correct answers: A, D
Explanation
To create a new DLT pipeline, you need to specify a storage location for metadata and logs, as well as provide the data transformation logic (e.g., via a notebook or Python script). Cluster management and monitoring queries are handled optionally by the Databricks platform and are not mandatory components.
- A. Correct.
Correct: A storage location is required to store the DLT pipeline's metadata, logs, and checkpoints.
- B. Incorrect.
Incorrect: DLT pipelines automatically handle cluster management during execution; you do not need to define a cluster manually.
- C. Incorrect.
Incorrect: While configurations can be set in the pipeline UI or via APIs, a JSON configuration file is not mandatory for creating a pipeline.
- D. Correct.
Correct: The transformation logic, either in a notebook or a Python script, is essential to define the data processing steps for a DLT pipeline.
- E. Incorrect.
Incorrect: While monitoring outputs is important, a Databricks SQL query is not a mandatory component for creating and deploying a DLT pipeline.