Databricks Data Engineer Professional Question 5
Select 3A data engineering team is using Databricks to orchestrate and manage a series of data pipelines in a production environment. They need to ensure reliable execution, consistent logging, and scheduling of their workflows. Which Databricks feature(s) should the team use to manage this use case effectively?
- A
Databricks Workflows
- B
Databricks SQL Dashboards
- C
Databricks Jobs
- D
Databricks Repos
- E
Databricks Delta Live Tables
Show answer and explanation
Correct answers: A, C, E
Explanation
To manage production-grade data pipelines in Databricks, the team should use Databricks Workflows for orchestration, Databricks Jobs for scheduling and execution, and Databricks Delta Live Tables for building and managing ETL pipelines. These features address the requirements for reliable execution, logging, and scheduling. Databricks SQL Dashboards and Repos, while useful in other contexts, are not designed for pipeline management.
- A. Correct.
Databricks Workflows is the recommended solution for orchestrating and managing pipelines, as it provides features for scheduling, monitoring, and error handling.
- B. Incorrect.
Databricks SQL Dashboards are primarily used for data visualization and monitoring insights, not for orchestrating or managing workflows.
- C. Correct.
Databricks Jobs allow users to schedule and monitor notebooks, JAR files, or Python scripts, making it a core feature for running production data pipelines.
- D. Incorrect.
Databricks Repos is a feature for version-controlling notebooks and code, but it does not provide workflow orchestration or execution capabilities.
- E. Correct.
Databricks Delta Live Tables is a framework specifically designed for declarative ETL pipelines, providing pipeline reliability and error handling, making it suitable for managing complex workflows.