Databricks Data Engineer Professional Question 278
Single answerYou are tasked with building a data pipeline in Databricks to process daily sales data. The pipeline should ingest raw data from an S3 bucket, transform it using a Databricks notebook, and write the results back to a Delta table. The process should run automatically every day at 7 AM. Which approach should you take to implement this pipeline using Databricks Jobs?
- A
Create a Databricks Job, add a task to run the notebook for the transformations, and configure a schedule to trigger the job daily at 7 AM.
- B
Manually trigger a Databricks notebook every day at 7 AM using the Databricks CLI.
- C
Use a Databricks SQL query to schedule the notebook to run daily at 7 AM.
- D
Create a Databricks Job, set up a cluster policy to attach the cluster, and manually start the job every day.
Show answer and explanation
Correct answer: A
Explanation
Databricks Jobs provide a robust framework for building and managing data pipelines. They allow you to define tasks, such as running notebooks, and schedule them with specific triggers, such as daily schedules. This approach ensures automation, reliability, and repeatability for data processing workflows.
- A. Correct.
This is the correct approach. Databricks Jobs support creating tasks to run notebooks, configuring dependencies, and scheduling runs. The scheduling feature is ideal for automating pipelines like the one described.
- B. Incorrect.
This is incorrect because manually triggering notebooks every day is not an automated solution and does not leverage the scheduling capabilities of Databricks Jobs.
- C. Incorrect.
This is incorrect because Databricks SQL queries are used for writing and scheduling queries, not for running notebook-based transformation pipelines.
- D. Incorrect.
This is incorrect because manually starting the job every day defeats the purpose of automation, and cluster policies are unrelated to scheduling jobs.