Databricks Data Engineer Professional Question 2
Select 2You are a data engineer working on a Databricks workspace and need to automate a data pipeline that ingests CSV files from an external source into a Delta table. The pipeline should handle the following requirements:
- Automatically trigger when new files arrive in the source location.
- Ensure idempotency (process each file only once).
- Provide visibility into the pipeline's execution status.
Which combination of Databricks tools would best meet the requirements?
- A
Databricks Auto Loader for file ingestion, Delta Lake for storage, and Databricks Job to orchestrate the pipeline
- B
Databricks Delta Live Tables with a streaming pipeline and built-in monitoring
- C
Databricks SQL to query the source files and manually trigger notebook execution
- D
Databricks Notebooks for file ingestion, Delta Lake for storage, and manual execution of pipeline tasks
- E
Databricks Auto Loader for file ingestion and Databricks REST API for manually tracking pipeline status
Show answer and explanation
Correct answers: A, B
Explanation
To meet the requirements of automating the ingestion, ensuring idempotency, and providing visibility into execution status, you can use either Databricks Auto Loader with Delta Lake and Databricks Jobs or Delta Live Tables. Auto Loader supports automatic file ingestion and idempotency, while Jobs provide orchestration. Alternatively, Delta Live Tables provides a declarative way to build pipelines with built-in monitoring and support for streaming ingestion. Other options lack either the automation, idempotency, or monitoring capabilities required for this scenario.
- A. Correct.
This option is correct because Databricks Auto Loader can handle incremental file ingestion with idempotency, Delta Lake provides a reliable storage layer, and Databricks Jobs allows for automation and orchestration of the pipeline.
- B. Correct.
This option is correct because Delta Live Tables (DLT) supports streaming ingestion, idempotency, and provides built-in monitoring for pipeline execution status.
- C. Incorrect.
This option is incorrect because Databricks SQL is not designed for file ingestion or pipeline automation, and manual triggering would not meet the automation requirement.
- D. Incorrect.
This option is incorrect because manual execution of tasks does not meet the automation or idempotency requirements, and there is no built-in mechanism for tracking execution status.
- E. Incorrect.
This option is incorrect because while Auto Loader supports file ingestion, manually tracking pipeline status using the REST API is not efficient or scalable for automation.