Databricks Data Engineer Associate Question 221
Select 3You are tasked with designing an incremental data pipeline in Databricks to process new records arriving in a Delta Lake table. Which of the following steps is critical to ensure the pipeline processes only new data during each run?
- A
Configure a watermark to handle late-arriving data.
- B
Use the
mergeoperation in Delta Lake for handling updates and deletions. - C
Filter the source data using a column such as 'timestamp' or 'ingestion_date'.
- D
Leverage a checkpoint location to track the state of the stream.
- E
Manually maintain a list of already processed records in an external database.
Show answer and explanation
Correct answers: A, C, D
Explanation
To process data incrementally in Databricks, it is critical to filter the input data to identify new records, use a checkpoint location to maintain the state of the stream, and set watermarks to handle late-arriving data. These practices ensure the pipeline processes new data efficiently and reliably.
- A. Correct.
Configuring a watermark is essential when dealing with streaming data to handle late-arriving records and ensure timely processing.
- B. Incorrect.
Using the
mergeoperation in Delta Lake is important for managing updates and deletions but is not directly related to filtering new data in an incremental pipeline. - C. Correct.
Filtering the source data using a timestamp or ingestion_date column is a critical step to ensure only new data is processed during each pipeline execution.
- D. Correct.
Leveraging a checkpoint location allows Databricks to track the state of the stream and ensures that data is processed incrementally.
- E. Incorrect.
Manually maintaining a list of processed records in an external database is error-prone and not a recommended practice in Databricks.