You are building a data pipeline using Databricks Structured Streaming and Delta Lake to process and store real-time data from IoT sensors. The requirements are as follows: 1. Guarantee exactly-once processing semantics. 2. Ensure that late-arriving data is handled without data loss. 3. Maintain an audit trail of all changes to the data for compliance purposes. Which combination of features and design patterns should you implement to meet these requirements?

A. Use Delta Lake's MERGE operation to handle late-arriving data and implement an upsert pattern.

B. Enable Delta Lake's Change Data Feed (CDF) to maintain an audit trail of changes.

C. Configure Structured Streaming with a Kafka source and checkpointing for exactly-once processing semantics.

A. Use Delta Lake's MERGE operation to handle late-arriving data and implement an upsert pattern.

B. Enable Delta Lake's Change Data Feed (CDF) to maintain an audit trail of changes.

C. Configure Structured Streaming with a Kafka source and checkpointing for exactly-once processing semantics.

D. Use Delta Lake's OPTIMIZE operation to reduce small files in the data pipeline.

E. Use Structured Streaming in complete output mode to handle late-arriving data.

Databricks Data Engineer Professional Question 76: You are building a data pipeline using Databricks Structured Streaming... | Databricks Data Engineer Professional Dumps

Databricks Certified Data Engineer Professional. Professional level, Databricks. Free question with the correct answer and a full explanation.

Databricks Data Engineer Professional Question 76

Select 3

You are building a data pipeline using Databricks Structured Streaming and Delta Lake to process and store real-time data from IoT sensors. The requirements are as follows:

Guarantee exactly-once processing semantics.
Ensure that late-arriving data is handled without data loss.
Maintain an audit trail of all changes to the data for compliance purposes.

Which combination of features and design patterns should you implement to meet these requirements?

A
Use Delta Lake's MERGE operation to handle late-arriving data and implement an upsert pattern.
B
Enable Delta Lake's Change Data Feed (CDF) to maintain an audit trail of changes.
C
Configure Structured Streaming with a Kafka source and checkpointing for exactly-once processing semantics.
D
Use Delta Lake's OPTIMIZE operation to reduce small files in the data pipeline.
E
Use Structured Streaming in complete output mode to handle late-arriving data.

Show answer and explanation

Correct answers: A, B, C

Explanation

To meet the requirements of exactly-once processing, handling late-arriving data, and maintaining an audit trail, you need to combine the strengths of Structured Streaming and Delta Lake. The MERGE operation allows late data to be seamlessly integrated, Change Data Feed ensures an immutable audit trail, and checkpointing with a reliable source like Kafka guarantees exactly-once processing semantics. Other features like OPTIMIZE or complete output mode do not directly address the requirements.

A. Correct.
Correct: The MERGE operation in Delta Lake allows you to implement an upsert pattern for handling late-arriving data by updating or inserting rows as necessary.
B. Correct.
Correct: Delta Lake's Change Data Feed (CDF) enables you to maintain a detailed audit trail of changes to the data, which is essential for compliance purposes.
C. Correct.
Correct: Structured Streaming supports exactly-once processing semantics when used with a reliable source like Kafka and checkpointing, ensuring no data is lost or processed more than once.
D. Incorrect.
Incorrect: While the OPTIMIZE operation is useful for reducing small files and improving query performance, it does not contribute to meeting the specified requirements.
E. Incorrect.
Incorrect: The complete output mode is not designed for handling late-arriving data effectively. Instead, use Delta Lake's features like MERGE for this purpose.

Databricks Data Engineer Professional practice question 76 of 313

Databricks Data Engineer Professional Question 76

Take a Databricks Data Engineer Professional practice test under exam conditions

More Databricks Data Engineer Professional practice questions