Databricks Machine Learning Professional Question 7
Single answerYou are working on a machine learning project in Databricks, and your team has decided to use Delta Lake for storing and managing training data. During the data preprocessing phase, you notice that the dataset contains duplicate records. What is the most efficient way to handle duplicate records in a Delta table while ensuring data consistency?
- A
Use the
DROP DUPLICATESSQL command directly on the Delta table. - B
Read the Delta table into a DataFrame, use the
distinct()method to remove duplicates, and overwrite the Delta table. - C
Enable Delta Lake's automatic duplicate removal feature in the table properties.
- D
Use the
MERGEcommand to remove duplicates by specifying a match condition.
Show answer and explanation
Correct answer: B
Explanation
Delta Lake is designed to ensure ACID compliance and efficient data management, but it does not have a built-in feature for automatic duplicate removal. To effectively handle duplicates, you must explicitly process the data. Reading the Delta table into a DataFrame, using the distinct() method to eliminate duplicate records, and then overwriting the Delta table is a reliable and efficient approach. This method ensures that the changes are consistent and maintain Delta Lake's transactional integrity.
- A. Incorrect.
The
DROP DUPLICATESSQL command does not exist in Databricks SQL or Delta Lake. Removing duplicates requires a more explicit approach. - B. Correct.
Reading the Delta table into a DataFrame, applying the
distinct()method to remove duplicates, and overwriting the table is a common and efficient method to handle duplicates while maintaining the Delta Lake's ACID properties. - C. Incorrect.
Delta Lake does not have an automatic duplicate removal feature. Managing duplicates requires explicit actions by the user.
- D. Incorrect.
The
MERGEcommand is primarily used for upserts, not for removing duplicates. It requires a match condition but does not inherently handle duplicate records.