Databricks Machine Learning Professional Question 4
Select 2You are working on a machine learning project in Databricks, and your team is preparing a large dataset for model training. The dataset is stored in a Delta Lake table. A data quality issue arises where some rows contain null values in critical columns, and duplicate records are also present. What actions should you take to ensure the dataset is clean and ready for training?
- A
Use Delta Lake's
DELETEstatement to remove rows with null values in critical columns. - B
Apply Delta Lake's
MERGEoperation to automatically deduplicate records. - C
Use Databricks Auto Loader to filter null values and drop duplicates during ingestion.
- D
Leverage Delta Lake's
UPDATEstatement to replace null values in critical columns with default values. - E
Use Delta Lake's
OPTIMIZEcommand to compact and clean the dataset.
Show answer and explanation
Correct answers: A, D
Explanation
To ensure the dataset is clean and ready for training, you should use Delta Lake's DELETE statement to remove rows with null values in critical columns and the UPDATE statement to replace null values with appropriate default values. These operations address the data quality issues directly. While deduplication and ingestion are important, the specific options provided (e.g., MERGE and Auto Loader) do not properly address the scenario described.
- A. Correct.
Correct. Delta Lake supports the
DELETEstatement, which can be used to remove rows with null values in critical columns, ensuring data quality for model training. - B. Incorrect.
Incorrect. The
MERGEoperation in Delta Lake is used for upserts (merging updates or inserts) but is not designed for deduplication. - C. Incorrect.
Incorrect. While Databricks Auto Loader is useful for efficient data ingestion, filtering null values and dropping duplicates need to be explicitly handled after ingestion when working with Delta Lake tables.
- D. Correct.
Correct. Delta Lake's
UPDATEstatement can be used to replace null values with default or imputed values in critical columns, which is a common data preparation step for machine learning. - E. Incorrect.
Incorrect. The
OPTIMIZEcommand improves query performance by compacting small files into larger files but does not directly address null values or duplicates.