Databricks Data Engineer Associate Question 216
Single answerYou are working on a data pipeline in Databricks and need to create a new column in a DataFrame, df, which categorizes customers based on their total purchase amount. Specifically, customers with a total purchase below 100 should be labeled as 'Low', those between 100 and 500 as 'Medium', and those above 500 as 'High'. Which of the following code snippets correctly implements this logic using the CASE/WHEN construct in PySpark?
- A
df.withColumn('customer_category', when(col('total_purchase') < 100, 'Low').when((col('total_purchase') >= 100) & (col('total_purchase') <= 500), 'Medium').otherwise('High'))
- B
df.withColumn('customer_category', when(col('total_purchase') < 100, 'Low').when(col('total_purchase') < 500, 'Medium').otherwise('High'))
- C
df.withColumn('customer_category', when(col('total_purchase') < 100, 'Low').otherwise('Medium').otherwise('High'))
- D
df.withColumn('customer_category', when(col('total_purchase') > 500, 'High').when((col('total_purchase') >= 100) & (col('total_purchase') <= 500), 'Medium').otherwise('Low'))
Show answer and explanation
Correct answer: A
Explanation
The CASE/WHEN construct in PySpark is used to create custom control flow for column values. The first option correctly implements the conditions for categorizing customers based on their total purchase amount, ensuring proper ranges and ordering. The other options either misuse the CASE/WHEN construct or have logical errors in the conditions.
- A. Correct.
This is the correct implementation. The CASE/WHEN construct is applied correctly with an initial condition for 'Low', a range condition for 'Medium', and an otherwise condition for 'High'.
- B. Incorrect.
This implementation is incorrect because the second
whencondition overlaps with the first (it does not properly account for the lower bound of 100), leading to incorrect categorization. - C. Incorrect.
This implementation is invalid because the
otherwisecondition is used multiple times, which is not allowed in a single CASE/WHEN chain. - D. Incorrect.
This implementation has incorrect logic. The conditions are out of order, as 'High' is checked before 'Medium', which would cause incorrect categorization for some values.