Databricks Data Engineer Associate Question 160
Single answerYou are working with a DataFrame in Databricks that contains a column named 'log_data' with entries like 'userID:12345, action:login, status:success'. You need to extract only the numeric user ID from this column into a new column. Which of the following code snippets correctly accomplish this task?
- A
df = df.withColumn('user_id', F.regexp_extract(F.col('log_data'), 'userID:(\d+)', 1))
- B
df = df.withColumn('user_id', F.extract(F.col('log_data'), 'userID:(\d+)', 1))
- C
df = df.withColumn('user_id', F.substring(F.col('log_data'), 'userID:(\d+)', 1))
- D
df = df.withColumn('user_id', F.regexp_replace(F.col('log_data'), 'userID:(\d+)', 1))
Show answer and explanation
Correct answer: A
Explanation
To extract a specific pattern from a string column in PySpark, the regexp_extract function should be used. It takes three arguments: the column to apply the regex on, the regex pattern, and the group index to extract. In this case, the pattern 'userID:(\d+)' matches 'userID:' followed by one or more digits, and the group index 1 extracts the numeric user ID.
- A. Correct.
This is the correct answer. The
regexp_extractfunction is used to extract a specific pattern from a string column, and the provided regex 'userID:(\d+)' captures the numeric user ID. - B. Incorrect.
This is incorrect. The
extractfunction does not exist in PySpark. The correct function to use isregexp_extract. - C. Incorrect.
This is incorrect. The
substringfunction is used to extract a substring based on position, not a pattern. It cannot handle regular expressions. - D. Incorrect.
This is incorrect. The
regexp_replacefunction is used to replace a pattern in a string with another value, not to extract a specific pattern.