Databricks Data Engineer Associate Question 162
Single answerYou are working with a DataFrame in Databricks that contains a column named email. You need to extract the domain (e.g., 'gmail.com') from the email addresses in this column and store it in a new column named domain. Which of the following PySpark operations would correctly achieve this?
- A
df.withColumn('domain', regexp_extract(col('email'), '@(.+)', 1))
- B
df.withColumn('domain', substring_index(col('email'), '@', -1))
- C
df.withColumn('domain', split(col('email'), '@')[1])
- D
df.withColumn('domain', col('email').substr(col('email').indexOf('@') + 1, length(col('email'))))
Show answer and explanation
Correct answer: A
Explanation
The regexp_extract function is the most suitable method for extracting specific patterns from a string in PySpark. The '@(.+)' regular expression captures the domain part of the email address (everything after the '@'), and the function's second argument specifies that the first capture group should be returned. Other options either use incorrect syntax for PySpark or rely on methods that are not available for PySpark column objects.
- A. Correct.
Correct. The
regexp_extractfunction is used to extract a specific pattern from a string based on a regular expression. Here, '@(.+)' captures everything after the '@' symbol, and the second argument (1) specifies the capture group to extract. - B. Incorrect.
Incorrect. While
substring_indexcan split a string at a delimiter like '@', it does not directly work with PySpark'scol()objects. It is primarily used in SQL expressions. - C. Incorrect.
Incorrect. The
splitfunction can split a string into an array, but you cannot directly use[1]to access the second element within a PySparkwithColumnoperation. Instead, you would need to usegetItem(1). - D. Incorrect.
Incorrect. PySpark's
col()objects do not have methods likeindexOforsubstrdirectly available. These operations are more suitable for Scala or plain Python strings, not PySpark columns.