Databricks Data Engineer Associate Question 156
Select 3You are working with a timestamp column named event_time in a Delta table. You want to extract the year and month as separate columns from this timestamp in a PySpark DataFrame. Which of the following code snippets would achieve this?
- A
df = df.withColumn('year', year(df['event_time'])).withColumn('month', month(df['event_time']))
- B
df = df.withColumn('year', F.year('event_time')).withColumn('month', F.month('event_time'))
- C
df = df.selectExpr('year(event_time) as year', 'month(event_time) as month')
- D
df = df.withColumn('year', extract('year', df['event_time'])).withColumn('month', extract('month', df['event_time']))
- E
df = df.withColumn('year', F.date_format('event_time', 'yyyy')).withColumn('month', F.date_format('event_time', 'MM'))
Show answer and explanation
Correct answers: B, C, E
Explanation
In PySpark, calendar data like year and month can be extracted using functions like year, month, and date_format from pyspark.sql.functions. Additionally, SQL expressions can be used with selectExpr to achieve the same result. However, there is no extract function for this purpose in PySpark. Using the correct methods ensures accurate data processing and compatibility with Spark APIs.
- A. Incorrect.
This option is incorrect because
yearandmonthfunctions cannot be directly called without importing them or referencing them from thepyspark.sql.functionsmodule. - B. Correct.
This option is correct because it uses the
yearandmonthfunctions frompyspark.sql.functions, which are the recommended functions to extract calendar data from a timestamp. - C. Correct.
This option is correct because
selectExprallows SQL expressions likeyear(event_time)andmonth(event_time)to extract the year and month from a timestamp column. - D. Incorrect.
This option is incorrect because there is no
extractfunction in PySpark for extracting specific calendar fields from a timestamp. - E. Correct.
This option is correct because
date_formatfrompyspark.sql.functionscan be used to extract specific parts of a timestamp as strings, such as the year ('yyyy') and month ('MM').