Databricks Data Engineer Associate Question 154
Single answerYou are working with a Delta table in Databricks that contains a column named 'event_date' stored as a string in the format 'yyyy-MM-dd HH:mm:ss'. You need to cast this column to a timestamp type for downstream analytics. Which of the following commands will correctly achieve this transformation?
- A
df = df.withColumn('event_date', df['event_date'].cast('timestamp'))
- B
df = df.withColumn('event_date', to_date(df['event_date']))
- C
df = df.withColumn('event_date', from_unixtime(df['event_date']))
- D
df = df.withColumn('event_date', date_format(df['event_date'], 'timestamp'))
Show answer and explanation
Correct answer: A
Explanation
To cast a column to a timestamp in PySpark, the .cast('timestamp') method is the correct approach. It ensures that the column is properly converted to the timestamp type, which includes both date and time information. Other functions like to_date and date_format are used for different purposes, such as extracting date portions or formatting timestamps as strings.
- A. Correct.
Correct. The
.cast('timestamp')method is the appropriate way to convert a column to the timestamp type in PySpark. - B. Incorrect.
Incorrect. The
to_datefunction extracts only the date part (yyyy-MM-dd) and does not retain the time information, so it will not produce a timestamp. - C. Incorrect.
Incorrect. The
from_unixtimefunction is used to convert Unix epoch time to a timestamp, but the input column here is a string, not a Unix epoch time. - D. Incorrect.
Incorrect. The
date_formatfunction is primarily used for formatting dates/timestamps as strings and does not convert a column to the timestamp type.