Databricks Data Engineer Associate Question 152
Single answerYou are working on a Databricks notebook and have a DataFrame df with a column event_date that is currently stored as a string in the format 'yyyy-MM-dd HH:mm:ss'. You need to cast this column to a timestamp type for downstream processing. Which of the following code snippets will correctly perform this cast?
- A
df = df.withColumn('event_date', df['event_date'].cast('timestamp'))
- B
df = df.withColumn('event_date', df['event_date'].astype('timestamp'))
- C
df = df.selectExpr('CAST(event_date AS timestamp) AS event_date')
- D
df = df.withColumn('event_date', df['event_date'].to_timestamp())
Show answer and explanation
Correct answer: A
Explanation
To cast a column to a timestamp type in PySpark, you should use the cast() method. This operation is performed using the withColumn method, which modifies the DataFrame by replacing the column with the casted version. The other options are either invalid in PySpark or do not meet the requirement of modifying the column in-place.
- A. Correct.
This is the correct syntax for casting a column to a
timestamptype in PySpark. Thecast()function is used to convert a column's data type. - B. Incorrect.
This option is incorrect because the
astypemethod is not valid in PySpark. It is used in Pandas, not in Spark DataFrames. - C. Incorrect.
This option is incorrect because while the
selectExprmethod can be used to cast a column, it creates a new DataFrame instead of modifying the existing one. The question requires modifying the column in-place. - D. Incorrect.
This option is incorrect because the
to_timestamp()method does not exist for PySpark columns. It is not a valid method for this operation.