Databricks Data Engineer Associate Question 150
Single answerYou are working with a Delta table in Databricks that contains a column named event_time as a string in the format 'yyyy-MM-dd HH:mm:ss'. You need to cast this column into a timestamp type to perform time-based aggregations. Which of the following code snippets correctly performs this operation?
- A
df = df.withColumn('event_time', df['event_time'].cast('timestamp'))
- B
df = df.withColumn('event_time', df['event_time'].cast(TimestampType()))
- C
df = df.withColumn('event_time', to_timestamp(df['event_time'], 'yyyy-MM-dd HH:mm:ss'))
- D
df = df.withColumn('event_time', df['event_time'].astype('timestamp'))
Show answer and explanation
Correct answer: A
Explanation
In PySpark, the correct way to cast a column to a different type is by using the cast method and passing the target type as a string (e.g., 'timestamp'). This ensures the data in the column is converted to the desired type correctly. Other methods, like astype, are not applicable in PySpark, and functions like to_timestamp are used for parsing, not casting.
- A. Correct.
This is the correct way to cast a column to a timestamp type in PySpark. The
castmethod is directly applied to the column, and 'timestamp' is passed as the target type. - B. Incorrect.
This option is incorrect because
castdoes not accept PySpark data types likeTimestampType()directly. It expects a string representation of the type, such as 'timestamp'. - C. Incorrect.
This option is incorrect because the
to_timestampfunction is used for parsing string timestamps with a specific format, not for casting. Casting assumes the format is already valid for conversion. - D. Incorrect.
This option is incorrect because the
astypemethod is not a valid PySpark function. It is used in libraries like pandas, but not in PySpark.