DEA-C01 Question 107
Single answerYou are working on a data processing pipeline using AWS Glue. Your task is to transform a large dataset stored in Amazon S3 using PySpark. The dataset contains a 'timestamp' column in string format (e.g., '2023-10-01T12:45:00Z'), and you need to extract the year from each timestamp to create a new column called 'year'. Which PySpark code snippet achieves this transformation correctly?
- A
df = df.withColumn('year', df['timestamp'].substr(1, 4))
- B
df = df.withColumn('year', df['timestamp'].substring(1, 4))
- C
df = df.withColumn('year', F.year(F.to_timestamp(df['timestamp'])))
- D
df = df.withColumn('year', F.substring(df['timestamp'], 1, 4))
Show answer and explanation
Correct answer: C
Explanation
The correct answer uses PySpark functions from the pyspark.sql.functions library to process the transformation. F.to_timestamp parses the string into a timestamp object, and F.year extracts the year. Other options either use incorrect syntax or do not handle the timestamp conversion properly.
- A. Incorrect.
Incorrect. The substr function is not directly accessible without importing the necessary PySpark functions such as pyspark.sql.functions.
- B. Incorrect.
Incorrect. The substring method is not directly applicable to PySpark DataFrame columns and would result in an error.
- C. Correct.
Correct. This option uses the correct PySpark functions,
to_timestampto convert the string to a timestamp, andyearto extract the year. - D. Incorrect.
Incorrect. While
substringis a valid PySpark function, it cannot correctly handle timestamps in ISO 8601 format without prior parsing.