Databricks Data Engineer Associate Question 179
Single answerYou are working with a Databricks notebook and have a column named json_data in a DataFrame that contains JSON strings. Each JSON string represents a nested structure with keys such as id (integer) and info (a struct with name and age fields). You need to parse this JSON column into a struct to access its fields. Which of the following approaches will correctly achieve this in PySpark?
- A
Use the
from_jsonfunction along with a defined schema to parse the JSON column. - B
Use the
explodefunction to directly parse the JSON column into a struct. - C
Use the
to_jsonfunction to convert the JSON column into a struct. - D
Use the
cast('struct')operation to cast the JSON column into a struct.
Show answer and explanation
Correct answer: A
Explanation
To parse a JSON string into a struct in PySpark, you must use the from_json function along with a defined schema that matches the structure of the JSON data. The other options either perform unrelated operations or do not support parsing JSON strings into structs.
- A. Correct.
Correct. The
from_jsonfunction allows you to parse JSON strings into a struct by providing a schema that defines the expected structure of the JSON data. - B. Incorrect.
Incorrect. The
explodefunction is used to flatten arrays or nested structures, not to parse JSON strings into a struct. - C. Incorrect.
Incorrect. The
to_jsonfunction converts a struct into a JSON string, but it does not parse a JSON string into a struct. - D. Incorrect.
Incorrect. The
cast('struct')operation cannot parse a JSON string into a struct. Casting works for simple types but not for converting JSON strings into complex types like structs.