Databricks Data Engineer Associate Question 388
Single answerA data engineer is using Auto Loader to ingest data from a JSON source. However, all the inferred column types in the DataFrame schema are set to STRING, even though some fields in the JSON source are numerical or boolean. What could be the reason for this behavior?
- A
The JSON files have inconsistent data types for certain fields across records.
- B
Auto Loader requires explicit schema definition for JSON sources to infer data types correctly.
- C
The JSON source contains only string values for all fields, so Auto Loader inferred all fields as STRING.
- D
The schema inference mode of Auto Loader is disabled by default for JSON sources.
Show answer and explanation
Correct answer: A
Explanation
When Auto Loader processes JSON files with inconsistent data types for a given field (e.g., mixing strings, integers, or booleans in the same field across records), it defaults to inferring the field as STRING to avoid errors and ensure all data can be read. This behavior ensures schema compatibility and avoids potential runtime issues.
- A. Correct.
If the JSON files contain inconsistent data types for certain fields (e.g., a field is a string in one record and an integer in another), Auto Loader will default to inferring those fields as STRING to ensure compatibility.
- B. Incorrect.
While you can provide an explicit schema for more control, Auto Loader does not require it for JSON sources. It can infer schema automatically unless overridden.
- C. Incorrect.
If the JSON source only contains string values for all fields, Auto Loader would infer all fields as STRING. However, this is not the described scenario since the fields are known to have numerical or boolean values.
- D. Incorrect.
The schema inference mode for Auto Loader is not disabled by default. Auto Loader automatically infers schema unless explicitly configured otherwise.