Databricks Data Engineer Associate Question 392
Single answerYou are using Auto Loader to ingest data from a JSON source into a Delta table, but you notice that all the columns are inferred as STRING types rather than their expected types (e.g., INTEGER or DOUBLE). What could be the reason for this behavior?
- A
The JSON source contains inconsistent data types for some fields across different records.
- B
Auto Loader does not support schema inference for JSON files.
- C
The JSON source contains null values for all records in some fields.
- D
You did not specify the schema explicitly while configuring Auto Loader.
Show answer and explanation
Correct answer: A
Explanation
When Auto Loader processes a JSON source and encounters inconsistent data types for the same field across different records, it defaults to inferring the field as STRING to ensure compatibility and avoid data loss. This behavior ensures that all data can be loaded without errors, but it may lead to all fields being inferred as STRING if such inconsistencies exist.
- A. Correct.
If the JSON source has inconsistent data types for a field (e.g., a field is INTEGER in one record and STRING in another), Auto Loader defaults to inferring the type as STRING to avoid potential data loss.
- B. Incorrect.
This is incorrect because Auto Loader supports schema inference for JSON files. However, it may infer all fields as STRING for specific reasons, such as inconsistent data types.
- C. Incorrect.
Null values in a field do not cause Auto Loader to infer the entire column as STRING. Instead, the column's type would still be inferred based on non-null values.
- D. Incorrect.
Although not specifying a schema for Auto Loader triggers schema inference, this alone does not explain why all fields are inferred as STRING. The issue lies in the data itself, not the lack of an explicit schema.