Databricks Data Engineer Associate Question 390
Single answerA data engineer is using Databricks Auto Loader to ingest JSON data into a Delta table. However, all the columns in the ingested data are inferred as STRING types, even though the JSON data contains numeric and boolean values. What could be the reason for this behavior?
- A
The JSON data contains inconsistent data types across records for the same field.
- B
The schema inference option is disabled in the Auto Loader configuration.
- C
The Auto Loader is using a schema that was manually defined as STRING for all fields.
- D
The JSON file is malformed and does not follow a standard JSON structure.
Show answer and explanation
Correct answer: A
Explanation
Auto Loader infers schemas based on the data it processes. If the JSON source contains inconsistent data types for a field (e.g., string in some records and numeric in others), Auto Loader will interpret the field as STRING to avoid failure during ingestion. This ensures data consistency but may not match the intended schema.
- A. Correct.
When JSON data has inconsistent data types for the same field (e.g., a field is a string in one record and a number in another), Auto Loader defaults to inferring the data type as STRING to handle the inconsistency.
- B. Incorrect.
Disabling schema inference in Auto Loader would not cause all data to be inferred as STRING; instead, it would rely on an explicitly defined schema or throw an error if none is provided.
- C. Incorrect.
If a manually defined schema specifies all fields as STRING, the behavior would match the issue, but the question specifies that the schema was inferred, not manually defined.
- D. Incorrect.
Malformed JSON files typically cause parsing errors, preventing Auto Loader from successfully processing the data. This does not lead to inferring all fields as STRING.