Databricks Data Engineer Associate Question 250
Single answerA data engineering team is working with a large dataset stored in an object storage system, such as AWS S3, that needs to be accessed frequently by multiple teams for data analysis. However, the team does not want to move the data into the Databricks managed storage due to storage cost concerns. Which of the following scenarios justifies the use of an external table in this case?
- A
The dataset is static, and there is no need for frequent updates or optimizations.
- B
The dataset needs to be physically moved into Databricks for performance optimization.
- C
The dataset needs to remain in the external storage system but be queryable using Databricks SQL.
- D
The dataset is small and frequently updated, requiring tight integration with Delta Lake.
Show answer and explanation
Correct answer: C
Explanation
An external table is used when the data resides outside of Databricks (e.g., in AWS S3, Azure Data Lake, etc.) and there is no need to move it into Databricks-managed storage. This allows teams to query the data using Databricks SQL without incurring additional storage costs. The scenario described aligns with this use case, as the data needs to stay in external storage but still be accessible for analysis.
- A. Incorrect.
An external table is appropriate when the data is stored in an external system and needs to remain there, but is queryable through Databricks SQL. This aligns perfectly with the scenario described.
- B. Incorrect.
This is incorrect because an external table is specifically used to keep the data in external storage, not to move it into Databricks.
- C. Correct.
This is the correct answer because it matches the scenario where data needs to remain in external storage but still be accessible through Databricks for querying.
- D. Incorrect.
This is incorrect because external tables are not suited for small, frequently updated datasets that require tight integration with Delta Lake.