Databricks Data Engineer Associate Question 249
Single answerYou are working on a Databricks project that involves analyzing log files generated by multiple applications in near real-time. These log files are stored in an external object storage system (e.g., AWS S3) and are continuously updated. You need to make this data queryable in Databricks without moving it into a managed Delta table. Which approach should you use?
- A
Create an external table pointing to the location of the log files in the object storage.
- B
Load the log files into a managed Delta table for better data management and performance.
- C
Use a temporary table to query the data directly from the object storage.
- D
Convert the log files to Delta format and register them as a managed Delta table.
Show answer and explanation
Correct answer: A
Explanation
External tables are ideal for situations where data resides in external storage, such as AWS S3, and you want to query it directly without moving it into the Databricks-managed storage. This approach is particularly useful when the data is updated frequently or needs to remain in its original location.
- A. Correct.
This is the correct approach. External tables allow you to query data stored in external object storage without moving it into Databricks-managed storage. This is ideal for scenarios where data is updated frequently or needs to remain in its original location.
- B. Incorrect.
Loading the log files into a managed Delta table would require moving the data into Databricks-managed storage, which is unnecessary in this scenario since the data is already stored externally and needs to remain so.
- C. Incorrect.
Temporary tables are session-scoped and not suitable for creating a persistent queryable interface for external data. They are also not designed for scenarios involving frequently updated data.
- D. Incorrect.
Converting the log files to Delta format and registering them as a managed Delta table would involve data movement, which is not required in this case. The requirement specifies querying the data directly from external storage.