Databricks Data Engineer Associate Question 408
Select 2You are tasked with auditing the operations and lineage of a specific Delta table in your Databricks workspace. Which of the following steps should you take to query the event logs to retrieve this information?
- A
Enable the Databricks cluster audit logs and configure their destination.
- B
Query the
event_logtable in thedeltaschema of the Databricks metastore. - C
Set up a cluster with a SparkListener to capture event logs in real time.
- D
Use the
DESCRIBE HISTORYcommand on the Delta table to examine its operation history. - E
Query the
cloudFilesschema to fetch event log details for Delta tables.
Show answer and explanation
Correct answers: A, D
Explanation
Auditing event logs and examining lineage in Databricks requires enabling cluster audit logs and using commands like DESCRIBE HISTORY for Delta tables. Audit logs provide detailed metrics and operational data, while DESCRIBE HISTORY gives a transactional history of the Delta table, helping with auditing and lineage tracking.
- A. Correct.
Correct. Enabling Databricks cluster audit logs and configuring their destination is necessary to capture detailed metrics, lineage, and audit data.
- B. Incorrect.
Incorrect. There is no
event_logtable in the Databricks metastore'sdeltaschema. Event logs are captured via audit log configurations. - C. Incorrect.
Incorrect. While SparkListeners can capture events, this is not the standard approach for querying Databricks event logs, which are obtained through audit log configuration.
- D. Correct.
Correct. The
DESCRIBE HISTORYcommand provides a history of operations (e.g., updates, inserts, deletes) performed on a Delta table, which is useful for lineage and auditing. - E. Incorrect.
Incorrect. The
cloudFilesschema is used for auto-loader configurations, not for querying event logs or Delta table lineage.