Databricks Data Engineer Associate Question 409
Select 3You are a data engineer tasked with monitoring and auditing job runs in a Databricks workspace. To investigate the lineage of a specific table and gather metrics about the jobs that modified it, you decide to query the event logs. Which of the following actions are appropriate to achieve this?
- A
Use the
eventLogtable in the Databricks workspace to query job events and filter by event type. - B
Access the audit logs stored in your cloud storage account to trace job operations and lineage.
- C
Enable the Databricks SQL Query History feature to retrieve the lineage of the specific table.
- D
Query the
jobsAPI endpoint to directly retrieve lineage details of the table. - E
Use the
delta.historycommand on the table to retrieve operation history and metadata changes.
Show answer and explanation
Correct answers: A, B, E
Explanation
To analyze job metrics, audit logs, and table lineage in Databricks, you can use the eventLog table for event-specific data, audit logs for broader tracking, and the delta.history command for Delta table-specific lineage. The Databricks SQL Query History feature and jobs API are not designed for direct lineage or operation history tracking.
- A. Correct.
This is correct. The
eventLogtable stores job and event-related data, which can be queried to understand metrics and lineage. Filtering by event type helps narrow down the relevant information. - B. Correct.
This is correct. Audit logs in cloud storage provide details about job operations, data access, and lineage, making them useful for investigations.
- C. Incorrect.
This is incorrect. The Databricks SQL Query History feature tracks SQL queries run in Databricks SQL but does not provide detailed lineage for a specific table.
- D. Incorrect.
This is incorrect. The
jobsAPI provides information about job configurations, statuses, and runs but does not directly provide table lineage details. - E. Correct.
This is correct. The
delta.historycommand retrieves the operation history for a Delta table, which can be used to understand its lineage and changes.