Databricks Data Engineer Associate Question 411
Select 3A data engineering team wants to monitor job performance and user activities within a Databricks workspace. They need to query the events log to analyze metrics, perform audit logging, and examine data lineage. Which of the following steps should they perform to achieve this?
- A
Enable cluster logging to send event logs to a cloud storage location.
- B
Use the
dbutils.fscommand to directly query the events log stored in the Databricks File System (DBFS). - C
Use the Databricks REST API to access workspace audit logs.
- D
Query the
event_logtable in the Databricks workspace's metastore database. - E
Set up structured streaming to process event log data in real time.
Show answer and explanation
Correct answers: A, C, E
Explanation
To analyze metrics, perform audit logging, and examine lineage using the events log, you need to enable logging on clusters to store logs, use the Databricks REST API for accessing audit logs, and optionally set up structured streaming for real-time processing. Direct querying using dbutils or non-existent tables like event_log is not applicable.
- A. Correct.
Correct: Enabling cluster logging ensures that event logs are written to a storage location, making them accessible for analysis.
- B. Incorrect.
Incorrect: The
dbutils.fscommand is used for file system operations in Databricks, but it is not suitable for directly querying the events log. - C. Correct.
Correct: The Databricks REST API provides access to audit logs, which can include workspace activities and lineage information.
- D. Incorrect.
Incorrect: The
event_logtable is not a standard table in Databricks' metastore database. Event logs are stored in logs, not as a table. - E. Correct.
Correct: Setting up structured streaming allows the processing of event log data in real time for metrics and lineage analysis.