Databricks Data Engineer Professional Question 219
Select 3You are tasked with monitoring a Databricks job that processes large volumes of data and occasionally fails due to memory issues. To ensure effective monitoring and debugging, you decide to implement structured logging and job metrics. Which of the following steps should you take to configure and monitor these logs and metrics effectively?
- A
Enable cluster log delivery to an external storage location, such as an S3 bucket or Azure Blob Storage.
- B
Configure a metrics sink, such as Prometheus, to capture job performance metrics.
- C
Set up a Databricks SQL dashboard to visualize logs in real-time.
- D
Use the Databricks REST API to fetch job run details and include them in your monitoring pipeline.
- E
Enable interactive mode in the Databricks notebook to view metrics during execution.
Show answer and explanation
Correct answers: A, B, D
Explanation
Effective monitoring and logging in Databricks involve setting up persistent storage for logs, configuring a metrics sink to capture performance metrics, and using APIs to integrate job run details into external monitoring systems. These steps ensure that you can analyze logs and metrics to debug issues like memory errors and optimize job performance.
- A. Correct.
Enabling cluster log delivery allows you to persist logs, such as stdout, stderr, and event logs, to an external storage location for further analysis and debugging. This is a correct step for monitoring and logging.
- B. Correct.
Configuring a metrics sink like Prometheus helps capture and store metrics for monitoring job performance over time. This is a recommended practice for effective monitoring.
- C. Incorrect.
While Databricks SQL dashboards are useful for querying and visualizing data, they are not directly used for visualizing logs in real-time or job metrics. This option is incorrect.
- D. Correct.
Using the Databricks REST API to fetch details about job runs allows you to programmatically integrate job metadata and status into a monitoring pipeline. This is a correct approach for monitoring.
- E. Incorrect.
Enabling interactive mode in a Databricks notebook is for debugging during development, but it does not help in monitoring or logging for production jobs. This option is incorrect.