Databricks Data Engineer Professional Question 231
Select 2While investigating the performance of a Spark job running on a Databricks cluster, you need to identify the stages and their execution timelines to optimize your job. Which of the following actions will help you inspect the event timelines and metrics for stages and jobs performed on the cluster?
- A
Navigate to the Spark UI from the Databricks cluster details page and review the 'Stages' tab.
- B
Use the Databricks Jobs API to retrieve detailed metrics for each stage.
- C
Access the Spark UI from the Databricks workspace and review the 'SQL' tab for job execution details.
- D
Enable Ganglia metrics on the Databricks cluster to collect stage-level performance data.
- E
Download the event logs from the Spark UI and analyze them using a compatible performance analysis tool.
Show answer and explanation
Correct answers: A, E
Explanation
To inspect event timelines and metrics for jobs and stages executed on a Databricks cluster, the Spark UI is the primary tool to use. The 'Stages' tab in the Spark UI provides detailed timelines and metrics for each stage. Additionally, downloading and analyzing event logs can provide further insights into job and stage performance. Other methods, such as the Databricks Jobs API or Ganglia, are not designed for detailed stage-level analysis.
- A. Correct.
Correct. The Spark UI provides detailed information about stages, their execution timelines, and associated metrics in the 'Stages' tab.
- B. Incorrect.
Incorrect. The Databricks Jobs API allows you to manage and monitor jobs but does not provide granular stage-level metrics or timelines.
- C. Incorrect.
Incorrect. The 'SQL' tab in the Spark UI is specific to SQL queries and does not provide stage-level execution details for general Spark jobs.
- D. Incorrect.
Incorrect. Ganglia is used for monitoring system-level metrics (e.g., CPU, memory) and does not provide Spark-specific stage timelines or metrics.
- E. Correct.
Correct. Event logs from the Spark UI can be downloaded and analyzed with external tools to gain detailed insights into job and stage performance.