Databricks Data Engineer Associate Question 455
Select 2You are working on a data pipeline in Databricks that interacts with an external API to retrieve data. Occasionally, the API fails due to rate limits, and you need to ensure your pipeline retries the request automatically with appropriate intervals. Which of the following approaches should you use to set up a retry policy in Databricks?
- A
Use the
retryfunction from thepyspark.utilmodule with a specified maximum number of retries and a backoff interval. - B
Implement a retry loop in your notebook using Python's
time.sleep()to introduce delays between retries in case of failure. - C
Leverage a third-party library like
tenacitywithin your notebook to define a retry policy with exponential backoff. - D
Configure the Databricks cluster to automatically retry failed API calls by enabling the 'Retry on Failure' setting.
- E
Use Databricks Workflows' built-in retry policy by specifying the maximum number of retries and the retry interval.
Show answer and explanation
Correct answers: C, E
Explanation
To implement a retry policy in Databricks, you can either use a third-party library like tenacity within your notebook for retrying API calls or leverage Databricks Workflows' built-in retry policy for managing retries at the workflow level. Both approaches are valid and widely used for handling transient failures effectively. Other options either refer to non-existent features or are less efficient methods.
- A. Incorrect.
The
pyspark.utilmodule does not have aretryfunction. This option is incorrect as it refers to a non-existent feature. - B. Incorrect.
While implementing a retry loop using Python's
time.sleep()can work, it is not an efficient or recommended approach for setting up a robust retry mechanism. This option is not the best practice. - C. Correct.
Using a third-party library like
tenacityis a valid approach to implement a retry policy with features like exponential backoff. This is a feasible solution for retrying API calls within a Databricks notebook. - D. Incorrect.
Databricks clusters do not have a specific 'Retry on Failure' setting for API calls. This option is incorrect as it references a non-existent feature.
- E. Correct.
Databricks Workflows has a built-in retry policy that allows you to specify the maximum number of retries and the retry interval. This is a recommended and native approach for handling retries in a Databricks workflow.