Databricks Data Engineer Associate Question 453
Select 3You are designing a data pipeline in Databricks that involves writing data to a REST API endpoint. The API occasionally fails due to transient network issues. Which approach ensures that your pipeline retries the API call upon failure with proper configuration for retries?
- A
Use the
withRetrymethod in Databricks to define a retry policy with maximum retry attempts and delay between retries. - B
Configure a retry mechanism using Spark's
retryparameter in theDataFrame.writemethod. - C
Implement a retry policy by wrapping the API call in a function that uses Python's
retrylibrary and invoke it from within a Databricks notebook. - D
Set up a retry configuration in the REST API's endpoint settings directly from Databricks.
- E
Leverage the Databricks Workflows' task retry settings to configure the number of retries and delay in case of failure.
Show answer and explanation
Correct answers: A, C, E
Explanation
To handle transient failures when writing to a REST API endpoint in Databricks, you can use retry mechanisms such as the withRetry method, Python's retry library, or the built-in retry settings in Databricks Workflows. These approaches allow you to specify retry policies (e.g., maximum retries and delays) that ensure robustness in data pipelines. Spark's DataFrame.write method does not support retries, and retry configurations cannot directly be set on the REST API endpoint from within Databricks.
- A. Correct.
Correct: The
withRetrymethod or similar retry mechanisms in Databricks allow you to define retry policies for transient failures, making this a valid approach. - B. Incorrect.
Incorrect: Spark's
DataFrame.writemethod does not natively support retry mechanisms, so this option is not applicable. - C. Correct.
Correct: Using Python's
retrylibrary is a valid approach for implementing a retry policy, especially for custom API calls within a Databricks notebook. - D. Incorrect.
Incorrect: The retry configuration cannot be set directly in the REST API's endpoint settings from within Databricks. This must be handled in the application code or Databricks workflow configuration.
- E. Correct.
Correct: Databricks Workflows provide built-in task retry settings, which can be configured to handle failures and retries automatically in pipelines.