Databricks Data Engineer Associate Question 452
Select 3You are designing a data pipeline in Databricks that ingests data from an external API and processes it in Delta Lake. The API occasionally fails due to transient network issues. To ensure reliability, you need to implement a retry policy when the API call fails. Which of the following configurations would appropriately handle retries in this scenario?
- A
Use the
withRetryutility to wrap the API call with a retry logic, specifying the maximum number of retries and a delay interval between retries. - B
Enable Auto Loader's retry mechanism by setting the
cloudFiles.retryCountandcloudFiles.retryWaitTimeoptions. - C
Implement a custom retry logic using a
try-exceptblock in your Python code, and use an exponential backoff strategy for retries. - D
Configure a Databricks job to automatically retry upon failure by setting the
max_retriesparameter in the job configuration. - E
Use Delta Lake's
MERGEoperation withWHEN NOT MATCHEDto automatically retry inserting missing data after a failure.
Show answer and explanation
Correct answers: A, C, D
Explanation
To handle retries for transient API failures in Databricks, you can use the withRetry utility or implement a custom retry logic with a try-except block and exponential backoff. Additionally, configuring the max_retries parameter in a Databricks job ensures that the job retries automatically upon failure. These approaches are appropriate for handling transient issues, whereas options like Auto Loader-specific settings or Delta Lake's MERGE operation are not relevant to this scenario.
- A. Correct.
Correct: The
withRetryutility is a common pattern used to handle transient failures. It allows you to specify retry behavior like the number of retries and the delay between attempts. - B. Incorrect.
Incorrect: The Auto Loader retry mechanism (
cloudFiles.retryCountandcloudFiles.retryWaitTime) is specific to file-based data ingestion using Auto Loader and does not apply to API calls. - C. Correct.
Correct: Writing custom retry logic with a
try-exceptblock and an exponential backoff strategy is a reliable way to manage transient failures in API calls. - D. Correct.
Correct: Configuring
max_retriesin a Databricks job ensures that the job itself will automatically retry in case of failure, which is useful for handling transient issues at the job level. - E. Incorrect.
Incorrect: Delta Lake's
MERGEoperation does not handle retries for transient API failures. It is used for upserts or merging data into Delta tables.