Databricks Machine Learning Professional Question 117
Select 4You are building a machine learning pipeline in Databricks to automate model retraining whenever new data is ingested into your data lake. You want to use HTTP webhooks to trigger the retraining job. Which of the following steps are required to correctly configure the webhook and its endpoint?
- A
Generate the Webhook URL from a third-party HTTP endpoint or API service that can handle the trigger.
- B
Create a Databricks Job that includes the retraining logic and retrieve its Job ID.
- C
Provide the Webhook URL to the external system responsible for sending the HTTP request when new data is ingested.
- D
Configure the Webhook URL directly in the Databricks cluster to listen for incoming HTTP requests.
- E
Set up a mechanism to validate and log incoming webhook requests for security and debugging purposes.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
To use HTTP webhooks for triggering a Databricks job, you must generate a Webhook URL from an external service that can act as the trigger point. The Databricks Job containing the retraining logic must be created beforehand, and its Job ID will be associated with the webhook configuration. Additionally, the Webhook URL must be shared with the external system responsible for sending the HTTP request. Implementing validation and logging mechanisms ensures secure and reliable webhook usage.
- A. Correct.
Correct: The Webhook URL must be obtained from an external system that can receive and handle HTTP requests. This is typically provided by a third-party service or infrastructure that triggers the event.
- B. Correct.
Correct: A Databricks Job must be created with the retraining logic as it will be triggered by the webhook. The Job ID is necessary for configuring the pipeline.
- C. Correct.
Correct: The Webhook URL must be shared with the external system responsible for triggering the HTTP requests (e.g., a data ingestion service). This ensures the webhook triggers the correct job in Databricks.
- D. Incorrect.
Incorrect: Databricks clusters are not designed to directly listen for incoming HTTP requests. Webhooks require external services or endpoints to handle such requests.
- E. Correct.
Correct: Validating and logging incoming webhook requests is a best practice to ensure security (e.g., validating signatures) and to debug potential issues with the trigger.