Databricks Generative AI Engineer Associate Question 181
Select 3You are tasked with building a machine learning pipeline in Databricks that includes pre-processing and post-processing steps for a text classifier. Using a PyFunc model, you implement the following steps: (1) tokenize and clean the input text in the pre-processing stage, (2) pass the processed text through the PyFunc model for prediction, and (3) convert the model's numerical output to a human-readable label in the post-processing stage. Which of the following steps are critical to ensure the pipeline functions correctly?
- A
Define a custom PyFunc wrapper that includes both pre- and post-processing logic.
- B
Save the pre-processing and post-processing functions separately from the PyFunc model to ensure modularity.
- C
Implement the
predictmethod in the PyFunc wrapper to handle both pre- and post-processing steps. - D
Save the PyFunc model with the pre- and post-processing logic included to enable end-to-end processing.
- E
Use Databricks MLflow to log and register only the core model, excluding pre- and post-processing.
Show answer and explanation
Correct answers: A, C, D
Explanation
To ensure the pipeline functions correctly, the PyFunc model must include pre- and post-processing logic. This is achieved by defining a custom PyFunc wrapper and implementing the predict method to handle input transformations, model predictions, and output transformations. Additionally, saving the PyFunc model with the logic included ensures seamless end-to-end processing. Omitting pre- or post-processing, or separating them from the PyFunc model in this context, would break the pipeline functionality.
- A. Correct.
Defining a custom PyFunc wrapper that includes both pre- and post-processing logic is essential as it ensures the PyFunc model can handle input and output transformations seamlessly.
- B. Incorrect.
Separating pre-processing and post-processing from the PyFunc model can be useful in some contexts, but it is not critical for ensuring the pipeline functions correctly in this scenario. The question emphasizes end-to-end functionality.
- C. Correct.
Implementing the
predictmethod in the PyFunc wrapper is critical because it defines the behavior of the model during inference, including pre- and post-processing steps. - D. Correct.
Saving the PyFunc model with the pre- and post-processing logic included enables end-to-end processing, which is necessary for the pipeline to function correctly.
- E. Incorrect.
Logging and registering only the core model without pre- and post-processing will make the pipeline incomplete and unusable for end-to-end inference.