Databricks Generative AI Engineer Associate Question 184
Select 3You are developing a machine learning pipeline in Databricks that uses a PyFunc model for generative AI tasks. The model requires input text to be tokenized during pre-processing and its output probabilities need to be converted to the most likely category during post-processing. Which of the following steps should you implement to properly chain the model with pre- and post-processing?
- A
Define a custom Python function to handle tokenization and integrate it into the pre-processing step of the PyFunc model.
- B
Implement the post-processing logic to map output probabilities to categories within the PyFunc model's predict method.
- C
Use the Databricks MLflow framework to register the PyFunc model along with its pre- and post-processing steps as a single unit.
- D
Skip defining pre- and post-processing logic in the PyFunc model and handle it externally in the pipeline.
- E
Ensure your PyFunc model pre- and post-processing steps are implemented using the MLflow pyfunc.PythonModel class.
Show answer and explanation
Correct answers: A, C, E
Explanation
To chain a PyFunc model with pre- and post-processing in Databricks, it is necessary to define the pre-processing (e.g., tokenization) and post-processing (e.g., mapping probabilities to categories) correctly. The MLflow framework's pyfunc.PythonModel class is specifically designed to handle such use cases, and registering the model in MLflow ensures the pipeline operates cohesively. Skipping or externalizing these steps would contradict the purpose of the PyFunc model.
- A. Correct.
Correct: Pre-processing, such as tokenization, needs to be explicitly defined to prepare input data for the model, and this can be integrated into the PyFunc model's workflow.
- B. Incorrect.
Incorrect: The post-processing logic should generally be implemented outside the model's predict method to separate concerns and allow flexibility.
- C. Correct.
Correct: MLflow allows you to register models with pre- and post-processing steps as a single unit, ensuring the pipeline is cohesive and executable.
- D. Incorrect.
Incorrect: Pre- and post-processing should not be skipped or handled externally in a Databricks pipeline, as the PyFunc model is designed to encapsulate this functionality.
- E. Correct.
Correct: The pyfunc.PythonModel class in MLflow is a standard way to define PyFunc models with integrated pre- and post-processing.