Databricks Generative AI Engineer Associate Question 94
Select 2You are working on a Databricks project where a large language model (LLM) needs training data from a database containing customer reviews. The data must be filtered to include only reviews written in the last 12 months and must exclude any reviews marked as 'spam.' Which of the following tools or approaches should you use to efficiently retrieve and preprocess this data?
- A
Use a Databricks SQL query to filter the data by date and 'spam' status before loading it into the LLM pipeline.
- B
Load the entire database into the Databricks environment and use Python to filter the data.
- C
Leverage Delta Lake's time travel feature to retrieve historical data for the required date range.
- D
Create a Databricks notebook to query the database using PySpark and perform filtering on the Spark DataFrame.
- E
Use a pre-trained LLM to automatically extract and filter the data from the database.
Show answer and explanation
Correct answers: A, D
Explanation
The correct answers involve using Databricks SQL or PySpark for filtering and retrieving data. Both approaches are optimized for handling large datasets and allow for efficient preprocessing, which is essential for preparing training data for an LLM. Databricks SQL is ideal for filtering at the source, while PySpark is a powerful option for distributed data processing within the Databricks environment.
- A. Correct.
Using a Databricks SQL query is an efficient way to filter data at the source, reducing the amount of unnecessary data being transferred and processed.
- B. Incorrect.
Loading the entire database and filtering it in Python is inefficient and can lead to performance bottlenecks, especially for large datasets.
- C. Incorrect.
Delta Lake's time travel feature is designed for accessing historical versions of data, not for filtering data by custom criteria such as 'spam' status.
- D. Correct.
Using PySpark in a Databricks notebook allows you to efficiently process and filter large datasets in a distributed manner, making it suitable for this scenario.
- E. Incorrect.
A pre-trained LLM is not designed for database querying or filtering tasks; it is used for tasks like natural language understanding and generation.