Databricks Machine Learning Professional Question 165
Select 4You are tasked with building a machine learning pipeline in Databricks that processes a continuous stream of IoT sensor data. The data is sent to a cloud storage system and needs to be processed in near real-time for anomaly detection. Which of the following steps are necessary to implement this streaming pipeline in Databricks using Structured Streaming?
- A
Define a streaming DataFrame by reading from the source with the appropriate format and schema.
- B
Use a batch processing operation like
.collect()to ensure all data is loaded before applying transformations. - C
Apply necessary data transformations such as filtering and feature engineering using Spark DataFrame APIs.
- D
Write the results to a sink that supports streaming outputs, such as a Delta table or a message queue.
- E
Trigger the stream using a continuous query with a specified output mode.
Show answer and explanation
Correct answers: A, C, D, E
Explanation
Structured Streaming in Databricks requires defining a streaming DataFrame, applying transformations, writing to a compatible sink, and triggering the stream with the correct output mode. Batch operations like .collect() are incompatible with streaming pipelines because they are designed for static datasets and would cause memory issues with continuous data streams. Following these steps ensures a robust and scalable streaming pipeline for real-time anomaly detection.
- A. Correct.
Defining a streaming DataFrame with the correct source, format, and schema is the first essential step in setting up a Structured Streaming pipeline in Databricks.
- B. Incorrect.
Batch operations like
.collect()are not suitable for streaming pipelines since they require loading all the data into memory and are not scalable for continuous data streams. - C. Correct.
Transformations like filtering and feature engineering are key steps in preparing the data for anomaly detection within the streaming pipeline.
- D. Correct.
Writing the results to a streaming-compatible sink, such as a Delta table or message queue, ensures the pipeline can handle continuous data outputs.
- E. Correct.
Triggering the stream with a continuous query and specifying the output mode (e.g., append, complete) is a required step to start the streaming process and manage how results are written.