Databricks Data Engineer Associate Question 84
Select 3A data engineering team is tasked with implementing an ELT pipeline using Apache Spark on Databricks. The source data resides in a Delta Lake table and needs to be transformed to calculate the monthly sales totals. The pipeline must ensure that transformations are efficient and incremental updates are supported. Which of the following steps should the team include in their implementation?
- A
Use Spark SQL to perform transformations and aggregate the data into monthly totals.
- B
Write the transformed data back into a new Delta Lake table.
- C
Use
mergeto incrementally load the source data into the Delta Lake table. - D
Use a JDBC connection to write the transformed data directly to an external database.
- E
Use
cache()to store intermediate results for all transformations.
Show answer and explanation
Correct answers: A, B, C
Explanation
To implement an ELT pipeline using Apache Spark on Databricks, the team should leverage Spark SQL for transformations, store the results in Delta Lake (which supports incremental updates), and use merge for efficient handling of new or updated source data. Writing data directly to an external database bypasses Delta Lake and is not aligned with the ELT methodology. Additionally, indiscriminate use of cache() can lead to inefficient resource usage.
- A. Correct.
This is correct because Spark SQL is an efficient way to perform transformations and aggregations, such as calculating monthly sales totals.
- B. Correct.
This is correct because writing the transformed data to a Delta Lake table allows for efficient storage and supports incremental updates.
- C. Correct.
This is correct because using
mergeensures that the pipeline can handle incremental data updates efficiently in Delta Lake. - D. Incorrect.
This is incorrect because writing data directly to an external database using JDBC is not an ELT approach and can be less efficient compared to leveraging Delta Lake.
- E. Incorrect.
This is incorrect because
cache()is not necessary for all transformations and can lead to inefficient resource utilization if not used appropriately.