Databricks Data Engineer Associate Question 85
Select 3You are working on an ELT pipeline using Apache Spark in Databricks. Your task is to load raw data from a Delta table, transform it to calculate monthly sales totals, and then write the transformed data back to another Delta table for reporting purposes. Which of the following steps in your Spark code is essential to ensure that the ELT process is both scalable and efficient?
- A
Use Spark’s
repartition()to optimize the number of partitions before writing the data. - B
Persist the intermediate transformed DataFrame in memory using
.persist()or.cache(). - C
Use Spark SQL to perform the transformation logic instead of DataFrame APIs.
- D
Enable Delta Lake’s
OPTIMIZEcommand on the target Delta table after writing the data. - E
Use
.coalesce(1)to write the output into a single file for easy access.
Show answer and explanation
Correct answers: A, B, D
Explanation
To ensure scalability and efficiency in an ELT pipeline with Apache Spark, it is critical to optimize partitioning using repartition(), persist intermediate DataFrames when reused, and leverage Delta Lake’s optimization features such as OPTIMIZE. Using .coalesce(1) should be avoided for large datasets as it creates a bottleneck, and choosing between Spark SQL and DataFrame APIs depends on use case specifics rather than inherent efficiency.
- A. Correct.
Correct: Adjusting the number of partitions using
repartition()ensures that the data is evenly distributed across the cluster for efficient parallel processing, especially for large datasets. - B. Correct.
Correct: Persisting the intermediate transformed DataFrame can improve performance when the same data is reused multiple times in the pipeline.
- C. Incorrect.
Incorrect: While Spark SQL is a valid method for transformations, it is not inherently more efficient than using DataFrame APIs. Efficiency depends on the specific query or transformation logic.
- D. Correct.
Correct: Delta Lake’s
OPTIMIZEcommand optimizes the layout of data files, improving query performance on the target Delta table. - E. Incorrect.
Incorrect: Using
.coalesce(1)forces all data to be written to a single file, which can severely impact scalability and performance in large-scale workflows.