Databricks Data Engineer Professional Question 62
Select 3You are working on a PySpark job where you need to write a DataFrame to disk in Parquet format. To optimize downstream processing, you want to control the size of each part-file generated during the write operation. Which of the following approaches will allow you to control the size of the part-files manually?
- A
Use the
coalesce(n)method before writing to reduce the number of partitions. - B
Use the
repartition(n)method before writing to increase or decrease the number of partitions. - C
Specify the desired file size in bytes using the
maxFileSizeoption when writing the DataFrame. - D
Use the
df.write.option('partitionBy', 'col_name')to control the size of the part-files. - E
Manually repartition the DataFrame and ensure that the size of each partition aligns with the desired part-file size.
Show answer and explanation
Correct answers: A, B, E
Explanation
Controlling the size of part-files during a PySpark write operation requires managing the size and number of partitions in the DataFrame. Methods like coalesce(n) and repartition(n) adjust the number of partitions, thereby affecting the size of the output files. Additionally, manually estimating and adjusting partition sizes based on the desired output can provide precise control. Options like maxFileSize or partitionBy do not directly address file size management.
- A. Correct.
Correct. The
coalesce(n)method reduces the number of partitions in a DataFrame, which can indirectly control the number and size of output part-files. This works well when you want fewer, larger files. - B. Correct.
Correct. The
repartition(n)method allows you to increase or decrease the number of partitions in a DataFrame. This gives you precise control over the number of part-files generated during the write operation. - C. Incorrect.
Incorrect. There is no
maxFileSizeoption available in PySpark for controlling part-file sizes during writes. This is not a valid method. - D. Incorrect.
Incorrect. The
partitionByoption is used to partition the output data by column values into separate folders, but it does not control the size of individual part-files. - E. Correct.
Correct. Manually repartitioning the DataFrame and tuning the size of each partition before writing allows you to control the size of the resulting part-files. This is a common and effective approach.