Databricks Data Engineer Professional Question 63
Select 3You are working on a PySpark job in Databricks, and you need to write a DataFrame to disk in Parquet format. However, your downstream system requires that each part-file written to disk is approximately 128 MB in size. Which of the following steps should you take to achieve this requirement?
- A
Use the
repartition()method on the DataFrame to increase or decrease the number of partitions before writing. - B
Set the
maxRecordsPerFileoption in the.writeoperation to control the size of the output files. - C
Use the
coalesce()method to reduce the number of partitions in your DataFrame before writing. - D
Set the
parquet.block.sizeconfiguration property in the Spark session to 128 MB. - E
Manually rename the part-files after writing them to ensure they match the desired size.
Show answer and explanation
Correct answers: A, C, D
Explanation
To control the size of part-files when writing a DataFrame to disk in PySpark, you can adjust the number of partitions using repartition() or coalesce() methods. Additionally, for Parquet files, you can configure the parquet.block.size property to influence the file size. The maxRecordsPerFile option is not relevant for controlling file size, and manually renaming files is not a valid solution. A combination of these techniques ensures that your files meet the required size constraints.
- A. Correct.
Repartitioning the DataFrame adjusts the number of partitions and indirectly influences the size of the output part-files. This is a recommended method.
- B. Incorrect.
The
maxRecordsPerFileoption controls the number of records per file, not the file size. This does not guarantee a specific file size. - C. Correct.
Using
coalesce()reduces the number of partitions, which can help control file sizes when used appropriately. This is valid for reducing the partition count. - D. Correct.
Setting the
parquet.block.sizeproperty in the Spark session directly impacts the block size of Parquet files, which helps control the part-file sizes. - E. Incorrect.
Manually renaming files does not control or change their size. This is not a valid approach for controlling part-file sizes.