Databricks Data Engineer Professional Question 66
Select 3You are working on a Databricks pipeline to write a PySpark DataFrame to disk in Parquet format, and you need to manually control the size of individual part-files being written to disk. Which of the following actions will help you achieve this?
- A
Use the
coalesce()method on the DataFrame before writing to reduce the number of output files. - B
Use the
repartition()method on the DataFrame before writing to control the number of output partitions. - C
Specify the
maxRecordsPerFileoption in the.writemethod to limit the number of rows per file. - D
Set the
spark.sql.files.maxPartitionBytesconfiguration property to control the size of each output file. - E
Use the
checkpoint()method on the DataFrame before writing to split large partitions into smaller files.
Show answer and explanation
Correct answers: A, B, C
Explanation
To manually control the size of individual part-files when writing a PySpark DataFrame to disk, you can use methods like coalesce() or repartition() to adjust the number of partitions, which directly correlates with the number of output files. Additionally, the maxRecordsPerFile option in the .write method allows control over the number of rows per file, indirectly managing file size. Configuration properties like spark.sql.files.maxPartitionBytes and operations like checkpoint() do not serve this purpose.
- A. Correct.
Using
coalesce()reduces the number of output partitions, which directly impacts the number of part-files written to disk. However, it does not increase the number of partitions, so its use is limited in scenarios requiring finer control. - B. Correct.
Using
repartition()allows you to explicitly define the number of partitions, which translates to the number of part-files written to disk. This provides fine-grained control over the output structure. - C. Correct.
The
maxRecordsPerFileoption in the.writemethod ensures that each file contains no more than the specified number of rows, thus helping control the size of individual part-files. - D. Incorrect.
spark.sql.files.maxPartitionBytescontrols how much data can be read into memory for processing, but it does not directly control the size of output part-files. - E. Incorrect.
checkpoint()is used for fault tolerance and resetting lineage but does not control the size of part-files written to disk.