Databricks Data Engineer Professional Question 65
Single answerYou are working with a PySpark DataFrame and need to write it to disk in Parquet format while ensuring that each part-file generated is approximately 128 MB in size. How can you achieve this?
- A
Use the repartition() method to specify the number of partitions based on the desired file size before writing the DataFrame to disk.
- B
Set the configuration spark.sql.files.maxPartitionBytes to 128 MB before writing the DataFrame.
- C
Use the coalesce() method to reduce the number of partitions to achieve the desired file size before writing the DataFrame to disk.
- D
Specify the maxRecordsPerFile option when writing the DataFrame to control the size of the part-files.
Show answer and explanation
Correct answer: A
Explanation
To control the size of part-files when writing a PySpark DataFrame to disk, you must manage the number of partitions in the DataFrame. Using the repartition() method allows you to evenly distribute the data into a specific number of partitions, which can be calculated based on the total data size and the desired file size. The other options either do not directly address file size or are not the most effective approach.
- A. Correct.
This is correct. You can use the repartition() method to control the number of partitions in the DataFrame, which directly affects the size of the part-files when written to disk. For example, calculate the number of partitions based on the total data size and the desired file size.
- B. Incorrect.
This is incorrect. The spark.sql.files.maxPartitionBytes configuration determines the maximum size of data processed per partition during queries, not the size of output part-files written to disk.
- C. Incorrect.
This is partially correct but not the best option. The coalesce() method can reduce the number of partitions, but it is typically used for optimization after shuffles and may not evenly distribute data for consistent file sizes.
- D. Incorrect.
This is incorrect. The maxRecordsPerFile option controls the number of rows in each file, not the file size itself. File size depends on the data's schema and compression.