Databricks Data Engineer Professional Question 61
Select 2You are working with a PySpark DataFrame in Databricks and need to write the data to disk in Parquet format. The goal is to control the size of individual part-files to approximately 100 MB. Which of the following approaches will help you achieve this?
- A
Use the repartition() function to increase the number of partitions based on an estimated file size and then write the DataFrame.
- B
Use the coalesce() function to reduce the number of partitions and then write the DataFrame.
- C
Set the 'spark.sql.files.maxPartitionBytes' configuration to 100 MB before writing the DataFrame.
- D
Write the DataFrame directly without modifying its partitioning, as Spark automatically handles file sizes.
- E
Use the option('maxRecordsPerFile', value) in the write operation to control file size.
Show answer and explanation
Correct answers: A, B
Explanation
To control the size of part-files when writing a PySpark DataFrame, you need to adjust the number of partitions using repartition() or coalesce() before writing the DataFrame. Repartitioning increases the number of partitions, resulting in smaller file sizes, while coalescing reduces the number of partitions, which can be helpful if there are too many small partitions. Other configurations and options mentioned in the question either do not influence the file size during writing or affect it indirectly.
- A. Correct.
Using the repartition() function allows you to control the number of partitions, which directly affects the size of the part-files generated during the write operation. By increasing the number of partitions, you reduce the size of each file.
- B. Correct.
Using the coalesce() function can reduce the number of partitions, which may be necessary if there are too many small partitions. This can help optimize the file size when writing the DataFrame.
- C. Incorrect.
The 'spark.sql.files.maxPartitionBytes' configuration controls the maximum size of a single partition when reading files, but it does not influence the size of part-files during a write operation.
- D. Incorrect.
Writing the DataFrame directly without adjusting partitioning will not allow you to control the size of part-files. Spark's default behavior may not align with the desired file size.
- E. Incorrect.
The option('maxRecordsPerFile', value) controls the number of records per file, not the file size. While this can indirectly affect file size, it is not a direct way to target a specific file size like 100 MB.