Databricks Data Engineer Professional Question 64
Select 3You are working on a PySpark job that processes a large dataset and writes the output to disk in Parquet format. To optimize downstream processing, you need to control the size of individual part-files in the output directory. Which of the following approaches will help you achieve this?
- A
Use the
coalesce()method to reduce the number of partitions before writing the DataFrame. - B
Use the
repartition()method to increase or decrease the number of partitions before writing the DataFrame. - C
Set the
spark.sql.files.maxRecordsPerFileconfiguration to limit the number of records per output file. - D
Manually rename the part-files generated after writing the DataFrame to enforce a desired file size.
- E
Use the
partitionBy()method when writing the DataFrame to control the size of individual files.
Show answer and explanation
Correct answers: A, B, C
Explanation
To control the size of individual part-files in a PySpark job, you can adjust the number of partitions using methods like coalesce() and repartition(), or configure spark.sql.files.maxRecordsPerFile to limit records per file. These approaches allow you to manage file sizes during the write process. Simply renaming files or using partitionBy() does not influence file sizes.
- A. Correct.
Correct: The
coalesce()method reduces the number of partitions, which can effectively increase the size of each individual part-file when writing to disk. This is useful for controlling file sizes after transformations. - B. Correct.
Correct: The
repartition()method allows you to explicitly control the number of partitions. Since the number of partitions dictates the number of output files, adjusting the partitions can help achieve the desired file size. - C. Correct.
Correct: The
spark.sql.files.maxRecordsPerFileconfiguration can limit the number of records written to each part-file, indirectly controlling the file size. - D. Incorrect.
Incorrect: Renaming part-files after they are written does not change their size. File sizes are determined during the write process based on partitions and configuration settings.
- E. Incorrect.
Incorrect: The
partitionBy()method is used for organizing output data into separate folders based on column values, but it does not control the size of individual files within those folders.