ARA-C01 Question 305
Single answerSparkA retail company runs a nightly Spark job on Databricks to transform clickstream data and write the results to Snowflake. The current implementation uses the Spark connector in a single batch at the end of the job. During peak season, the job frequently fails near completion because the Spark cluster autoscaling policy removes executors, causing long-running tasks to be retried and the final write to Snowflake to restart. The architects need a design that improves reliability and write throughput without changing the upstream transformation logic. Which approach should they recommend?
- A
Configure the Spark connector to write via an internal stage using parallel data transfer, and repartition the DataFrame before the write so output is distributed across multiple partitions
- B
Replace the Snowflake Spark connector with JDBC and increase the JDBC batch size so each executor writes larger INSERT statements directly into Snowflake tables
- C
Cache the final DataFrame in Spark memory before writing so that executor loss does not affect the Snowflake load phase
- D
Disable autoscaling on the Spark cluster and reduce Snowflake virtual warehouse size so the write operation completes with fewer concurrent connections
Show answer and explanation
Correct answer: A
Explanation
For Spark-to-Snowflake data movement at scale, the recommended architecture is to use the Snowflake Connector for Spark, which is optimized for bulk transfer through staging and COPY-based loading rather than direct row-by-row JDBC inserts. In practice, architects should also ensure that Spark partitioning supports sufficient parallelism for the connector's transfer phase. This is especially important when clusters are elastic or subject to executor churn, because a properly partitioned staged load is generally more robust and performant than a monolithic final write pattern. Snowflake documentation for the Spark connector and bulk loading guidance emphasizes using the native connector for scalable data exchange and leveraging partitioning/parallelism appropriately.
- A. Correct.
Correct. The Snowflake Connector for Spark is designed to move data efficiently by unloading/loading through a stage rather than row-by-row JDBC INSERT patterns. Repartitioning the DataFrame can improve parallelism and throughput by producing multiple output partitions for the connector to transfer concurrently. This approach improves resiliency compared with a single narrow write path and aligns with Snowflake best practices for large Spark-to-Snowflake writes.
- B. Incorrect.
Incorrect. Using generic JDBC for large-scale Spark writes is typically less efficient and less reliable than the Snowflake Spark connector because JDBC commonly results in row-oriented or small-batch insert behavior. Increasing JDBC batch size may help somewhat, but it does not match the staged bulk-load pattern that the Snowflake connector uses for scalable loads.
- C. Incorrect.
Incorrect. Caching the DataFrame may reduce recomputation if tasks are retried, but it does not eliminate the core issue of how data is written to Snowflake or guarantee resilience when executors are removed. Cached data can also be lost when executors are decommissioned, especially if not fully materialized with sufficient replication, so this is not the best architectural fix.
- D. Incorrect.
Incorrect. Disabling autoscaling might reduce one source of instability, but it does not address the need for an optimized Snowflake write strategy. Reducing the Snowflake warehouse size would likely hurt load performance, not improve it. The requirement is to improve reliability and throughput without changing upstream logic, and the connector's staged parallel load mechanism is the appropriate recommendation.