Google Professional Machine Learning Engineer Question 291
Single answerGoogle Cloud PlatformYou are training a machine learning model on Vertex AI using a custom training job. The dataset is large, and you want to accelerate training by leveraging multiple GPUs or TPUs with distributed training. After setting up your training script, you notice that the model fails to converge properly when scaling to multiple devices. What could be the most likely cause, and how can you fix it?
- A
The model's learning rate is too high. Reduce the learning rate when scaling to multiple devices.
- B
The dataset is not being correctly sharded across the devices. Use a library like TensorFlow's
tf.datato shard the data. - C
The distributed training setup doesn't include a Reduction Server. Add a Reduction Server to handle gradient aggregation.
- D
The communication backend, such as Horovod or TensorFlow's
tf.distribute, is not configured correctly. Verify the backend settings.
Show answer and explanation
Correct answer: B
Explanation
In distributed training, it is critical to ensure that the dataset is correctly sharded across devices. Sharding ensures that each device processes a unique portion of the data, preventing data duplication that can disrupt convergence. Libraries like TensorFlow's tf.data provide tools to shard datasets efficiently. While learning rate adjustments and proper backend configuration are important, the issue described here is specifically related to improper data sharding.
- A. Incorrect.
While learning rate adjustments might help with training stability in some cases, this is not the most likely issue when the model fails to converge in a multi-device distributed training scenario.
- B. Correct.
Correct. When scaling to multiple devices, the dataset must be properly sharded so that each device receives a unique subset of the data. Without sharding, some devices might process duplicate data, which can lead to poor model convergence.
- C. Incorrect.
While a Reduction Server can improve gradient aggregation efficiency in distributed setups, its absence typically results in slower performance rather than convergence issues.
- D. Incorrect.
Incorrect backend configuration could cause training to fail entirely or lead to runtime errors, but it is less likely to be the specific cause of convergence problems.