| （英） |
In distributed deep learning, where multiple processors are used, the learning time can be significantly reduced by executing the learning process on each processor. However, distributed deep learning requires communication of gradient data between processors during learning, and the communication processing time significantly affects the learning time. To reduce the communication processing time, pipelining methods based on pipeline processing and gradient compression methods based on data compression are used. Here, we have proposed a communication scheduling based on a heuristic algorithm that considers multiple data compression techniques and communication methods. The proposed method can quickly derive a unique solution from multiple compression techniques and communication schemes at each layer using a heuristic algorithm, enabling distributed processing on multiple processors. In this paper, we investigate the optimal compression rate in each layer for data compression and communication methods.
We show the impact of compression rate for the parallel distributed deep learning. |