This paper presents a note on accurate distress classification using deep learning considering confidence in attention maps. Specifically, this paper proposes confidence-aware attention branch network (ConfABN), which introduces a confidence-aware attention mechanism that reduces the influence of ineffective attention maps by considering the corresponding confidence in the attention maps. The confidence can be calculated from the entropy of the estimated class probabilities in generating the attention map, and it enables the weighting of feature maps using the effective attention map strongly and the ineffective attention map weakly. ConfABN can effectively utilize the attention mechanism to focus more attention on the regions that are important for the final estimation by considering the confidence. Experiments using images taken during inspections of actual infrastructures verify the effectiveness of the proposed method.