Paper Abstract and Keywords |
Presentation |
2020-10-30 10:50
statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks Haruka Asanuma, Shiro Takagi, Yoshihiro Nagano, Yuki Yoshida (Tokyo Univ.), Yasuhiko Igarashi (Tsukuba Univ.), Masato Okada (Tokyo Univ.) NC2020-18 |
Abstract |
(in Japanese) |
(See Japanese page) |
(in English) |
When single neural networks sequentially learns more than one task, catastrophic forgetting occurs except for the last task.
It is empirically known from previous studies that the input space similarity and the task-to-task similarity are important factors that govern catastrophic forgetting.
While the method to avoid catastrophic forgetting has been studied, it is not clear how each factor specifically affects catastrophic forgetting as no consensus on a framework has been established to discuss catastrophic forgetting in a unified manner.
In this study, we modeled and analyzed a solvable model of single-layer linear neural networks of continual learning that incorporates input space similarity and task-to-task similarity using teacher and student learning and statistical mechanics formulation.
As a result of analysis, we found that the input space similarity and task-to-task similarity affect the catastrophic forgetting by multiplication. |
Keyword |
(in Japanese) |
(See Japanese page) |
(in English) |
catastrophic forgetting / a single-layer linear model / a regression problem / online learning / teacher and student learning / statistical mechanical analysis / generalization error / |
Reference Info. |
IEICE Tech. Rep., vol. 120, no. 216, NC2020-18, pp. 50-55, Oct. 2020. |
Paper # |
NC2020-18 |
Date of Issue |
2020-10-22 (NC) |
ISSN |
Online edition: ISSN 2432-6380 |
Copyright and reproduction |
All rights are reserved and no part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Notwithstanding, instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. (License No.: 10GA0019/12GB0052/13GB0056/17GB0034/18GB0034) |
Download PDF |
NC2020-18 |
|