Paper Abstract and Keywords |
Presentation |
2017-11-10 13:00
Statistical Mechanical Analysis of Learning with Two-Layer Perceptron with Multiple Output Units
-- Reconsidering Plateau Phenomenon -- Yuki Yoshida (UTokyo), Ryo Karakida (AIST), Masato Okada (UTokyo/AIST/RIKEN BSI), Shun-ichi Amari (RIKEN BSI) IBISML2017-83 |
Abstract |
(in Japanese) |
(See Japanese page) |
(in English) |
The plateau phenomenon --- stopping decrease of error in the middle of learning --- is problematic.
Since Amari et al. pointed out that singular regions stemming from structure of neural network cause such plateaus,
dynamics of learning with two-layered networks has been studied intensively. However, all these researches deal
with networks which have one-dimensional output, thus two-or-more dimensional output cases are
overlooked.according to Amari et al., such networks might be unlikely to be trapped in a plateau.
In this article, we analyze the dynamics of learning with two-layer perceptron with
multidimensional output under a statistical mechanical formalization.
We derive order parameters which capture macroscopic characteristics of connection weights and
differential equations which they follow.
We show that singular-region-driven plateaus do diminish or vanish
at least in a simple setting with multidimensional output.
We found that more the model being non-degenerative (i.e. far from one-dimensional output),
the more plateaus are alleviated.
Moreover, we showed theoretically that singular-region-driven plateaus are not shown at all
in the case with orthogonalized initializations. |
Keyword |
(in Japanese) |
(See Japanese page) |
(in English) |
Neural network / Perceptron / Plateau / Statistical Mechanics / / / / |
Reference Info. |
IEICE Tech. Rep., vol. 117, no. 293, IBISML2017-83, pp. 347-354, Nov. 2017. |
Paper # |
IBISML2017-83 |
Date of Issue |
2017-11-02 (IBISML) |
ISSN |
Print edition: ISSN 0913-5685 Online edition: ISSN 2432-6380 |
Copyright and reproduction |
All rights are reserved and no part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Notwithstanding, instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. (License No.: 10GA0019/12GB0052/13GB0056/17GB0034/18GB0034) |
Download PDF |
IBISML2017-83 |
Conference Information |
Committee |
IBISML |
Conference Date |
2017-11-08 - 2017-11-10 |
Place (in Japanese) |
(See Japanese page) |
Place (in English) |
Univ. of Tokyo |
Topics (in Japanese) |
(See Japanese page) |
Topics (in English) |
Information-Based Induction Science Workshop (IBIS2017) |
Paper Information |
Registration To |
IBISML |
Conference Code |
2017-11-IBISML |
Language |
Japanese |
Title (in Japanese) |
(See Japanese page) |
Sub Title (in Japanese) |
(See Japanese page) |
Title (in English) |
Statistical Mechanical Analysis of Learning with Two-Layer Perceptron with Multiple Output Units |
Sub Title (in English) |
Reconsidering Plateau Phenomenon |
Keyword(1) |
Neural network |
Keyword(2) |
Perceptron |
Keyword(3) |
Plateau |
Keyword(4) |
Statistical Mechanics |
Keyword(5) |
|
Keyword(6) |
|
Keyword(7) |
|
Keyword(8) |
|
1st Author's Name |
Yuki Yoshida |
1st Author's Affiliation |
The University of Tokyo (UTokyo) |
2nd Author's Name |
Ryo Karakida |
2nd Author's Affiliation |
National Institute of Advanced Industrial Science and Technology (AIST) |
3rd Author's Name |
Masato Okada |
3rd Author's Affiliation |
The University of Tokyo/National Institute of Advanced Industrial Science and Technology/RIKEN Brain Science Institute (UTokyo/AIST/RIKEN BSI) |
4th Author's Name |
Shun-ichi Amari |
4th Author's Affiliation |
RIKEN Brain Science Institute (RIKEN BSI) |
5th Author's Name |
|
5th Author's Affiliation |
() |
6th Author's Name |
|
6th Author's Affiliation |
() |
7th Author's Name |
|
7th Author's Affiliation |
() |
8th Author's Name |
|
8th Author's Affiliation |
() |
9th Author's Name |
|
9th Author's Affiliation |
() |
10th Author's Name |
|
10th Author's Affiliation |
() |
11th Author's Name |
|
11th Author's Affiliation |
() |
12th Author's Name |
|
12th Author's Affiliation |
() |
13th Author's Name |
|
13th Author's Affiliation |
() |
14th Author's Name |
|
14th Author's Affiliation |
() |
15th Author's Name |
|
15th Author's Affiliation |
() |
16th Author's Name |
|
16th Author's Affiliation |
() |
17th Author's Name |
|
17th Author's Affiliation |
() |
18th Author's Name |
|
18th Author's Affiliation |
() |
19th Author's Name |
|
19th Author's Affiliation |
() |
20th Author's Name |
|
20th Author's Affiliation |
() |
Speaker |
Author-1 |
Date Time |
2017-11-10 13:00:00 |
Presentation Time |
150 minutes |
Registration for |
IBISML |
Paper # |
IBISML2017-83 |
Volume (vol) |
vol.117 |
Number (no) |
no.293 |
Page |
pp.347-354 |
#Pages |
8 |
Date of Issue |
2017-11-02 (IBISML) |
|