Affordable Access

deepdyve-link
Publisher Website

A New Delay Connection for Long Short-Term Memory Networks.

Authors
  • Wang, Jianyong1
  • Zhang, Lei1
  • Chen, Yuanyuan1
  • Yi, Zhang1
  • 1 1 Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, Sichuan, P. R. China. , (China)
Type
Published Article
Journal
International journal of neural systems
Publication Date
Aug 01, 2018
Volume
28
Issue
6
Pages
1750061–1750061
Identifiers
DOI: 10.1142/S0129065717500617
PMID: 29382286
Source
Medline
Keywords
Language
English
License
Unknown

Abstract

Connections play a crucial role in neural network (NN) learning because they determine how information flows in NNs. Suitable connection mechanisms may extensively enlarge the learning capability and reduce the negative effect of gradient problems. In this paper, a new delay connection is proposed for Long Short-Term Memory (LSTM) unit to develop a more sophisticated recurrent unit, called Delay Connected LSTM (DCLSTM). The proposed delay connection brings two main merits to DCLSTM with introducing no extra parameters. First, it allows the output of the DCLSTM unit to maintain LSTM, which is absent in the LSTM unit. Second, the proposed delay connection helps to bridge the error signals to previous time steps and allows it to be back-propagated across several layers without vanishing too quickly. To evaluate the performance of the proposed delay connections, the DCLSTM model with and without peephole connections was compared with four state-of-the-art recurrent model on two sequence classification tasks. DCLSTM model outperformed the other models with higher accuracy and F1[Formula: see text]score. Furthermore, the networks with multiple stacked DCLSTM layers and the standard LSTM layer were evaluated on Penn Treebank (PTB) language modeling. The DCLSTM model achieved lower perplexity (PPL)/bit-per-character (BPC) than the standard LSTM model. The experiments demonstrate that the learning of the DCLSTM models is more stable and efficient.

Report this publication

Statistics

Seen <100 times