An Evaluation of Progressive Neural Networks for Transfer Learning in Natural Language Processing
Published in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020
A major challenge in modern neural networks is the utilization of previous knowledge for new tasks in an effective manner, otherwise known as transfer learning. Fine-tuning, the most widely used method for achieving this, suffers from catastrophic forgetting. The problem is often exacerbated in natural language processing (NLP). In this work, we assess progressive neural networks (PNNs) as an alternative to fine-tuning. The evaluation is based on common NLP tasks such as sequence labeling and text classification. By gauging PNNs across a range of architectures, datasets, and tasks, we observe improvements over the baselines throughout all experiments.
Recommended citation: Gerhard Hagerer1†, Abdul Moeed1†, Sumit Dugar1†, Sarthak Gupta1†, Mainak Ghosh1†, Hannah Danner, Oliver Mitevski, Andreas Nawroth, Georg Groh, LREC 2020) https://aclanthology.org/2020.lrec-1.172.pdf