Deep Learning Seminar  /  July 11, 2019

Emergence of Implicit Filter Sparsity in Convolutional Neural Networks: Examining the Causes and Implications

Abstract:

In typical convolutional neural networks comprised of Conv-BN-ReLU blocks, a significant number of convolutional filters get pruned away during the course of training. For ResNet-50 on ImageNet, about 20% of the filters are implicitly pruned when using ADAM, while other networks see up to 70% of the filters pruned for simpler tasks like CIFAR-10. We study the emergence of this puzzling structured sparsity, and how seemingly unrelated hyperparameters such as mini-batch size, the choice of optimizer, task difficulty etc. affect the extent of sparsity. Our experimentally backed hypothesis suggests that selective features (i.e., those that activate for a small subset of the training corpus) see disproportionately higher regularization than more general features, and consequently decay away. Various hyperparameters influence the extent of sparsity by either impacting the strength of regularization or the selectivity of features. The emergent structured sparsity can be exploited for convolutional network speedup on current hardware. Further, since network capacity directly interplays with training error and generalization ability, both theoreticians and practitioners need to be aware of this inadvertent reduction in network capacity which is impacted by seemingly innocuous hyperparameters.

The seminar »KL-Regelungstechnik« (Kaiserslautern – Control Theory and Control Engineering) is organized by our department as well as several research groups of the TU Kaiserslautern:

  • Technomathematics (Dep. of Mathematics)
  • Mechatronics in Mechenical and Automotive Engineering (Dep. of Mechanical and Process Engineering)
  • Automation Control (Dep. of Electrical and Computer Engineering)
  • Electromobility (Dep. of Electrical and Computer Engineering)

The seminar takes place at the ITWM every 1st Tuesday of a month (besides holidays and summer break). Aims are broadening of experiences and exchange of scientific views – also beyond the organizing groups.

Typical subjects of talks are:

  • ongoing or recently finished graduations and doctoral theses
  • current research and projects

The topics vary from mathematical methods to technical implementations. Usually, the talks present research results. However, some show open issues for brainstorming and inputs from the audience.

The seminar »KL-Regelungstechnik« (Kaiserslautern – Control Theory and Control Engineering) is organized by our department as well as several research groups of the TU Kaiserslautern:

  • Technomathematics (Dep. of Mathematics)
  • Mechatronics in Mechenical and Automotive Engineering (Dep. of Mechanical and Process Engineering)
  • Automation Control (Dep. of Electrical and Computer Engineering)
  • Electromobility (Dep. of Electrical and Computer Engineering)

The seminar takes place at the ITWM every 1st Tuesday of a month (besides holidays and summer break). Aims are broadening of experiences and exchange of scientific views – also beyond the organizing groups.

Typical subjects of talks are:

  • ongoing or recently finished graduations and doctoral theses
  • current research and projects

The topics vary from mathematical methods to technical implementations. Usually, the talks present research results. However, some show open issues for brainstorming and inputs from the audience.

The seminar »KL-Regelungstechnik« (Kaiserslautern – Control Theory and Control Engineering) is organized by our department as well as several research groups of the TU Kaiserslautern:

  • Technomathematics (Dep. of Mathematics)
  • Mechatronics in Mechenical and Automotive Engineering (Dep. of Mechanical and Process Engineering)
  • Automation Control (Dep. of Electrical and Computer Engineering)
  • Electromobility (Dep. of Electrical and Computer Engineering)

The seminar takes place at the ITWM every 1st Tuesday of a month (besides holidays and summer break). Aims are broadening of experiences and exchange of scientific views – also beyond the organizing groups.

Typical subjects of talks are:

  • ongoing or recently finished graduations and doctoral theses
  • current research and projects

The topics vary from mathematical methods to technical implementations. Usually, the talks present research results. However, some show open issues for brainstorming and inputs from the audience.