NeuroSurgeon

NeuroSurgeon is a python toolkit built to enable deep learning researchers to easily uncover and manipulate subnetworks within trained models. NeuroSurgeon provides a simple API to inject differentiable binary masks techniques into linear, attention, and convolution layers in BERT, GPT, ResNet, and ViT-style models within Huggingface Transformers. Differentiable masking has a variety of use cases for deep learning research, such as:

• Pruning to uncover functional subnetworks
  • Relevant Papers:
    • Csordás et al. 2021
    • Lepori et al. 2023
    • Panigrahi et al. 2023
• Subnetwork Probing
  • Relevant Papers:
    • Cao et al. 2021
    • Conmy et al. 2023
• Training with L0 Regularization
  • Relevant Papers:
    • Louizos et al. 2018
    • Savarese et al. 2020

Documentation

Read the NeuroSurgeon Documentation

Install

NeuroSurgeon requires python 3.9 or higher and several libraries, including Transformers and PyTorch. Installation can be done using PyPi:

pip install NeuroSurgeon