PyTorch implementation of ZerO Initialization
pip install zero-init==0.2
A PyTorch implementation of ZerO initialization (for Linear and Conv2d layers)
Original Paper: https://arxiv.org/abs/2110.12661
# Import
from zero_init import ZerO_Init
# Define your model
model = ...
# Apply ZerO Initialization
# NOTE: Only initializes torch.nn.Linear and torch.nn.Conv2d layers
model.apply(ZerO_Init)
The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.
Learn more →
Login to resync this project