torch-resnet

Unified torch implementation of resnets with or without pre-activation/width.

We implement (pre-act) resnets for ImageNet for each size described in [1] and [2] (18, 34, 50, 101, 152, 200). We also implement (pre-act) resnets for Cifar for each size described in [1] and [2] (20, 32, 44, 56, 110, 164, 1001, 1202).

Following what is used in the literature we also propose a version of the ImageNet resnets for Cifar10 (See SimClr paper https://arxiv.org/pdf/2002.05709.pdf). To adapt the architectures to smaller images (32x32), the initial layers conv 7x7 and max_pool are replaced by a simple conv 3x3. The other layers/parameters are kept, resulting in a variant of wide-resnets for Cifar.

Finally, we implement wide resnets (with or without pre-activation) for Cifar and ImageNet following [3].

Additional models can easily be created using the default class ResNet or PreActResNet. It is also possible to create your own block following the same model as those implemented. (To create for instance the different variations of resnets proposed in [1], [2])

We have validated our implementation by testing it on Cifar10/Cifar100 (See Results).

Install

$ pip install torch-resnet

Getting started

import torch

import torch_resnet
from torch_resnet.utils import count_layer, count_parameters

model = torch_resnet.PreActResNet50()  # Build a backbone Resnet50 with pre-activation for ImageNet
model.set_head(nn.Linear(model.out_planes, 1000))  # Set a final linear head

count_layers(model)  # -> 54 (In the original paper they do not count shortcut/downsampling layers)
count_parameters(model) / 10**6  # Nb. parameters in millions

out = model(torch.randn(1, 3, 224, 224))

See example/example.py for a more complete example.

Results

Results obtained with example/example.py following closely papers indications. Most are reported as mean $\pm$ std (on 5 runs with different seed). If a single number is reported, only a single run has been done due to computational time. All training are done with Automatique Mixed Precision.

For all resnets and pre-act resnets, the learning rate is scheduled following [1] and [2] with a warm-up in the 400 first iterations at 0.01. The initial learning rate is then set at 0.1 and decreased by 10 at 32k and 48k iterations. Training is stopped after 160 epochs (~62.5k iterations).

Contrary to [1], the training set is not split in 45k/5k to perform validation, and we directly took the final model to evaluate the performances on the test set (No validation is done).

Follow the link to access the training curves for each model and each seed used (111, 222, 333, 444, 555). (Upload in coming...)

Note that in [2] and in most github implementation, the test set is used as a validation set (taking the max acc reached on it as the final result), obviously leading to falsely better performances. When dropping AMP and taking the max value rather than the last value, we also reach better performances. (TODO)

Analysis

TODO

References

• [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Deep Residual Learning for Image Recognition. https://arxiv.org/pdf/1512.03385
• [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Identity Mappings in Deep Residual Networks. https://arxiv.org/pdf/1603.05027
• [3] Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks. https://arxiv.org/pdf/1605.07146

Build and Deploy

$ python -m build
$ python -m twine upload dist/*