diff options
Diffstat (limited to 'python-torchlayers-nightly.spec')
| -rw-r--r-- | python-torchlayers-nightly.spec | 766 |
1 files changed, 766 insertions, 0 deletions
diff --git a/python-torchlayers-nightly.spec b/python-torchlayers-nightly.spec new file mode 100644 index 0000000..eb76365 --- /dev/null +++ b/python-torchlayers-nightly.spec @@ -0,0 +1,766 @@ +%global _empty_manifest_terminate_build 0 +Name: python-torchlayers-nightly +Version: 1681171628 +Release: 1 +Summary: Input shape inference and SOTA custom layers for PyTorch. +License: MIT +URL: https://github.com/pypa/torchlayers +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/f0/36/65b339c5b9cafbb957cc5663454075369fdd9ec99f2d5fbfaf99ad2811c2/torchlayers-nightly-1681171628.tar.gz +BuildArch: noarch + +Requires: python3-torch + +%description +| Version | Docs | Tests | Coverage | Style | PyPI | Python | PyTorch | Docker | +|---------|------|-------|----------|-------|------|--------|---------|--------| +| [](https://github.com/szymonmaszke/torchlayers/releases) | [](https://szymonmaszke.github.io/torchlayers/) |  | [](https://codecov.io/gh/szymonmaszke/torchlayers) | [](https://codebeat.co/projects/github-com-szymonmaszke-torchlayers-master) | [](https://pypi.org/project/torchlayers/) | [](https://www.python.org/) | [](https://pytorch.org/) | [](https://hub.docker.com/r/szymonmaszke/torchlayers) | +[__torchlayers__](https://szymonmaszke.github.io/torchlayers/) is a library based on [__PyTorch__](https://pytorch.org/) +providing __automatic shape and dimensionality inference of `torch.nn` layers__ + additional +building blocks featured in current SOTA architectures (e.g. [Efficient-Net](https://arxiv.org/abs/1905.11946)). +Above requires no user intervention (except single call to `torchlayers.build`) +similarly to the one seen in [__Keras__](https://www.tensorflow.org/guide/keras). +### Main functionalities: +* __Shape inference__ for most of `torch.nn` module (__convolutional, recurrent, transformer, attention and linear layers__) +* __Dimensionality inference__ (e.g. `torchlayers.Conv` working as `torch.nn.Conv1d/2d/3d` based on `input shape`) +* __Shape inference of custom modules__ (see examples section) +* __Additional [Keras-like](https://www.tensorflow.org/guide/keras) layers__ (e.g. `torchlayers.Reshape` or `torchlayers.StandardNormalNoise`) +* __Additional SOTA layers__ mostly from ImageNet competitions +(e.g. [PolyNet](https://arxiv.org/abs/1608.06993), +[Squeeze-And-Excitation](https://arxiv.org/abs/1709.01507), +[StochasticDepth](www.arxiv.org/abs/1512.03385>)) +* __Useful defaults__ (`"same"` padding and default `kernel_size=3` for `Conv`, dropout rates etc.) +* __Zero overhead and [torchscript](https://pytorch.org/docs/stable/jit.html) support__ +__Keep in mind this library works almost exactly like PyTorch originally__. +What that means is you can use `Sequential`, __define your own networks of any complexity using +`torch.nn.Module`__, create new layers with shape inference etc. +_See below to get some intuition about library_. +# Examples +For full functionality please check [__torchlayers documentation__](https://szymonmaszke.github.io/torchlayers/). +Below examples should introduce all necessary concepts you should know. +## Basic classifier +__All__ `torch.nn` modules can be used through `torchlayers` and __each module with input shape__ +will be appropriately modified with it's input inferable counterpart. +```python +import torchlayers as tl +class Classifier(tl.Module): + def __init__(self): + super().__init__() + self.conv1 = tl.Conv2d(64, kernel_size=6) + self.conv2 = tl.Conv2d(128, kernel_size=3) + self.conv3 = tl.Conv2d(256, kernel_size=3, padding=1) + # New layer, more on that in the next example + self.pooling = tl.GlobalMaxPool() + self.dense = tl.Linear(10) + def forward(self, x): + x = torch.relu(self.conv1(x)) + x = torch.relu(self.conv2(x)) + x = torch.relu(self.conv3(x)) + return self.dense(self.pooling(x)) +# Pass model and any example inputs afterwards +clf = tl.build(Classifier(), torch.randn(1, 3, 32, 32)) +``` +Above `torchlayers.Linear(out_features=10)` is used. It is "equivalent" to +original PyTorch's `torch.nn.Linear(in_features=?, out_features=10)` where `in_features` +will be inferred from example input input during `torchlayers.build` call. +Same thing happens with `torch.nn.Conv2d(in_channels, out_channels, kernel_size, ...)` +which can be replaced directly by `tl.Conv2d(out_channels, kernel_size, ...)`. +__Just remember to pass example input through the network!__ +## Simple image and text classifier in one! +* We will use single "model" for both tasks. +Firstly let's define it using `torch.nn` and `torchlayers`: +```python +import torch +import torchlayers as tl +# torch.nn and torchlayers can be mixed easily +model = torch.nn.Sequential( + tl.Conv(64), # specify ONLY out_channels + torch.nn.ReLU(), # use torch.nn wherever you wish + tl.BatchNorm(), # BatchNormNd inferred from input + tl.Conv(128), # Default kernel_size equal to 3 + tl.ReLU(), + tl.Conv(256, kernel_size=11), # "same" padding as default + tl.GlobalMaxPool(), # Known from Keras + tl.Linear(10), # Output for 10 classes +) +print(model) +``` +Above would give you model's summary like this (__notice question marks for not yet inferred values__): +```python +Sequential( + (0): Conv(in_channels=?, out_channels=64, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (1): ReLU() + (2): BatchNorm(num_features=?, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (3): Conv(in_channels=?, out_channels=128, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (4): ReLU() + (5): Conv(in_channels=?, out_channels=256, kernel_size=11, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (6): GlobalMaxPool() + (7): Linear(in_features=?, out_features=10, bias=True) +) +``` +* Now you can __build__/instantiate your model with example input (in this case MNIST-like): +```python +mnist_model = tl.build(model, torch.randn(1, 3, 28, 28)) +``` +* Or if it's text classification you are after, same model could be built with different +`input shape` (e.g. for text classification using `300` dimensional pretrained embedding): +```python +# [batch, embedding, timesteps], first dimension > 1 for BatchNorm1d to work +text_model = tl.build(model, torch.randn(2, 300, 1)) +``` +* Finally, you can `print` both models after instantiation, provided below side +by-side for readability (__notice different dimenstionality, e.g. `Conv2d` vs `Conv1d` after `torchlayers.build`__): +```python + # TEXT CLASSIFIER MNIST CLASSIFIER + Sequential( Sequential( + (0): Conv1d(300, 64) (0): Conv2d(3, 64) + (1): ReLU() (1): ReLU() + (2): BatchNorm1d(64) (2): BatchNorm2d(64) + (3): Conv1d(64, 128) (3): Conv2d(64, 128) + (4): ReLU() (4): ReLU() + (5): Conv1d(128, 256) (5): Conv2d(128, 256) + (6): GlobalMaxPool() (6): GlobalMaxPool() + (7): Linear(256, 10) (7): Linear(256, 10) + ) ) +``` +As you can see both modules "compiled" into original `pytorch` layers. +## Custom modules with shape inference capabilities +User can define any module and make it shape inferable with `torchlayers.infer` +function: +```python + # Class defined with in_features + # It might be a good practice to use _ prefix and Impl as postfix + # to differentiate from shape inferable version +class _MyLinearImpl(torch.nn.Module): + def __init__(self, in_features: int, out_features: int): + super().__init__() + self.weight = torch.nn.Parameter(torch.randn(out_features, in_features)) + self.bias = torch.nn.Parameter(torch.randn(out_features)) + def forward(self, inputs): + return torch.nn.functional.linear(inputs, self.weight, self.bias) +MyLinear = tl.infer(_MyLinearImpl) +# Build and use just like any other layer in this library +layer =tl.build(MyLinear(out_features=32), torch.randn(1, 64)) +layer(torch.randn(1, 64)) +``` +By default `inputs.shape[1]` will be used as `in_features` value +during initial `forward` pass. If you wish to use different `index` (e.g. to infer using +`inputs.shape[3]`) use `MyLayer = tl.infer(_MyLayerImpl, index=3)` as a decorator. +## Autoencoder with inverted residual bottleneck and pixel shuffle +Please check code comments and [__documentation__](https://szymonmaszke.github.io/torchlayers/) +if needed. If you are unsure what autoencoder is you could see +[__this example blog post__](https://towardsdatascience.com/auto-encoder-what-is-it-and-what-is-it-used-for-part-1-3e5c6f017726). +Below is a convolutional denoising autoencoder example for `ImageNet`-like images. +Think of it like a demonstration of capabilities of different layers +and building blocks provided by `torchlayers`. +```python +# Input - 3 x 256 x 256 for ImageNet reconstruction +class AutoEncoder(torch.nn.Module): + def __init__(self): + super().__init__() + self.encoder = tl.Sequential( + tl.StandardNormalNoise(), # Apply noise to input images + tl.Conv(64, kernel_size=7), + tl.activations.Swish(), # Direct access to module .activations + tl.InvertedResidualBottleneck(squeeze_excitation=False), + tl.AvgPool(), # shape 64 x 128 x 128, kernel_size=2 by default + tl.HardSwish(), # Access simply through tl + tl.SeparableConv(128), # Up number of channels to 128 + tl.InvertedResidualBottleneck(), # Default with squeeze excitation + torch.nn.ReLU(), + tl.AvgPool(), # shape 128 x 64 x 64, kernel_size=2 by default + tl.DepthwiseConv(256), # DepthwiseConv easier to use + # Pass input thrice through the same weights like in PolyNet + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ReLU(), # all torch.nn can be accessed via tl + tl.MaxPool(), # shape 256 x 32 x 32 + tl.Fire(out_channels=512), # shape 512 x 32 x 32 + tl.SqueezeExcitation(hidden=64), + tl.InvertedResidualBottleneck(), + tl.MaxPool(), # shape 512 x 16 x 16 + tl.InvertedResidualBottleneck(squeeze_excitation=False), + # Randomly switch off the last two layers with 0.5 probability + tl.StochasticDepth( + torch.nn.Sequential( + tl.InvertedResidualBottleneck(squeeze_excitation=False), + tl.InvertedResidualBottleneck(squeeze_excitation=False), + ), + p=0.5, + ), + tl.AvgPool(), # shape 512 x 8 x 8 + ) + # This one is more "standard" + self.decoder = tl.Sequential( + tl.Poly(tl.InvertedResidualBottleneck(), order=2), + # Has ICNR initialization by default after calling `build` + tl.ConvPixelShuffle(out_channels=512, upscale_factor=2), + # Shape 512 x 16 x 16 after PixelShuffle + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ConvPixelShuffle(out_channels=256, upscale_factor=2), + # Shape 256 x 32 x 32 + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ConvPixelShuffle(out_channels=128, upscale_factor=2), + # Shape 128 x 64 x 64 + tl.Poly(tl.InvertedResidualBottleneck(), order=4), + tl.ConvPixelShuffle(out_channels=64, upscale_factor=2), + # Shape 64 x 128 x 128 + tl.InvertedResidualBottleneck(), + tl.Conv(256), + tl.Dropout(), # Defaults to 0.5 and Dropout2d for images + tl.Swish(), + tl.InstanceNorm(), + tl.ConvPixelShuffle(out_channels=32, upscale_factor=2), + # Shape 32 x 256 x 256 + tl.Conv(16), + tl.Swish(), + tl.Conv(3), + # Shape 3 x 256 x 256 + ) + def forward(self, inputs): + return self.decoder(self.encoder(inputs)) +``` +Now one can instantiate the module and use it with `torch.nn.MSELoss` as per usual. +```python +autoencoder = tl.build(AutoEncoder(), torch.randn(1, 3, 256, 256)) +``` +# Installation +## [pip](<https://pypi.org/project/torchlayers/>) +### Latest release: +```shell +pip install --user torchlayers +``` +### Nightly: +```shell +pip install --user torchlayers-nightly +``` +## [Docker](https://hub.docker.com/r/szymonmaszke/torchlayers) +__CPU standalone__ and various versions of __GPU enabled__ images are available +at [dockerhub](https://hub.docker.com/r/szymonmaszke/torchlayers/tags). +For CPU quickstart, issue: +```shell +docker pull szymonmaszke/torchlayers:18.04 +``` +Nightly builds are also available, just prefix tag with `nightly_`. If you are going for `GPU` image make sure you have +[nvidia/docker](https://github.com/NVIDIA/nvidia-docker) installed and it's runtime set. +# Contributing +If you find issue or would like to see some functionality (or implement one), please [open new Issue](https://help.github.com/en/articles/creating-an-issue) or [create Pull Request](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork). + +%package -n python3-torchlayers-nightly +Summary: Input shape inference and SOTA custom layers for PyTorch. +Provides: python-torchlayers-nightly +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-torchlayers-nightly +| Version | Docs | Tests | Coverage | Style | PyPI | Python | PyTorch | Docker | +|---------|------|-------|----------|-------|------|--------|---------|--------| +| [](https://github.com/szymonmaszke/torchlayers/releases) | [](https://szymonmaszke.github.io/torchlayers/) |  | [](https://codecov.io/gh/szymonmaszke/torchlayers) | [](https://codebeat.co/projects/github-com-szymonmaszke-torchlayers-master) | [](https://pypi.org/project/torchlayers/) | [](https://www.python.org/) | [](https://pytorch.org/) | [](https://hub.docker.com/r/szymonmaszke/torchlayers) | +[__torchlayers__](https://szymonmaszke.github.io/torchlayers/) is a library based on [__PyTorch__](https://pytorch.org/) +providing __automatic shape and dimensionality inference of `torch.nn` layers__ + additional +building blocks featured in current SOTA architectures (e.g. [Efficient-Net](https://arxiv.org/abs/1905.11946)). +Above requires no user intervention (except single call to `torchlayers.build`) +similarly to the one seen in [__Keras__](https://www.tensorflow.org/guide/keras). +### Main functionalities: +* __Shape inference__ for most of `torch.nn` module (__convolutional, recurrent, transformer, attention and linear layers__) +* __Dimensionality inference__ (e.g. `torchlayers.Conv` working as `torch.nn.Conv1d/2d/3d` based on `input shape`) +* __Shape inference of custom modules__ (see examples section) +* __Additional [Keras-like](https://www.tensorflow.org/guide/keras) layers__ (e.g. `torchlayers.Reshape` or `torchlayers.StandardNormalNoise`) +* __Additional SOTA layers__ mostly from ImageNet competitions +(e.g. [PolyNet](https://arxiv.org/abs/1608.06993), +[Squeeze-And-Excitation](https://arxiv.org/abs/1709.01507), +[StochasticDepth](www.arxiv.org/abs/1512.03385>)) +* __Useful defaults__ (`"same"` padding and default `kernel_size=3` for `Conv`, dropout rates etc.) +* __Zero overhead and [torchscript](https://pytorch.org/docs/stable/jit.html) support__ +__Keep in mind this library works almost exactly like PyTorch originally__. +What that means is you can use `Sequential`, __define your own networks of any complexity using +`torch.nn.Module`__, create new layers with shape inference etc. +_See below to get some intuition about library_. +# Examples +For full functionality please check [__torchlayers documentation__](https://szymonmaszke.github.io/torchlayers/). +Below examples should introduce all necessary concepts you should know. +## Basic classifier +__All__ `torch.nn` modules can be used through `torchlayers` and __each module with input shape__ +will be appropriately modified with it's input inferable counterpart. +```python +import torchlayers as tl +class Classifier(tl.Module): + def __init__(self): + super().__init__() + self.conv1 = tl.Conv2d(64, kernel_size=6) + self.conv2 = tl.Conv2d(128, kernel_size=3) + self.conv3 = tl.Conv2d(256, kernel_size=3, padding=1) + # New layer, more on that in the next example + self.pooling = tl.GlobalMaxPool() + self.dense = tl.Linear(10) + def forward(self, x): + x = torch.relu(self.conv1(x)) + x = torch.relu(self.conv2(x)) + x = torch.relu(self.conv3(x)) + return self.dense(self.pooling(x)) +# Pass model and any example inputs afterwards +clf = tl.build(Classifier(), torch.randn(1, 3, 32, 32)) +``` +Above `torchlayers.Linear(out_features=10)` is used. It is "equivalent" to +original PyTorch's `torch.nn.Linear(in_features=?, out_features=10)` where `in_features` +will be inferred from example input input during `torchlayers.build` call. +Same thing happens with `torch.nn.Conv2d(in_channels, out_channels, kernel_size, ...)` +which can be replaced directly by `tl.Conv2d(out_channels, kernel_size, ...)`. +__Just remember to pass example input through the network!__ +## Simple image and text classifier in one! +* We will use single "model" for both tasks. +Firstly let's define it using `torch.nn` and `torchlayers`: +```python +import torch +import torchlayers as tl +# torch.nn and torchlayers can be mixed easily +model = torch.nn.Sequential( + tl.Conv(64), # specify ONLY out_channels + torch.nn.ReLU(), # use torch.nn wherever you wish + tl.BatchNorm(), # BatchNormNd inferred from input + tl.Conv(128), # Default kernel_size equal to 3 + tl.ReLU(), + tl.Conv(256, kernel_size=11), # "same" padding as default + tl.GlobalMaxPool(), # Known from Keras + tl.Linear(10), # Output for 10 classes +) +print(model) +``` +Above would give you model's summary like this (__notice question marks for not yet inferred values__): +```python +Sequential( + (0): Conv(in_channels=?, out_channels=64, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (1): ReLU() + (2): BatchNorm(num_features=?, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (3): Conv(in_channels=?, out_channels=128, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (4): ReLU() + (5): Conv(in_channels=?, out_channels=256, kernel_size=11, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (6): GlobalMaxPool() + (7): Linear(in_features=?, out_features=10, bias=True) +) +``` +* Now you can __build__/instantiate your model with example input (in this case MNIST-like): +```python +mnist_model = tl.build(model, torch.randn(1, 3, 28, 28)) +``` +* Or if it's text classification you are after, same model could be built with different +`input shape` (e.g. for text classification using `300` dimensional pretrained embedding): +```python +# [batch, embedding, timesteps], first dimension > 1 for BatchNorm1d to work +text_model = tl.build(model, torch.randn(2, 300, 1)) +``` +* Finally, you can `print` both models after instantiation, provided below side +by-side for readability (__notice different dimenstionality, e.g. `Conv2d` vs `Conv1d` after `torchlayers.build`__): +```python + # TEXT CLASSIFIER MNIST CLASSIFIER + Sequential( Sequential( + (0): Conv1d(300, 64) (0): Conv2d(3, 64) + (1): ReLU() (1): ReLU() + (2): BatchNorm1d(64) (2): BatchNorm2d(64) + (3): Conv1d(64, 128) (3): Conv2d(64, 128) + (4): ReLU() (4): ReLU() + (5): Conv1d(128, 256) (5): Conv2d(128, 256) + (6): GlobalMaxPool() (6): GlobalMaxPool() + (7): Linear(256, 10) (7): Linear(256, 10) + ) ) +``` +As you can see both modules "compiled" into original `pytorch` layers. +## Custom modules with shape inference capabilities +User can define any module and make it shape inferable with `torchlayers.infer` +function: +```python + # Class defined with in_features + # It might be a good practice to use _ prefix and Impl as postfix + # to differentiate from shape inferable version +class _MyLinearImpl(torch.nn.Module): + def __init__(self, in_features: int, out_features: int): + super().__init__() + self.weight = torch.nn.Parameter(torch.randn(out_features, in_features)) + self.bias = torch.nn.Parameter(torch.randn(out_features)) + def forward(self, inputs): + return torch.nn.functional.linear(inputs, self.weight, self.bias) +MyLinear = tl.infer(_MyLinearImpl) +# Build and use just like any other layer in this library +layer =tl.build(MyLinear(out_features=32), torch.randn(1, 64)) +layer(torch.randn(1, 64)) +``` +By default `inputs.shape[1]` will be used as `in_features` value +during initial `forward` pass. If you wish to use different `index` (e.g. to infer using +`inputs.shape[3]`) use `MyLayer = tl.infer(_MyLayerImpl, index=3)` as a decorator. +## Autoencoder with inverted residual bottleneck and pixel shuffle +Please check code comments and [__documentation__](https://szymonmaszke.github.io/torchlayers/) +if needed. If you are unsure what autoencoder is you could see +[__this example blog post__](https://towardsdatascience.com/auto-encoder-what-is-it-and-what-is-it-used-for-part-1-3e5c6f017726). +Below is a convolutional denoising autoencoder example for `ImageNet`-like images. +Think of it like a demonstration of capabilities of different layers +and building blocks provided by `torchlayers`. +```python +# Input - 3 x 256 x 256 for ImageNet reconstruction +class AutoEncoder(torch.nn.Module): + def __init__(self): + super().__init__() + self.encoder = tl.Sequential( + tl.StandardNormalNoise(), # Apply noise to input images + tl.Conv(64, kernel_size=7), + tl.activations.Swish(), # Direct access to module .activations + tl.InvertedResidualBottleneck(squeeze_excitation=False), + tl.AvgPool(), # shape 64 x 128 x 128, kernel_size=2 by default + tl.HardSwish(), # Access simply through tl + tl.SeparableConv(128), # Up number of channels to 128 + tl.InvertedResidualBottleneck(), # Default with squeeze excitation + torch.nn.ReLU(), + tl.AvgPool(), # shape 128 x 64 x 64, kernel_size=2 by default + tl.DepthwiseConv(256), # DepthwiseConv easier to use + # Pass input thrice through the same weights like in PolyNet + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ReLU(), # all torch.nn can be accessed via tl + tl.MaxPool(), # shape 256 x 32 x 32 + tl.Fire(out_channels=512), # shape 512 x 32 x 32 + tl.SqueezeExcitation(hidden=64), + tl.InvertedResidualBottleneck(), + tl.MaxPool(), # shape 512 x 16 x 16 + tl.InvertedResidualBottleneck(squeeze_excitation=False), + # Randomly switch off the last two layers with 0.5 probability + tl.StochasticDepth( + torch.nn.Sequential( + tl.InvertedResidualBottleneck(squeeze_excitation=False), + tl.InvertedResidualBottleneck(squeeze_excitation=False), + ), + p=0.5, + ), + tl.AvgPool(), # shape 512 x 8 x 8 + ) + # This one is more "standard" + self.decoder = tl.Sequential( + tl.Poly(tl.InvertedResidualBottleneck(), order=2), + # Has ICNR initialization by default after calling `build` + tl.ConvPixelShuffle(out_channels=512, upscale_factor=2), + # Shape 512 x 16 x 16 after PixelShuffle + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ConvPixelShuffle(out_channels=256, upscale_factor=2), + # Shape 256 x 32 x 32 + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ConvPixelShuffle(out_channels=128, upscale_factor=2), + # Shape 128 x 64 x 64 + tl.Poly(tl.InvertedResidualBottleneck(), order=4), + tl.ConvPixelShuffle(out_channels=64, upscale_factor=2), + # Shape 64 x 128 x 128 + tl.InvertedResidualBottleneck(), + tl.Conv(256), + tl.Dropout(), # Defaults to 0.5 and Dropout2d for images + tl.Swish(), + tl.InstanceNorm(), + tl.ConvPixelShuffle(out_channels=32, upscale_factor=2), + # Shape 32 x 256 x 256 + tl.Conv(16), + tl.Swish(), + tl.Conv(3), + # Shape 3 x 256 x 256 + ) + def forward(self, inputs): + return self.decoder(self.encoder(inputs)) +``` +Now one can instantiate the module and use it with `torch.nn.MSELoss` as per usual. +```python +autoencoder = tl.build(AutoEncoder(), torch.randn(1, 3, 256, 256)) +``` +# Installation +## [pip](<https://pypi.org/project/torchlayers/>) +### Latest release: +```shell +pip install --user torchlayers +``` +### Nightly: +```shell +pip install --user torchlayers-nightly +``` +## [Docker](https://hub.docker.com/r/szymonmaszke/torchlayers) +__CPU standalone__ and various versions of __GPU enabled__ images are available +at [dockerhub](https://hub.docker.com/r/szymonmaszke/torchlayers/tags). +For CPU quickstart, issue: +```shell +docker pull szymonmaszke/torchlayers:18.04 +``` +Nightly builds are also available, just prefix tag with `nightly_`. If you are going for `GPU` image make sure you have +[nvidia/docker](https://github.com/NVIDIA/nvidia-docker) installed and it's runtime set. +# Contributing +If you find issue or would like to see some functionality (or implement one), please [open new Issue](https://help.github.com/en/articles/creating-an-issue) or [create Pull Request](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork). + +%package help +Summary: Development documents and examples for torchlayers-nightly +Provides: python3-torchlayers-nightly-doc +%description help +| Version | Docs | Tests | Coverage | Style | PyPI | Python | PyTorch | Docker | +|---------|------|-------|----------|-------|------|--------|---------|--------| +| [](https://github.com/szymonmaszke/torchlayers/releases) | [](https://szymonmaszke.github.io/torchlayers/) |  | [](https://codecov.io/gh/szymonmaszke/torchlayers) | [](https://codebeat.co/projects/github-com-szymonmaszke-torchlayers-master) | [](https://pypi.org/project/torchlayers/) | [](https://www.python.org/) | [](https://pytorch.org/) | [](https://hub.docker.com/r/szymonmaszke/torchlayers) | +[__torchlayers__](https://szymonmaszke.github.io/torchlayers/) is a library based on [__PyTorch__](https://pytorch.org/) +providing __automatic shape and dimensionality inference of `torch.nn` layers__ + additional +building blocks featured in current SOTA architectures (e.g. [Efficient-Net](https://arxiv.org/abs/1905.11946)). +Above requires no user intervention (except single call to `torchlayers.build`) +similarly to the one seen in [__Keras__](https://www.tensorflow.org/guide/keras). +### Main functionalities: +* __Shape inference__ for most of `torch.nn` module (__convolutional, recurrent, transformer, attention and linear layers__) +* __Dimensionality inference__ (e.g. `torchlayers.Conv` working as `torch.nn.Conv1d/2d/3d` based on `input shape`) +* __Shape inference of custom modules__ (see examples section) +* __Additional [Keras-like](https://www.tensorflow.org/guide/keras) layers__ (e.g. `torchlayers.Reshape` or `torchlayers.StandardNormalNoise`) +* __Additional SOTA layers__ mostly from ImageNet competitions +(e.g. [PolyNet](https://arxiv.org/abs/1608.06993), +[Squeeze-And-Excitation](https://arxiv.org/abs/1709.01507), +[StochasticDepth](www.arxiv.org/abs/1512.03385>)) +* __Useful defaults__ (`"same"` padding and default `kernel_size=3` for `Conv`, dropout rates etc.) +* __Zero overhead and [torchscript](https://pytorch.org/docs/stable/jit.html) support__ +__Keep in mind this library works almost exactly like PyTorch originally__. +What that means is you can use `Sequential`, __define your own networks of any complexity using +`torch.nn.Module`__, create new layers with shape inference etc. +_See below to get some intuition about library_. +# Examples +For full functionality please check [__torchlayers documentation__](https://szymonmaszke.github.io/torchlayers/). +Below examples should introduce all necessary concepts you should know. +## Basic classifier +__All__ `torch.nn` modules can be used through `torchlayers` and __each module with input shape__ +will be appropriately modified with it's input inferable counterpart. +```python +import torchlayers as tl +class Classifier(tl.Module): + def __init__(self): + super().__init__() + self.conv1 = tl.Conv2d(64, kernel_size=6) + self.conv2 = tl.Conv2d(128, kernel_size=3) + self.conv3 = tl.Conv2d(256, kernel_size=3, padding=1) + # New layer, more on that in the next example + self.pooling = tl.GlobalMaxPool() + self.dense = tl.Linear(10) + def forward(self, x): + x = torch.relu(self.conv1(x)) + x = torch.relu(self.conv2(x)) + x = torch.relu(self.conv3(x)) + return self.dense(self.pooling(x)) +# Pass model and any example inputs afterwards +clf = tl.build(Classifier(), torch.randn(1, 3, 32, 32)) +``` +Above `torchlayers.Linear(out_features=10)` is used. It is "equivalent" to +original PyTorch's `torch.nn.Linear(in_features=?, out_features=10)` where `in_features` +will be inferred from example input input during `torchlayers.build` call. +Same thing happens with `torch.nn.Conv2d(in_channels, out_channels, kernel_size, ...)` +which can be replaced directly by `tl.Conv2d(out_channels, kernel_size, ...)`. +__Just remember to pass example input through the network!__ +## Simple image and text classifier in one! +* We will use single "model" for both tasks. +Firstly let's define it using `torch.nn` and `torchlayers`: +```python +import torch +import torchlayers as tl +# torch.nn and torchlayers can be mixed easily +model = torch.nn.Sequential( + tl.Conv(64), # specify ONLY out_channels + torch.nn.ReLU(), # use torch.nn wherever you wish + tl.BatchNorm(), # BatchNormNd inferred from input + tl.Conv(128), # Default kernel_size equal to 3 + tl.ReLU(), + tl.Conv(256, kernel_size=11), # "same" padding as default + tl.GlobalMaxPool(), # Known from Keras + tl.Linear(10), # Output for 10 classes +) +print(model) +``` +Above would give you model's summary like this (__notice question marks for not yet inferred values__): +```python +Sequential( + (0): Conv(in_channels=?, out_channels=64, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (1): ReLU() + (2): BatchNorm(num_features=?, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (3): Conv(in_channels=?, out_channels=128, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (4): ReLU() + (5): Conv(in_channels=?, out_channels=256, kernel_size=11, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros) + (6): GlobalMaxPool() + (7): Linear(in_features=?, out_features=10, bias=True) +) +``` +* Now you can __build__/instantiate your model with example input (in this case MNIST-like): +```python +mnist_model = tl.build(model, torch.randn(1, 3, 28, 28)) +``` +* Or if it's text classification you are after, same model could be built with different +`input shape` (e.g. for text classification using `300` dimensional pretrained embedding): +```python +# [batch, embedding, timesteps], first dimension > 1 for BatchNorm1d to work +text_model = tl.build(model, torch.randn(2, 300, 1)) +``` +* Finally, you can `print` both models after instantiation, provided below side +by-side for readability (__notice different dimenstionality, e.g. `Conv2d` vs `Conv1d` after `torchlayers.build`__): +```python + # TEXT CLASSIFIER MNIST CLASSIFIER + Sequential( Sequential( + (0): Conv1d(300, 64) (0): Conv2d(3, 64) + (1): ReLU() (1): ReLU() + (2): BatchNorm1d(64) (2): BatchNorm2d(64) + (3): Conv1d(64, 128) (3): Conv2d(64, 128) + (4): ReLU() (4): ReLU() + (5): Conv1d(128, 256) (5): Conv2d(128, 256) + (6): GlobalMaxPool() (6): GlobalMaxPool() + (7): Linear(256, 10) (7): Linear(256, 10) + ) ) +``` +As you can see both modules "compiled" into original `pytorch` layers. +## Custom modules with shape inference capabilities +User can define any module and make it shape inferable with `torchlayers.infer` +function: +```python + # Class defined with in_features + # It might be a good practice to use _ prefix and Impl as postfix + # to differentiate from shape inferable version +class _MyLinearImpl(torch.nn.Module): + def __init__(self, in_features: int, out_features: int): + super().__init__() + self.weight = torch.nn.Parameter(torch.randn(out_features, in_features)) + self.bias = torch.nn.Parameter(torch.randn(out_features)) + def forward(self, inputs): + return torch.nn.functional.linear(inputs, self.weight, self.bias) +MyLinear = tl.infer(_MyLinearImpl) +# Build and use just like any other layer in this library +layer =tl.build(MyLinear(out_features=32), torch.randn(1, 64)) +layer(torch.randn(1, 64)) +``` +By default `inputs.shape[1]` will be used as `in_features` value +during initial `forward` pass. If you wish to use different `index` (e.g. to infer using +`inputs.shape[3]`) use `MyLayer = tl.infer(_MyLayerImpl, index=3)` as a decorator. +## Autoencoder with inverted residual bottleneck and pixel shuffle +Please check code comments and [__documentation__](https://szymonmaszke.github.io/torchlayers/) +if needed. If you are unsure what autoencoder is you could see +[__this example blog post__](https://towardsdatascience.com/auto-encoder-what-is-it-and-what-is-it-used-for-part-1-3e5c6f017726). +Below is a convolutional denoising autoencoder example for `ImageNet`-like images. +Think of it like a demonstration of capabilities of different layers +and building blocks provided by `torchlayers`. +```python +# Input - 3 x 256 x 256 for ImageNet reconstruction +class AutoEncoder(torch.nn.Module): + def __init__(self): + super().__init__() + self.encoder = tl.Sequential( + tl.StandardNormalNoise(), # Apply noise to input images + tl.Conv(64, kernel_size=7), + tl.activations.Swish(), # Direct access to module .activations + tl.InvertedResidualBottleneck(squeeze_excitation=False), + tl.AvgPool(), # shape 64 x 128 x 128, kernel_size=2 by default + tl.HardSwish(), # Access simply through tl + tl.SeparableConv(128), # Up number of channels to 128 + tl.InvertedResidualBottleneck(), # Default with squeeze excitation + torch.nn.ReLU(), + tl.AvgPool(), # shape 128 x 64 x 64, kernel_size=2 by default + tl.DepthwiseConv(256), # DepthwiseConv easier to use + # Pass input thrice through the same weights like in PolyNet + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ReLU(), # all torch.nn can be accessed via tl + tl.MaxPool(), # shape 256 x 32 x 32 + tl.Fire(out_channels=512), # shape 512 x 32 x 32 + tl.SqueezeExcitation(hidden=64), + tl.InvertedResidualBottleneck(), + tl.MaxPool(), # shape 512 x 16 x 16 + tl.InvertedResidualBottleneck(squeeze_excitation=False), + # Randomly switch off the last two layers with 0.5 probability + tl.StochasticDepth( + torch.nn.Sequential( + tl.InvertedResidualBottleneck(squeeze_excitation=False), + tl.InvertedResidualBottleneck(squeeze_excitation=False), + ), + p=0.5, + ), + tl.AvgPool(), # shape 512 x 8 x 8 + ) + # This one is more "standard" + self.decoder = tl.Sequential( + tl.Poly(tl.InvertedResidualBottleneck(), order=2), + # Has ICNR initialization by default after calling `build` + tl.ConvPixelShuffle(out_channels=512, upscale_factor=2), + # Shape 512 x 16 x 16 after PixelShuffle + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ConvPixelShuffle(out_channels=256, upscale_factor=2), + # Shape 256 x 32 x 32 + tl.Poly(tl.InvertedResidualBottleneck(), order=3), + tl.ConvPixelShuffle(out_channels=128, upscale_factor=2), + # Shape 128 x 64 x 64 + tl.Poly(tl.InvertedResidualBottleneck(), order=4), + tl.ConvPixelShuffle(out_channels=64, upscale_factor=2), + # Shape 64 x 128 x 128 + tl.InvertedResidualBottleneck(), + tl.Conv(256), + tl.Dropout(), # Defaults to 0.5 and Dropout2d for images + tl.Swish(), + tl.InstanceNorm(), + tl.ConvPixelShuffle(out_channels=32, upscale_factor=2), + # Shape 32 x 256 x 256 + tl.Conv(16), + tl.Swish(), + tl.Conv(3), + # Shape 3 x 256 x 256 + ) + def forward(self, inputs): + return self.decoder(self.encoder(inputs)) +``` +Now one can instantiate the module and use it with `torch.nn.MSELoss` as per usual. +```python +autoencoder = tl.build(AutoEncoder(), torch.randn(1, 3, 256, 256)) +``` +# Installation +## [pip](<https://pypi.org/project/torchlayers/>) +### Latest release: +```shell +pip install --user torchlayers +``` +### Nightly: +```shell +pip install --user torchlayers-nightly +``` +## [Docker](https://hub.docker.com/r/szymonmaszke/torchlayers) +__CPU standalone__ and various versions of __GPU enabled__ images are available +at [dockerhub](https://hub.docker.com/r/szymonmaszke/torchlayers/tags). +For CPU quickstart, issue: +```shell +docker pull szymonmaszke/torchlayers:18.04 +``` +Nightly builds are also available, just prefix tag with `nightly_`. If you are going for `GPU` image make sure you have +[nvidia/docker](https://github.com/NVIDIA/nvidia-docker) installed and it's runtime set. +# Contributing +If you find issue or would like to see some functionality (or implement one), please [open new Issue](https://help.github.com/en/articles/creating-an-issue) or [create Pull Request](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork). + +%prep +%autosetup -n torchlayers-nightly-1681171628 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-torchlayers-nightly -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Tue Apr 11 2023 Python_Bot <Python_Bot@openeuler.org> - 1681171628-1 +- Package Spec generated |