%global _empty_manifest_terminate_build 0 Name: python-adamp Version: 0.3.0 Release: 1 Summary: AdamP optimizer: Slowing Down the Weight Norm Increase in Momentum-based Optimizers License: MIT License URL: https://github.com/clovaai/AdamP Source0: https://mirrors.aliyun.com/pypi/web/packages/c8/56/182b8c93f18feb0244b83f9b2eff1c6b036c04d4c3880e8d222750b0d5e5/adamp-0.3.0.tar.gz BuildArch: noarch %description ## Slowing Down the Weight Norm Increase in Momentum-based Optimizers Official PyTorch implementation of AdamP and SGDP optimizers | [Paper](https://arxiv.org/abs/2006.08217) | [Project page](https://clovaai.github.io/AdamP/) **Byeongho Heo\*, Sanghyuk Chun\*, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Youngjung Uh, Jung-Woo Ha.**
\* indicates equal contribution Clova AI Research, NAVER Corp. ### Abstract Normalization techniques, such as batch normalization (BN), have led to significant improvements in deep neural network performances. Prior studies have analyzed the benefits of the resulting scale invariance of the weights for the gradient descent (GD) optimizers: it leads to a stabilized training due to the auto-tuning of step sizes. However, we show that, combined with the momentum-based algorithms, the scale invariance tends to induce an excessive growth of the weight norms. This in turn overly suppresses the effective step sizes during training, potentially leading to sub-optimal performances in deep neural networks. We analyze this phenomenon both theoretically and empirically. We propose a simple and effective solution: at each iteration of momentum-based GD optimizers (e.g., SGD or Adam) applied on scale-invariant weights (e.g., Conv weights preceding a BN layer), we remove the radial component (i.e., parallel to the weight vector) from the update vector. Intuitively, this operation prevents the unnecessary update along the radial direction that only increases the weight norm without contributing to the loss minimization. We verify that the modified optimizers SGDP and AdamP successfully regularize the norm growth and improve the performance of a broad set of models. Our experiments cover tasks including image classification and retrieval, object detection, robustness benchmarks, and audio classification. ## How does it work? Please visit our [project page](https://clovaai.github.io/AdamP/). ## Updates - **Jun 19, 2020**: built-in cosine similarity and fix warning (v0.3.0) - **Jun 19, 2020**: nesterov update (v0.2.0) - **Jun 15, 2020**: Initial upload (v0.1.0) ## Getting Started ### Installation ``` pip3 install adamp ``` ### Usage Usage is exactly same as [torch.optim](https://pytorch.org/docs/stable/optim.html) library! ```python from adamp import AdamP # define your params optimizer = AdamP(params, lr=0.001, betas=(0.9, 0.999), weight_decay=1e-2) ``` ```python from adamp import SGDP # define your params optimizer = SGDP(params, lr=0.1, weight_decay=1e-5, momentum=0.9, nesterov=True) ``` ## Arguments `SGDP` and `AdamP` share arguments with [torch.optim.SGD](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD) and [torch.optim.Adam](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam). There are two additional hyperparameters; we recommend using the default values. - `delta` : threhold that determines whether a set of parameters is scale invariant or not (default: 0.1) - `wd_ratio` : relative weight decay applied on _scale-invariant_ parameters compared to that applied on _scale-variant_ parameters (default: 0.1) Both `SGDP` and `AdamP` support Nesterov momentum. - `nesterov` : enables Nesterov momentum (default: False) ## License This project is distributed under MIT license. ``` Copyright (c) 2020-present NAVER Corp. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ## How to cite ``` @article{heo2020adamp, title={Slowing Down the Weight Norm Increase in Momentum-based Optimizers}, author={Heo, Byeongho and Chun, Sanghyuk and Oh, Seong Joon and Han, Dongyoon and Yun, Sangdoo and Uh, Youngjung and Ha, Jung-Woo}, year={2020}, journal={arXiv preprint arXiv:2006.08217}, } ``` %package -n python3-adamp Summary: AdamP optimizer: Slowing Down the Weight Norm Increase in Momentum-based Optimizers Provides: python-adamp BuildRequires: python3-devel BuildRequires: python3-setuptools BuildRequires: python3-pip %description -n python3-adamp ## Slowing Down the Weight Norm Increase in Momentum-based Optimizers Official PyTorch implementation of AdamP and SGDP optimizers | [Paper](https://arxiv.org/abs/2006.08217) | [Project page](https://clovaai.github.io/AdamP/) **Byeongho Heo\*, Sanghyuk Chun\*, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Youngjung Uh, Jung-Woo Ha.**
\* indicates equal contribution Clova AI Research, NAVER Corp. ### Abstract Normalization techniques, such as batch normalization (BN), have led to significant improvements in deep neural network performances. Prior studies have analyzed the benefits of the resulting scale invariance of the weights for the gradient descent (GD) optimizers: it leads to a stabilized training due to the auto-tuning of step sizes. However, we show that, combined with the momentum-based algorithms, the scale invariance tends to induce an excessive growth of the weight norms. This in turn overly suppresses the effective step sizes during training, potentially leading to sub-optimal performances in deep neural networks. We analyze this phenomenon both theoretically and empirically. We propose a simple and effective solution: at each iteration of momentum-based GD optimizers (e.g., SGD or Adam) applied on scale-invariant weights (e.g., Conv weights preceding a BN layer), we remove the radial component (i.e., parallel to the weight vector) from the update vector. Intuitively, this operation prevents the unnecessary update along the radial direction that only increases the weight norm without contributing to the loss minimization. We verify that the modified optimizers SGDP and AdamP successfully regularize the norm growth and improve the performance of a broad set of models. Our experiments cover tasks including image classification and retrieval, object detection, robustness benchmarks, and audio classification. ## How does it work? Please visit our [project page](https://clovaai.github.io/AdamP/). ## Updates - **Jun 19, 2020**: built-in cosine similarity and fix warning (v0.3.0) - **Jun 19, 2020**: nesterov update (v0.2.0) - **Jun 15, 2020**: Initial upload (v0.1.0) ## Getting Started ### Installation ``` pip3 install adamp ``` ### Usage Usage is exactly same as [torch.optim](https://pytorch.org/docs/stable/optim.html) library! ```python from adamp import AdamP # define your params optimizer = AdamP(params, lr=0.001, betas=(0.9, 0.999), weight_decay=1e-2) ``` ```python from adamp import SGDP # define your params optimizer = SGDP(params, lr=0.1, weight_decay=1e-5, momentum=0.9, nesterov=True) ``` ## Arguments `SGDP` and `AdamP` share arguments with [torch.optim.SGD](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD) and [torch.optim.Adam](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam). There are two additional hyperparameters; we recommend using the default values. - `delta` : threhold that determines whether a set of parameters is scale invariant or not (default: 0.1) - `wd_ratio` : relative weight decay applied on _scale-invariant_ parameters compared to that applied on _scale-variant_ parameters (default: 0.1) Both `SGDP` and `AdamP` support Nesterov momentum. - `nesterov` : enables Nesterov momentum (default: False) ## License This project is distributed under MIT license. ``` Copyright (c) 2020-present NAVER Corp. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ## How to cite ``` @article{heo2020adamp, title={Slowing Down the Weight Norm Increase in Momentum-based Optimizers}, author={Heo, Byeongho and Chun, Sanghyuk and Oh, Seong Joon and Han, Dongyoon and Yun, Sangdoo and Uh, Youngjung and Ha, Jung-Woo}, year={2020}, journal={arXiv preprint arXiv:2006.08217}, } ``` %package help Summary: Development documents and examples for adamp Provides: python3-adamp-doc %description help ## Slowing Down the Weight Norm Increase in Momentum-based Optimizers Official PyTorch implementation of AdamP and SGDP optimizers | [Paper](https://arxiv.org/abs/2006.08217) | [Project page](https://clovaai.github.io/AdamP/) **Byeongho Heo\*, Sanghyuk Chun\*, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Youngjung Uh, Jung-Woo Ha.**
\* indicates equal contribution Clova AI Research, NAVER Corp. ### Abstract Normalization techniques, such as batch normalization (BN), have led to significant improvements in deep neural network performances. Prior studies have analyzed the benefits of the resulting scale invariance of the weights for the gradient descent (GD) optimizers: it leads to a stabilized training due to the auto-tuning of step sizes. However, we show that, combined with the momentum-based algorithms, the scale invariance tends to induce an excessive growth of the weight norms. This in turn overly suppresses the effective step sizes during training, potentially leading to sub-optimal performances in deep neural networks. We analyze this phenomenon both theoretically and empirically. We propose a simple and effective solution: at each iteration of momentum-based GD optimizers (e.g., SGD or Adam) applied on scale-invariant weights (e.g., Conv weights preceding a BN layer), we remove the radial component (i.e., parallel to the weight vector) from the update vector. Intuitively, this operation prevents the unnecessary update along the radial direction that only increases the weight norm without contributing to the loss minimization. We verify that the modified optimizers SGDP and AdamP successfully regularize the norm growth and improve the performance of a broad set of models. Our experiments cover tasks including image classification and retrieval, object detection, robustness benchmarks, and audio classification. ## How does it work? Please visit our [project page](https://clovaai.github.io/AdamP/). ## Updates - **Jun 19, 2020**: built-in cosine similarity and fix warning (v0.3.0) - **Jun 19, 2020**: nesterov update (v0.2.0) - **Jun 15, 2020**: Initial upload (v0.1.0) ## Getting Started ### Installation ``` pip3 install adamp ``` ### Usage Usage is exactly same as [torch.optim](https://pytorch.org/docs/stable/optim.html) library! ```python from adamp import AdamP # define your params optimizer = AdamP(params, lr=0.001, betas=(0.9, 0.999), weight_decay=1e-2) ``` ```python from adamp import SGDP # define your params optimizer = SGDP(params, lr=0.1, weight_decay=1e-5, momentum=0.9, nesterov=True) ``` ## Arguments `SGDP` and `AdamP` share arguments with [torch.optim.SGD](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD) and [torch.optim.Adam](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam). There are two additional hyperparameters; we recommend using the default values. - `delta` : threhold that determines whether a set of parameters is scale invariant or not (default: 0.1) - `wd_ratio` : relative weight decay applied on _scale-invariant_ parameters compared to that applied on _scale-variant_ parameters (default: 0.1) Both `SGDP` and `AdamP` support Nesterov momentum. - `nesterov` : enables Nesterov momentum (default: False) ## License This project is distributed under MIT license. ``` Copyright (c) 2020-present NAVER Corp. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ## How to cite ``` @article{heo2020adamp, title={Slowing Down the Weight Norm Increase in Momentum-based Optimizers}, author={Heo, Byeongho and Chun, Sanghyuk and Oh, Seong Joon and Han, Dongyoon and Yun, Sangdoo and Uh, Youngjung and Ha, Jung-Woo}, year={2020}, journal={arXiv preprint arXiv:2006.08217}, } ``` %prep %autosetup -n adamp-0.3.0 %build %py3_build %install %py3_install install -d -m755 %{buildroot}/%{_pkgdocdir} if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi pushd %{buildroot} if [ -d usr/lib ]; then find usr/lib -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi if [ -d usr/lib64 ]; then find usr/lib64 -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi if [ -d usr/bin ]; then find usr/bin -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi if [ -d usr/sbin ]; then find usr/sbin -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi touch doclist.lst if [ -d usr/share/man ]; then find usr/share/man -type f -printf "\"/%h/%f.gz\"\n" >> doclist.lst fi popd mv %{buildroot}/filelist.lst . mv %{buildroot}/doclist.lst . %files -n python3-adamp -f filelist.lst %dir %{python3_sitelib}/* %files help -f doclist.lst %{_docdir}/* %changelog * Thu Jun 08 2023 Python_Bot - 0.3.0-1 - Package Spec generated