diff options
| author | CoprDistGit <infra@openeuler.org> | 2023-04-11 08:00:31 +0000 |
|---|---|---|
| committer | CoprDistGit <infra@openeuler.org> | 2023-04-11 08:00:31 +0000 |
| commit | 643bfbdf764b512506dac1066eb6c9581fde2266 (patch) | |
| tree | 6ebdc94a27e0f2ef45f39fc0f8063cb77fc31aaa /python-loky.spec | |
| parent | feb7af46600a84655103003d21d2b3bd0e20e6a2 (diff) | |
automatic import of python-loky
Diffstat (limited to 'python-loky.spec')
| -rw-r--r-- | python-loky.spec | 514 |
1 files changed, 514 insertions, 0 deletions
diff --git a/python-loky.spec b/python-loky.spec new file mode 100644 index 0000000..4a36821 --- /dev/null +++ b/python-loky.spec @@ -0,0 +1,514 @@ +%global _empty_manifest_terminate_build 0 +Name: python-loky +Version: 3.3.0 +Release: 1 +Summary: A robust implementation of concurrent.futures.ProcessPoolExecutor +License: BSD +URL: https://github.com/joblib/loky/ +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/35/0e/97520e1a9317637727fd05d00ae4d834eaf82b19ca7ef3e2dcedf4446656/loky-3.3.0.tar.gz +BuildArch: noarch + +Requires: python3-cloudpickle + +%description + +<a href="https://loky.readthedocs.io"> +<img src="docs/_static/loky_logo.svg" +alt="Loky logo" width=96/></a> + + +# Reusable Process Pool Executor +[](https://dev.azure.com/joblib/loky/_build/latest?definitionId=2&branchName=master) +[](https://loky.readthedocs.io/en/latest/?badge=latest) +[](https://codecov.io/gh/joblib/loky) +[](https://zenodo.org/badge/latestdoi/48578152) + + +### Goal + +The aim of this project is to provide a robust, cross-platform and +cross-version implementation of the `ProcessPoolExecutor` class of +`concurrent.futures`. It notably features: + + * __Consistent and robust spawn behavior__: All processes are started + using fork + exec on POSIX systems. This ensures safer interactions with + third party libraries. On the contrary, `multiprocessing.Pool` uses + fork without exec by default, causing third party runtimes to crash + (e.g. OpenMP, macOS Accelerate...). + + * __Reusable executor__: strategy to avoid re-spawning a complete + executor every time. A singleton executor instance can be reused (and + dynamically resized if necessary) across consecutive calls to limit + spawning and shutdown overhead. The worker processes can be shutdown + automatically after a configurable idling timeout to free system + resources. + + * __Transparent cloudpickle integration__: to call interactively + defined functions and lambda expressions in parallel. It is also + possible to register a custom pickler implementation to handle + inter-process communications. + + * __No need for ``if __name__ == "__main__":`` in scripts__: thanks + to the use of ``cloudpickle`` to call functions defined in the + ``__main__`` module, it is not required to protect the code calling + parallel functions under Windows. + + * __Deadlock free implementation__: one of the major concern in + standard `multiprocessing` and `concurrent.futures` modules is the + ability of the `Pool/Executor` to handle crashes of worker + processes. This library intends to fix those possible deadlocks and + send back meaningful errors. Note that the implementation of + `concurrent.futures.ProcessPoolExecutor` that comes with Python 3.7+ + is as robust as the executor from loky but the latter also works for + older versions of Python. + + +### Installation + +The recommended way to install `loky` is with `pip`, +```bash +pip install loky +``` + +`loky` can also be installed from sources using +```bash +git clone https://github.com/joblib/loky +cd loky +python setup.py install +``` + +Note that `loky` has an optional dependency on [`psutil`][1] to allow early +memory leak detections. + +### Usage + +The basic usage of `loky` relies on the `get_reusable_executor`, which +internally manages a custom `ProcessPoolExecutor` object, which is reused or +re-spawned depending on the context. + +```python +import os +from time import sleep +from loky import get_reusable_executor + + +def say_hello(k): + pid = os.getpid() + print(f"Hello from {pid} with arg {k}") + sleep(.01) + return pid + + +# Create an executor with 4 worker processes, that will +# automatically shutdown after idling for 2s +executor = get_reusable_executor(max_workers=4, timeout=2) + +res = executor.submit(say_hello, 1) +print("Got results:", res.result()) + +results = executor.map(say_hello, range(50)) +n_workers = len(set(results)) +print("Number of used processes:", n_workers) +assert n_workers == 4 +``` + +For more advance usage, see our +[documentation](https://loky.readthedocs.io/en/stable/) + +### Workflow to contribute + +To contribute to **loky**, first create an account on +[github](http://github.com/). Once this is done, fork the +[loky repository](http://github.com/loky/loky) to have your own repository, +clone it using 'git clone' on the computers where you want to work. Make your +changes in your clone, push them to your github account, test them on several +computers, and when you are happy with them, send a pull request to the main +repository. + +### Running the test suite + +To run the test suite, you need the `pytest` (version >= 3) and `psutil` +modules. From the root of the project, run the test suite using: + +```sh + pip install -e . + pytest . +``` + +### Why was the project named `loky`? + +While developping `loky`, we had some bad experiences trying to debug deadlocks +when using `multiprocessing.Pool` and `concurrent.futures.ProcessPoolExecutor`, +especially when calling functions with non-picklable arguments or returned +values at the beginning of the project. When we had to chose a name, we had +dealt with so many deadlocks that we wanted some kind of invocation to repel +them! Hence `loky`: a mix of a god, locks and the `y` that make it somehow +cooler and nicer : (and also less likely to result in name conflict in google +results ^^). + +Fixes to avoid those deadlocks in `concurrent.futures` were also contributed +upstream in Python 3.7+, as a less mystical way to repel the deadlocks :D + +### Acknowledgement + +This work is supported by the Center for Data Science, funded by the IDEX +Paris-Saclay, ANR-11-IDEX-0003-02 + + +[1]: https://github.com/giampaolo/psutil + + + + +%package -n python3-loky +Summary: A robust implementation of concurrent.futures.ProcessPoolExecutor +Provides: python-loky +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-loky + +<a href="https://loky.readthedocs.io"> +<img src="docs/_static/loky_logo.svg" +alt="Loky logo" width=96/></a> + + +# Reusable Process Pool Executor +[](https://dev.azure.com/joblib/loky/_build/latest?definitionId=2&branchName=master) +[](https://loky.readthedocs.io/en/latest/?badge=latest) +[](https://codecov.io/gh/joblib/loky) +[](https://zenodo.org/badge/latestdoi/48578152) + + +### Goal + +The aim of this project is to provide a robust, cross-platform and +cross-version implementation of the `ProcessPoolExecutor` class of +`concurrent.futures`. It notably features: + + * __Consistent and robust spawn behavior__: All processes are started + using fork + exec on POSIX systems. This ensures safer interactions with + third party libraries. On the contrary, `multiprocessing.Pool` uses + fork without exec by default, causing third party runtimes to crash + (e.g. OpenMP, macOS Accelerate...). + + * __Reusable executor__: strategy to avoid re-spawning a complete + executor every time. A singleton executor instance can be reused (and + dynamically resized if necessary) across consecutive calls to limit + spawning and shutdown overhead. The worker processes can be shutdown + automatically after a configurable idling timeout to free system + resources. + + * __Transparent cloudpickle integration__: to call interactively + defined functions and lambda expressions in parallel. It is also + possible to register a custom pickler implementation to handle + inter-process communications. + + * __No need for ``if __name__ == "__main__":`` in scripts__: thanks + to the use of ``cloudpickle`` to call functions defined in the + ``__main__`` module, it is not required to protect the code calling + parallel functions under Windows. + + * __Deadlock free implementation__: one of the major concern in + standard `multiprocessing` and `concurrent.futures` modules is the + ability of the `Pool/Executor` to handle crashes of worker + processes. This library intends to fix those possible deadlocks and + send back meaningful errors. Note that the implementation of + `concurrent.futures.ProcessPoolExecutor` that comes with Python 3.7+ + is as robust as the executor from loky but the latter also works for + older versions of Python. + + +### Installation + +The recommended way to install `loky` is with `pip`, +```bash +pip install loky +``` + +`loky` can also be installed from sources using +```bash +git clone https://github.com/joblib/loky +cd loky +python setup.py install +``` + +Note that `loky` has an optional dependency on [`psutil`][1] to allow early +memory leak detections. + +### Usage + +The basic usage of `loky` relies on the `get_reusable_executor`, which +internally manages a custom `ProcessPoolExecutor` object, which is reused or +re-spawned depending on the context. + +```python +import os +from time import sleep +from loky import get_reusable_executor + + +def say_hello(k): + pid = os.getpid() + print(f"Hello from {pid} with arg {k}") + sleep(.01) + return pid + + +# Create an executor with 4 worker processes, that will +# automatically shutdown after idling for 2s +executor = get_reusable_executor(max_workers=4, timeout=2) + +res = executor.submit(say_hello, 1) +print("Got results:", res.result()) + +results = executor.map(say_hello, range(50)) +n_workers = len(set(results)) +print("Number of used processes:", n_workers) +assert n_workers == 4 +``` + +For more advance usage, see our +[documentation](https://loky.readthedocs.io/en/stable/) + +### Workflow to contribute + +To contribute to **loky**, first create an account on +[github](http://github.com/). Once this is done, fork the +[loky repository](http://github.com/loky/loky) to have your own repository, +clone it using 'git clone' on the computers where you want to work. Make your +changes in your clone, push them to your github account, test them on several +computers, and when you are happy with them, send a pull request to the main +repository. + +### Running the test suite + +To run the test suite, you need the `pytest` (version >= 3) and `psutil` +modules. From the root of the project, run the test suite using: + +```sh + pip install -e . + pytest . +``` + +### Why was the project named `loky`? + +While developping `loky`, we had some bad experiences trying to debug deadlocks +when using `multiprocessing.Pool` and `concurrent.futures.ProcessPoolExecutor`, +especially when calling functions with non-picklable arguments or returned +values at the beginning of the project. When we had to chose a name, we had +dealt with so many deadlocks that we wanted some kind of invocation to repel +them! Hence `loky`: a mix of a god, locks and the `y` that make it somehow +cooler and nicer : (and also less likely to result in name conflict in google +results ^^). + +Fixes to avoid those deadlocks in `concurrent.futures` were also contributed +upstream in Python 3.7+, as a less mystical way to repel the deadlocks :D + +### Acknowledgement + +This work is supported by the Center for Data Science, funded by the IDEX +Paris-Saclay, ANR-11-IDEX-0003-02 + + +[1]: https://github.com/giampaolo/psutil + + + + +%package help +Summary: Development documents and examples for loky +Provides: python3-loky-doc +%description help + +<a href="https://loky.readthedocs.io"> +<img src="docs/_static/loky_logo.svg" +alt="Loky logo" width=96/></a> + + +# Reusable Process Pool Executor +[](https://dev.azure.com/joblib/loky/_build/latest?definitionId=2&branchName=master) +[](https://loky.readthedocs.io/en/latest/?badge=latest) +[](https://codecov.io/gh/joblib/loky) +[](https://zenodo.org/badge/latestdoi/48578152) + + +### Goal + +The aim of this project is to provide a robust, cross-platform and +cross-version implementation of the `ProcessPoolExecutor` class of +`concurrent.futures`. It notably features: + + * __Consistent and robust spawn behavior__: All processes are started + using fork + exec on POSIX systems. This ensures safer interactions with + third party libraries. On the contrary, `multiprocessing.Pool` uses + fork without exec by default, causing third party runtimes to crash + (e.g. OpenMP, macOS Accelerate...). + + * __Reusable executor__: strategy to avoid re-spawning a complete + executor every time. A singleton executor instance can be reused (and + dynamically resized if necessary) across consecutive calls to limit + spawning and shutdown overhead. The worker processes can be shutdown + automatically after a configurable idling timeout to free system + resources. + + * __Transparent cloudpickle integration__: to call interactively + defined functions and lambda expressions in parallel. It is also + possible to register a custom pickler implementation to handle + inter-process communications. + + * __No need for ``if __name__ == "__main__":`` in scripts__: thanks + to the use of ``cloudpickle`` to call functions defined in the + ``__main__`` module, it is not required to protect the code calling + parallel functions under Windows. + + * __Deadlock free implementation__: one of the major concern in + standard `multiprocessing` and `concurrent.futures` modules is the + ability of the `Pool/Executor` to handle crashes of worker + processes. This library intends to fix those possible deadlocks and + send back meaningful errors. Note that the implementation of + `concurrent.futures.ProcessPoolExecutor` that comes with Python 3.7+ + is as robust as the executor from loky but the latter also works for + older versions of Python. + + +### Installation + +The recommended way to install `loky` is with `pip`, +```bash +pip install loky +``` + +`loky` can also be installed from sources using +```bash +git clone https://github.com/joblib/loky +cd loky +python setup.py install +``` + +Note that `loky` has an optional dependency on [`psutil`][1] to allow early +memory leak detections. + +### Usage + +The basic usage of `loky` relies on the `get_reusable_executor`, which +internally manages a custom `ProcessPoolExecutor` object, which is reused or +re-spawned depending on the context. + +```python +import os +from time import sleep +from loky import get_reusable_executor + + +def say_hello(k): + pid = os.getpid() + print(f"Hello from {pid} with arg {k}") + sleep(.01) + return pid + + +# Create an executor with 4 worker processes, that will +# automatically shutdown after idling for 2s +executor = get_reusable_executor(max_workers=4, timeout=2) + +res = executor.submit(say_hello, 1) +print("Got results:", res.result()) + +results = executor.map(say_hello, range(50)) +n_workers = len(set(results)) +print("Number of used processes:", n_workers) +assert n_workers == 4 +``` + +For more advance usage, see our +[documentation](https://loky.readthedocs.io/en/stable/) + +### Workflow to contribute + +To contribute to **loky**, first create an account on +[github](http://github.com/). Once this is done, fork the +[loky repository](http://github.com/loky/loky) to have your own repository, +clone it using 'git clone' on the computers where you want to work. Make your +changes in your clone, push them to your github account, test them on several +computers, and when you are happy with them, send a pull request to the main +repository. + +### Running the test suite + +To run the test suite, you need the `pytest` (version >= 3) and `psutil` +modules. From the root of the project, run the test suite using: + +```sh + pip install -e . + pytest . +``` + +### Why was the project named `loky`? + +While developping `loky`, we had some bad experiences trying to debug deadlocks +when using `multiprocessing.Pool` and `concurrent.futures.ProcessPoolExecutor`, +especially when calling functions with non-picklable arguments or returned +values at the beginning of the project. When we had to chose a name, we had +dealt with so many deadlocks that we wanted some kind of invocation to repel +them! Hence `loky`: a mix of a god, locks and the `y` that make it somehow +cooler and nicer : (and also less likely to result in name conflict in google +results ^^). + +Fixes to avoid those deadlocks in `concurrent.futures` were also contributed +upstream in Python 3.7+, as a less mystical way to repel the deadlocks :D + +### Acknowledgement + +This work is supported by the Center for Data Science, funded by the IDEX +Paris-Saclay, ANR-11-IDEX-0003-02 + + +[1]: https://github.com/giampaolo/psutil + + + + +%prep +%autosetup -n loky-3.3.0 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-loky -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Tue Apr 11 2023 Python_Bot <Python_Bot@openeuler.org> - 3.3.0-1 +- Package Spec generated |
