summaryrefslogtreecommitdiff
path: root/python-afar.spec
diff options
context:
space:
mode:
Diffstat (limited to 'python-afar.spec')
-rw-r--r--python-afar.spec593
1 files changed, 593 insertions, 0 deletions
diff --git a/python-afar.spec b/python-afar.spec
new file mode 100644
index 0000000..01cff0b
--- /dev/null
+++ b/python-afar.spec
@@ -0,0 +1,593 @@
+%global _empty_manifest_terminate_build 0
+Name: python-afar
+Version: 0.6.1
+Release: 1
+Summary: Run code on a Dask cluster via a context manager or IPython magic
+License: BSD
+URL: https://github.com/eriknw/afar
+Source0: https://mirrors.nju.edu.cn/pypi/web/packages/4e/1f/8ed4bcfd09861713d23ade76e4dc3e479cefe02e13a88aec4ab8ce55f42a/afar-0.6.1.tar.gz
+BuildArch: noarch
+
+Requires: python3-innerscope
+Requires: python3-distributed
+
+%description
+# **Afar**
+[![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)
+[![Version](https://img.shields.io/pypi/v/afar.svg)](https://pypi.org/project/afar/)
+[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/eriknw/afar/blob/main/LICENSE)
+[![Build Status](https://github.com/eriknw/afar/workflows/Test/badge.svg)](https://github.com/eriknw/afar/actions)
+[![Coverage Status](https://coveralls.io/repos/eriknw/afar/badge.svg?branch=main)](https://coveralls.io/r/eriknw/afar)
+[![Code style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+> **_One man's magic is another man's engineering_**<br>
+> _Robert A. Heinlein_
+<hr>
+
+## Installation
+`afar` may be installed with pip:
+```bash
+pip install afar
+```
+or with [conda](https://docs.conda.io/en/latest/):
+```bash
+conda install -c conda-forge afar
+```
+
+## What is it?
+`afar` allows you to run code on a remote [Dask](https://dask.org/) [cluster](https://distributed.dask.org/en/latest/) using context managers and [IPython magics](#Magic). For example:
+```python
+import afar
+from dask.distributed import Client
+client = Client()
+
+with afar.run, remotely:
+ import dask_cudf
+ df = dask_cudf.read_parquet("s3://...")
+ result = df.sum().compute()
+```
+Outside the context, `result` is a [Dask Future](https://docs.dask.org/en/latest/futures.html) whose data resides on a worker. `result.result()` is necessary to copy the data locally.
+
+By default, only the last assignment is saved. One can specify which variables to save:
+```python
+with afar.run("one", "two"), remotely:
+ one = 1
+ two = one + 1
+```
+`one` and `two` are now both Futures. They can be used directly in other `afar.run` contexts:
+```python
+with afar.run as data, remotely:
+ three = one + two
+
+assert three.result() == 3
+assert data["three"].result() == 3
+```
+`data` above is a dictionary of variable names to Futures. It may be necessary at times to get the data from here. Alternatively, you may pass a mapping to `afar.run` to use as the data.
+```python
+run = afar.run(data={"four": 4})
+with run, remotely:
+ seven = three + four
+assert run.data["seven"].result() == 7
+```
+If you want to automatically gather the data locally (to avoid calling `.result()`), use `afar.get` instead of `afar.run`:
+```python
+with afar.get, remotely:
+ five = two + three
+assert five == 5
+```
+## Interactivity in Jupyter
+There are several enhancements when using `afar` in Jupyter Notebook or Qt console, JupyterLab, or any IPython-based frontend that supports rich display.
+
+The rich repr of the final expression will be displayed if it's not an assignment:
+```python
+with afar.run, remotely:
+ three + seven
+# displays 10!
+```
+
+Printing is captured and displayed locally:
+```python
+with afar.run, remotely:
+ print(three)
+ print(seven, file=sys.stderr)
+# 3
+# 7
+```
+These are done asynchronously using `ipywidgets`.
+
+### Magic!
+First load `afar` magic extension:
+```python
+%load_ext afar
+```
+Now you can use `afar` as line or cell magic. `%%afar` is like `with afar.run, remotely:`. It can optionally accept a list of variable names to save:
+```python
+%%afar x, y
+x = 1
+y = x + 1
+```
+and
+```python
+z = %afar x + y
+```
+## Is this a good idea?
+
+I don't know, but it sure is a joy to use 😃 !
+
+For motivation, see https://github.com/dask/distributed/issues/4003
+
+It's natural to be skeptical of unconventional syntax. And magic. `afar` is both unconventional and magical, yet it also works well and is surprisingly *fun*! Why not give it a try to see what you think?
+
+We're still exploring the usability of `afar` [and want to hear what you think](https://github.com/eriknw/afar/discussions). As you're learning `afar`, please ask yourself questions such as:
+- can we spell anything better?
+- does this offer opportunities?
+- what is surprising?
+- what is lacking?
+
+Here's an example of an opportunity:
+```python
+on_gpus = afar.remotely(resources={"GPU": 1})
+
+with afar.run, on_gpus:
+ ...
+```
+This now works! Keyword arguments to `remotely` will be passed to [`client.submit`](https://distributed.dask.org/en/latest/api.html#distributed.Client.submit).
+
+I don't know about you, but I think this is starting to look and feel kinda nice, and it could probably be even better :)
+
+## Caveats and Gotchas
+
+### Repeatedly copying data
+
+`afar` automatically gets the data it needs--and only the data it needs--from the outer scope
+and sends it to the Dask cluster to compute on. Since we don't know whether local data has been modified
+between calls to `afar`, we serialize and send local variables every time we use `run` or `get`.
+This is generally fine: it works, it's safe, and is usually fast enough. However, if you do this
+frequently with large-ish data, the performance could suffer, and you may be using
+more memory on your local machine than necessary.
+
+With Dask, a common pattern is to send data to the cluster with `scatter` and get a `Future` back. This works:
+```python
+A = np.arange(10**7)
+A = client.scatter(A)
+with afar.run, remotely:
+ B = A + 1
+# A and B are now both Futures; their data is on the cluster
+```
+
+Another option is to pass `data` to `run`:
+```python
+run = afar.run(data={"A": np.arange(10**7)})
+with afar.run, remotely:
+ B = A + 1
+# run.data["A"] and B are now both Futures; their data is on the cluster
+```
+Here's a nifty trick to use if you're in an IPython notebook: use `data=globals()`!
+```python
+run = afar.run(data=globals())
+A = np.arange(10**7)
+with run, remotely:
+ B = A + 1
+# A and B are now both Futures; their data is on the cluster
+```
+### Mutating remote data
+As with any Dask workload, one should be careful to not modify remote data that may be reused.
+
+### Mutating local data
+Similarly, code run remotely isn't able to mutate local variables. For example:
+```python
+d = {}
+with afar.run, remotely:
+ d['key'] = 'value'
+# d == {}
+```
+## *✨ This code is highly experimental and magical! ✨*
+
+
+
+
+
+%package -n python3-afar
+Summary: Run code on a Dask cluster via a context manager or IPython magic
+Provides: python-afar
+BuildRequires: python3-devel
+BuildRequires: python3-setuptools
+BuildRequires: python3-pip
+%description -n python3-afar
+# **Afar**
+[![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)
+[![Version](https://img.shields.io/pypi/v/afar.svg)](https://pypi.org/project/afar/)
+[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/eriknw/afar/blob/main/LICENSE)
+[![Build Status](https://github.com/eriknw/afar/workflows/Test/badge.svg)](https://github.com/eriknw/afar/actions)
+[![Coverage Status](https://coveralls.io/repos/eriknw/afar/badge.svg?branch=main)](https://coveralls.io/r/eriknw/afar)
+[![Code style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+> **_One man's magic is another man's engineering_**<br>
+> _Robert A. Heinlein_
+<hr>
+
+## Installation
+`afar` may be installed with pip:
+```bash
+pip install afar
+```
+or with [conda](https://docs.conda.io/en/latest/):
+```bash
+conda install -c conda-forge afar
+```
+
+## What is it?
+`afar` allows you to run code on a remote [Dask](https://dask.org/) [cluster](https://distributed.dask.org/en/latest/) using context managers and [IPython magics](#Magic). For example:
+```python
+import afar
+from dask.distributed import Client
+client = Client()
+
+with afar.run, remotely:
+ import dask_cudf
+ df = dask_cudf.read_parquet("s3://...")
+ result = df.sum().compute()
+```
+Outside the context, `result` is a [Dask Future](https://docs.dask.org/en/latest/futures.html) whose data resides on a worker. `result.result()` is necessary to copy the data locally.
+
+By default, only the last assignment is saved. One can specify which variables to save:
+```python
+with afar.run("one", "two"), remotely:
+ one = 1
+ two = one + 1
+```
+`one` and `two` are now both Futures. They can be used directly in other `afar.run` contexts:
+```python
+with afar.run as data, remotely:
+ three = one + two
+
+assert three.result() == 3
+assert data["three"].result() == 3
+```
+`data` above is a dictionary of variable names to Futures. It may be necessary at times to get the data from here. Alternatively, you may pass a mapping to `afar.run` to use as the data.
+```python
+run = afar.run(data={"four": 4})
+with run, remotely:
+ seven = three + four
+assert run.data["seven"].result() == 7
+```
+If you want to automatically gather the data locally (to avoid calling `.result()`), use `afar.get` instead of `afar.run`:
+```python
+with afar.get, remotely:
+ five = two + three
+assert five == 5
+```
+## Interactivity in Jupyter
+There are several enhancements when using `afar` in Jupyter Notebook or Qt console, JupyterLab, or any IPython-based frontend that supports rich display.
+
+The rich repr of the final expression will be displayed if it's not an assignment:
+```python
+with afar.run, remotely:
+ three + seven
+# displays 10!
+```
+
+Printing is captured and displayed locally:
+```python
+with afar.run, remotely:
+ print(three)
+ print(seven, file=sys.stderr)
+# 3
+# 7
+```
+These are done asynchronously using `ipywidgets`.
+
+### Magic!
+First load `afar` magic extension:
+```python
+%load_ext afar
+```
+Now you can use `afar` as line or cell magic. `%%afar` is like `with afar.run, remotely:`. It can optionally accept a list of variable names to save:
+```python
+%%afar x, y
+x = 1
+y = x + 1
+```
+and
+```python
+z = %afar x + y
+```
+## Is this a good idea?
+
+I don't know, but it sure is a joy to use 😃 !
+
+For motivation, see https://github.com/dask/distributed/issues/4003
+
+It's natural to be skeptical of unconventional syntax. And magic. `afar` is both unconventional and magical, yet it also works well and is surprisingly *fun*! Why not give it a try to see what you think?
+
+We're still exploring the usability of `afar` [and want to hear what you think](https://github.com/eriknw/afar/discussions). As you're learning `afar`, please ask yourself questions such as:
+- can we spell anything better?
+- does this offer opportunities?
+- what is surprising?
+- what is lacking?
+
+Here's an example of an opportunity:
+```python
+on_gpus = afar.remotely(resources={"GPU": 1})
+
+with afar.run, on_gpus:
+ ...
+```
+This now works! Keyword arguments to `remotely` will be passed to [`client.submit`](https://distributed.dask.org/en/latest/api.html#distributed.Client.submit).
+
+I don't know about you, but I think this is starting to look and feel kinda nice, and it could probably be even better :)
+
+## Caveats and Gotchas
+
+### Repeatedly copying data
+
+`afar` automatically gets the data it needs--and only the data it needs--from the outer scope
+and sends it to the Dask cluster to compute on. Since we don't know whether local data has been modified
+between calls to `afar`, we serialize and send local variables every time we use `run` or `get`.
+This is generally fine: it works, it's safe, and is usually fast enough. However, if you do this
+frequently with large-ish data, the performance could suffer, and you may be using
+more memory on your local machine than necessary.
+
+With Dask, a common pattern is to send data to the cluster with `scatter` and get a `Future` back. This works:
+```python
+A = np.arange(10**7)
+A = client.scatter(A)
+with afar.run, remotely:
+ B = A + 1
+# A and B are now both Futures; their data is on the cluster
+```
+
+Another option is to pass `data` to `run`:
+```python
+run = afar.run(data={"A": np.arange(10**7)})
+with afar.run, remotely:
+ B = A + 1
+# run.data["A"] and B are now both Futures; their data is on the cluster
+```
+Here's a nifty trick to use if you're in an IPython notebook: use `data=globals()`!
+```python
+run = afar.run(data=globals())
+A = np.arange(10**7)
+with run, remotely:
+ B = A + 1
+# A and B are now both Futures; their data is on the cluster
+```
+### Mutating remote data
+As with any Dask workload, one should be careful to not modify remote data that may be reused.
+
+### Mutating local data
+Similarly, code run remotely isn't able to mutate local variables. For example:
+```python
+d = {}
+with afar.run, remotely:
+ d['key'] = 'value'
+# d == {}
+```
+## *✨ This code is highly experimental and magical! ✨*
+
+
+
+
+
+%package help
+Summary: Development documents and examples for afar
+Provides: python3-afar-doc
+%description help
+# **Afar**
+[![Python Version](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue)
+[![Version](https://img.shields.io/pypi/v/afar.svg)](https://pypi.org/project/afar/)
+[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/eriknw/afar/blob/main/LICENSE)
+[![Build Status](https://github.com/eriknw/afar/workflows/Test/badge.svg)](https://github.com/eriknw/afar/actions)
+[![Coverage Status](https://coveralls.io/repos/eriknw/afar/badge.svg?branch=main)](https://coveralls.io/r/eriknw/afar)
+[![Code style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+> **_One man's magic is another man's engineering_**<br>
+> _Robert A. Heinlein_
+<hr>
+
+## Installation
+`afar` may be installed with pip:
+```bash
+pip install afar
+```
+or with [conda](https://docs.conda.io/en/latest/):
+```bash
+conda install -c conda-forge afar
+```
+
+## What is it?
+`afar` allows you to run code on a remote [Dask](https://dask.org/) [cluster](https://distributed.dask.org/en/latest/) using context managers and [IPython magics](#Magic). For example:
+```python
+import afar
+from dask.distributed import Client
+client = Client()
+
+with afar.run, remotely:
+ import dask_cudf
+ df = dask_cudf.read_parquet("s3://...")
+ result = df.sum().compute()
+```
+Outside the context, `result` is a [Dask Future](https://docs.dask.org/en/latest/futures.html) whose data resides on a worker. `result.result()` is necessary to copy the data locally.
+
+By default, only the last assignment is saved. One can specify which variables to save:
+```python
+with afar.run("one", "two"), remotely:
+ one = 1
+ two = one + 1
+```
+`one` and `two` are now both Futures. They can be used directly in other `afar.run` contexts:
+```python
+with afar.run as data, remotely:
+ three = one + two
+
+assert three.result() == 3
+assert data["three"].result() == 3
+```
+`data` above is a dictionary of variable names to Futures. It may be necessary at times to get the data from here. Alternatively, you may pass a mapping to `afar.run` to use as the data.
+```python
+run = afar.run(data={"four": 4})
+with run, remotely:
+ seven = three + four
+assert run.data["seven"].result() == 7
+```
+If you want to automatically gather the data locally (to avoid calling `.result()`), use `afar.get` instead of `afar.run`:
+```python
+with afar.get, remotely:
+ five = two + three
+assert five == 5
+```
+## Interactivity in Jupyter
+There are several enhancements when using `afar` in Jupyter Notebook or Qt console, JupyterLab, or any IPython-based frontend that supports rich display.
+
+The rich repr of the final expression will be displayed if it's not an assignment:
+```python
+with afar.run, remotely:
+ three + seven
+# displays 10!
+```
+
+Printing is captured and displayed locally:
+```python
+with afar.run, remotely:
+ print(three)
+ print(seven, file=sys.stderr)
+# 3
+# 7
+```
+These are done asynchronously using `ipywidgets`.
+
+### Magic!
+First load `afar` magic extension:
+```python
+%load_ext afar
+```
+Now you can use `afar` as line or cell magic. `%%afar` is like `with afar.run, remotely:`. It can optionally accept a list of variable names to save:
+```python
+%%afar x, y
+x = 1
+y = x + 1
+```
+and
+```python
+z = %afar x + y
+```
+## Is this a good idea?
+
+I don't know, but it sure is a joy to use 😃 !
+
+For motivation, see https://github.com/dask/distributed/issues/4003
+
+It's natural to be skeptical of unconventional syntax. And magic. `afar` is both unconventional and magical, yet it also works well and is surprisingly *fun*! Why not give it a try to see what you think?
+
+We're still exploring the usability of `afar` [and want to hear what you think](https://github.com/eriknw/afar/discussions). As you're learning `afar`, please ask yourself questions such as:
+- can we spell anything better?
+- does this offer opportunities?
+- what is surprising?
+- what is lacking?
+
+Here's an example of an opportunity:
+```python
+on_gpus = afar.remotely(resources={"GPU": 1})
+
+with afar.run, on_gpus:
+ ...
+```
+This now works! Keyword arguments to `remotely` will be passed to [`client.submit`](https://distributed.dask.org/en/latest/api.html#distributed.Client.submit).
+
+I don't know about you, but I think this is starting to look and feel kinda nice, and it could probably be even better :)
+
+## Caveats and Gotchas
+
+### Repeatedly copying data
+
+`afar` automatically gets the data it needs--and only the data it needs--from the outer scope
+and sends it to the Dask cluster to compute on. Since we don't know whether local data has been modified
+between calls to `afar`, we serialize and send local variables every time we use `run` or `get`.
+This is generally fine: it works, it's safe, and is usually fast enough. However, if you do this
+frequently with large-ish data, the performance could suffer, and you may be using
+more memory on your local machine than necessary.
+
+With Dask, a common pattern is to send data to the cluster with `scatter` and get a `Future` back. This works:
+```python
+A = np.arange(10**7)
+A = client.scatter(A)
+with afar.run, remotely:
+ B = A + 1
+# A and B are now both Futures; their data is on the cluster
+```
+
+Another option is to pass `data` to `run`:
+```python
+run = afar.run(data={"A": np.arange(10**7)})
+with afar.run, remotely:
+ B = A + 1
+# run.data["A"] and B are now both Futures; their data is on the cluster
+```
+Here's a nifty trick to use if you're in an IPython notebook: use `data=globals()`!
+```python
+run = afar.run(data=globals())
+A = np.arange(10**7)
+with run, remotely:
+ B = A + 1
+# A and B are now both Futures; their data is on the cluster
+```
+### Mutating remote data
+As with any Dask workload, one should be careful to not modify remote data that may be reused.
+
+### Mutating local data
+Similarly, code run remotely isn't able to mutate local variables. For example:
+```python
+d = {}
+with afar.run, remotely:
+ d['key'] = 'value'
+# d == {}
+```
+## *✨ This code is highly experimental and magical! ✨*
+
+
+
+
+
+%prep
+%autosetup -n afar-0.6.1
+
+%build
+%py3_build
+
+%install
+%py3_install
+install -d -m755 %{buildroot}/%{_pkgdocdir}
+if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
+if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
+if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
+if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
+pushd %{buildroot}
+if [ -d usr/lib ]; then
+ find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+if [ -d usr/lib64 ]; then
+ find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+if [ -d usr/bin ]; then
+ find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+if [ -d usr/sbin ]; then
+ find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+touch doclist.lst
+if [ -d usr/share/man ]; then
+ find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
+fi
+popd
+mv %{buildroot}/filelist.lst .
+mv %{buildroot}/doclist.lst .
+
+%files -n python3-afar -f filelist.lst
+%dir %{python3_sitelib}/*
+
+%files help -f doclist.lst
+%{_docdir}/*
+
+%changelog
+* Wed May 17 2023 Python_Bot <Python_Bot@openeuler.org> - 0.6.1-1
+- Package Spec generated