diff options
| author | CoprDistGit <infra@openeuler.org> | 2023-05-29 11:47:44 +0000 |
|---|---|---|
| committer | CoprDistGit <infra@openeuler.org> | 2023-05-29 11:47:44 +0000 |
| commit | 8dc0aae7f3526d593205d94eca4fe4eadeb103b8 (patch) | |
| tree | 2b2a5809a49ec86a426236f18c6d199d19bc787c | |
| parent | 764ec4bc382ea4af37ebd26ed742ea79cd3ad394 (diff) | |
automatic import of python-ssb-ipython-kernels
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | python-ssb-ipython-kernels.spec | 389 | ||||
| -rw-r--r-- | sources | 1 |
3 files changed, 391 insertions, 0 deletions
@@ -0,0 +1 @@ +/ssb-ipython-kernels-0.3.3.tar.gz diff --git a/python-ssb-ipython-kernels.spec b/python-ssb-ipython-kernels.spec new file mode 100644 index 0000000..5e62573 --- /dev/null +++ b/python-ssb-ipython-kernels.spec @@ -0,0 +1,389 @@ +%global _empty_manifest_terminate_build 0 +Name: python-ssb-ipython-kernels +Version: 0.3.3 +Release: 1 +Summary: Jupyter kernels for working with dapla services +License: MIT +URL: https://github.com/statisticsnorway/dapla-ipython-kernels +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/c1/50/0dbd3944013356007b66c31ca1102cd26dbe060e7fd02b7699fa4bd1146d/ssb-ipython-kernels-0.3.3.tar.gz +BuildArch: noarch + +Requires: python3-ipython +Requires: python3-pyspark +Requires: python3-jupyterhub +Requires: python3-oauthenticator +Requires: python3-requests +Requires: python3-requests-cache +Requires: python3-responses +Requires: python3-ipykernel +Requires: python3-notebook +Requires: python3-tornado +Requires: python3-gcsfs +Requires: python3-pyarrow +Requires: python3-pandas +Requires: python3-google-auth +Requires: python3-google-auth-oauthlib +Requires: python3-ipywidgets +Requires: python3-pyjwt + +%description +# dapla-ipython-kernels +Python module for use within Jupyter notebooks. It contains kernel extensions for integrating with Apache Spark, +Google Cloud Storage and custom dapla services. + +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) + +## Getting Started + +Install the module from pip: + +```bash +# pip +pip install dapla-ipython-kernels +``` + +Now the module is ready to use with a single import: + +```python +import dapla as dp +``` + +This module is targeted to python kernels in Jupyter, but it may work in any IPython environment. +It also depends on a number of custom services, e.g. [the custom auth service](dapla/jupyterextensions/authextension.py) + +To test, simply create any Pandas dataframe. This can be stored in Google Cloud Storage at a specific path: + +```python +import pandas as pd +import dapla as dp + +data = { + 'apples': [3, 2, 0, 1], + 'oranges': [0, 3, 7, 2] +} +# Create pandas DataFrame +purchases = pd.DataFrame(data, index=['June', 'Robert', 'Lily', 'David']) + +# Write pandas DataFrame to parquet +dp.write_pandas(purchases, '/testfolder/python/purchases', valuation='INTERNAL', state= 'INPUT') +``` + +Conversely, parquet files can be read from a path directly into a pandas DataFrame. + +```python +import dapla as dp +# Read path into pandas dataframe +purchases = dp.read_pandas('/testfolder/python/purchases') +``` + +## Other functions + +Since the python module integrates with Google Cloud Storage and custom dapla services, +some other functions exist as well: + +```python +import dapla as dp + +# List path by prefix +dp.show('/testfolder/python') +``` +| Path | Timestamp | +| ----------------------------- | ------------- | +| /testfolder/python/purchases | 1593120298095 | +| /testfolder/python/other | 1593157667793 | + + +```python +import dapla as dp + +# Show file details +dp.details('/testfolder/python/purchases') +``` +| Size | Name | +| ----- | -------------------------------------- | +| 2908 | 42331105444c9ca0ce049ef6de7160.parquet | + + +See also the [example notebook](examples/dapla_notebook.ipynb) written for Jupyter. + +## Deploy to SSB jupyter + +### Release version pypi +Make sure you have a clean master branch.<br> +run `make bump-version-patch` - this will update version and commit to git.<br> +run `git push --tags origin master` - important to have --tags to make it auto deploy to pypi + +If everything was ok we should see a new release her: https://pypi.org/project/ssb-ipython-kernels/ + +### Update jupyter image on staging +* Bump ssb-ipython-kernels in dapla-gcp-jupyter [Dockerfile](https://github.com/statisticsnorway/dapla-gcp-jupyter/blob/master/jupyter/Dockerfile) <br> + * Example of previous [update]( https://github.com/statisticsnorway/dapla-gcp-jupyter/commit/8027dc1cbad15dadb1347fe452c78711463e9f3c) <br> +* Check new tag from build on [azure piplines](https://dev.azure.com/statisticsnorway/Dapla/_build/results?buildId=11202&view=logs&jobId=2143f898-48de-5476-aeb8-70e74f8d7c33&j=667c30d6-a912-540e-a406-35cd05a9f751&t=fb539ba6-e537-5346-19c8-c46f7dd4b185) +* update [platform dev jupyter-kubespawner-config](https://github.com/statisticsnorway/platform-dev/blob/master/flux/staging-bip-app/dapla-spark/jupyter/kubespawner-config.yaml) with tag + * [Example](https://github.com/statisticsnorway/platform-dev/commit/b063b830deb6bc0d6a485d7f08fda473cf340ff6) + +For now, we have to delete the running jupyer hub instance to make it use this new config + + + + +%package -n python3-ssb-ipython-kernels +Summary: Jupyter kernels for working with dapla services +Provides: python-ssb-ipython-kernels +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-ssb-ipython-kernels +# dapla-ipython-kernels +Python module for use within Jupyter notebooks. It contains kernel extensions for integrating with Apache Spark, +Google Cloud Storage and custom dapla services. + +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) + +## Getting Started + +Install the module from pip: + +```bash +# pip +pip install dapla-ipython-kernels +``` + +Now the module is ready to use with a single import: + +```python +import dapla as dp +``` + +This module is targeted to python kernels in Jupyter, but it may work in any IPython environment. +It also depends on a number of custom services, e.g. [the custom auth service](dapla/jupyterextensions/authextension.py) + +To test, simply create any Pandas dataframe. This can be stored in Google Cloud Storage at a specific path: + +```python +import pandas as pd +import dapla as dp + +data = { + 'apples': [3, 2, 0, 1], + 'oranges': [0, 3, 7, 2] +} +# Create pandas DataFrame +purchases = pd.DataFrame(data, index=['June', 'Robert', 'Lily', 'David']) + +# Write pandas DataFrame to parquet +dp.write_pandas(purchases, '/testfolder/python/purchases', valuation='INTERNAL', state= 'INPUT') +``` + +Conversely, parquet files can be read from a path directly into a pandas DataFrame. + +```python +import dapla as dp +# Read path into pandas dataframe +purchases = dp.read_pandas('/testfolder/python/purchases') +``` + +## Other functions + +Since the python module integrates with Google Cloud Storage and custom dapla services, +some other functions exist as well: + +```python +import dapla as dp + +# List path by prefix +dp.show('/testfolder/python') +``` +| Path | Timestamp | +| ----------------------------- | ------------- | +| /testfolder/python/purchases | 1593120298095 | +| /testfolder/python/other | 1593157667793 | + + +```python +import dapla as dp + +# Show file details +dp.details('/testfolder/python/purchases') +``` +| Size | Name | +| ----- | -------------------------------------- | +| 2908 | 42331105444c9ca0ce049ef6de7160.parquet | + + +See also the [example notebook](examples/dapla_notebook.ipynb) written for Jupyter. + +## Deploy to SSB jupyter + +### Release version pypi +Make sure you have a clean master branch.<br> +run `make bump-version-patch` - this will update version and commit to git.<br> +run `git push --tags origin master` - important to have --tags to make it auto deploy to pypi + +If everything was ok we should see a new release her: https://pypi.org/project/ssb-ipython-kernels/ + +### Update jupyter image on staging +* Bump ssb-ipython-kernels in dapla-gcp-jupyter [Dockerfile](https://github.com/statisticsnorway/dapla-gcp-jupyter/blob/master/jupyter/Dockerfile) <br> + * Example of previous [update]( https://github.com/statisticsnorway/dapla-gcp-jupyter/commit/8027dc1cbad15dadb1347fe452c78711463e9f3c) <br> +* Check new tag from build on [azure piplines](https://dev.azure.com/statisticsnorway/Dapla/_build/results?buildId=11202&view=logs&jobId=2143f898-48de-5476-aeb8-70e74f8d7c33&j=667c30d6-a912-540e-a406-35cd05a9f751&t=fb539ba6-e537-5346-19c8-c46f7dd4b185) +* update [platform dev jupyter-kubespawner-config](https://github.com/statisticsnorway/platform-dev/blob/master/flux/staging-bip-app/dapla-spark/jupyter/kubespawner-config.yaml) with tag + * [Example](https://github.com/statisticsnorway/platform-dev/commit/b063b830deb6bc0d6a485d7f08fda473cf340ff6) + +For now, we have to delete the running jupyer hub instance to make it use this new config + + + + +%package help +Summary: Development documents and examples for ssb-ipython-kernels +Provides: python3-ssb-ipython-kernels-doc +%description help +# dapla-ipython-kernels +Python module for use within Jupyter notebooks. It contains kernel extensions for integrating with Apache Spark, +Google Cloud Storage and custom dapla services. + +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) +[](https://pypi.python.org/pypi/ssb-ipython-kernels/) + +## Getting Started + +Install the module from pip: + +```bash +# pip +pip install dapla-ipython-kernels +``` + +Now the module is ready to use with a single import: + +```python +import dapla as dp +``` + +This module is targeted to python kernels in Jupyter, but it may work in any IPython environment. +It also depends on a number of custom services, e.g. [the custom auth service](dapla/jupyterextensions/authextension.py) + +To test, simply create any Pandas dataframe. This can be stored in Google Cloud Storage at a specific path: + +```python +import pandas as pd +import dapla as dp + +data = { + 'apples': [3, 2, 0, 1], + 'oranges': [0, 3, 7, 2] +} +# Create pandas DataFrame +purchases = pd.DataFrame(data, index=['June', 'Robert', 'Lily', 'David']) + +# Write pandas DataFrame to parquet +dp.write_pandas(purchases, '/testfolder/python/purchases', valuation='INTERNAL', state= 'INPUT') +``` + +Conversely, parquet files can be read from a path directly into a pandas DataFrame. + +```python +import dapla as dp +# Read path into pandas dataframe +purchases = dp.read_pandas('/testfolder/python/purchases') +``` + +## Other functions + +Since the python module integrates with Google Cloud Storage and custom dapla services, +some other functions exist as well: + +```python +import dapla as dp + +# List path by prefix +dp.show('/testfolder/python') +``` +| Path | Timestamp | +| ----------------------------- | ------------- | +| /testfolder/python/purchases | 1593120298095 | +| /testfolder/python/other | 1593157667793 | + + +```python +import dapla as dp + +# Show file details +dp.details('/testfolder/python/purchases') +``` +| Size | Name | +| ----- | -------------------------------------- | +| 2908 | 42331105444c9ca0ce049ef6de7160.parquet | + + +See also the [example notebook](examples/dapla_notebook.ipynb) written for Jupyter. + +## Deploy to SSB jupyter + +### Release version pypi +Make sure you have a clean master branch.<br> +run `make bump-version-patch` - this will update version and commit to git.<br> +run `git push --tags origin master` - important to have --tags to make it auto deploy to pypi + +If everything was ok we should see a new release her: https://pypi.org/project/ssb-ipython-kernels/ + +### Update jupyter image on staging +* Bump ssb-ipython-kernels in dapla-gcp-jupyter [Dockerfile](https://github.com/statisticsnorway/dapla-gcp-jupyter/blob/master/jupyter/Dockerfile) <br> + * Example of previous [update]( https://github.com/statisticsnorway/dapla-gcp-jupyter/commit/8027dc1cbad15dadb1347fe452c78711463e9f3c) <br> +* Check new tag from build on [azure piplines](https://dev.azure.com/statisticsnorway/Dapla/_build/results?buildId=11202&view=logs&jobId=2143f898-48de-5476-aeb8-70e74f8d7c33&j=667c30d6-a912-540e-a406-35cd05a9f751&t=fb539ba6-e537-5346-19c8-c46f7dd4b185) +* update [platform dev jupyter-kubespawner-config](https://github.com/statisticsnorway/platform-dev/blob/master/flux/staging-bip-app/dapla-spark/jupyter/kubespawner-config.yaml) with tag + * [Example](https://github.com/statisticsnorway/platform-dev/commit/b063b830deb6bc0d6a485d7f08fda473cf340ff6) + +For now, we have to delete the running jupyer hub instance to make it use this new config + + + + +%prep +%autosetup -n ssb-ipython-kernels-0.3.3 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-ssb-ipython-kernels -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Mon May 29 2023 Python_Bot <Python_Bot@openeuler.org> - 0.3.3-1 +- Package Spec generated @@ -0,0 +1 @@ +9af992771ac12376c0dd875a5e08807a ssb-ipython-kernels-0.3.3.tar.gz |
