diff options
author | CoprDistGit <infra@openeuler.org> | 2023-06-08 18:20:05 +0000 |
---|---|---|
committer | CoprDistGit <infra@openeuler.org> | 2023-06-08 18:20:05 +0000 |
commit | 032ee3d4c7d30be2e8a22980d6b472a692a21d79 (patch) | |
tree | 45a0360a6e25528039abcb115861e29a288ff8ce | |
parent | 77a1f014075d58616e5efacdf7f7378796215d51 (diff) |
automatic import of python-odd-modelsopeneuler20.03
-rw-r--r-- | .gitignore | 1 | ||||
-rw-r--r-- | python-odd-models.spec | 421 | ||||
-rw-r--r-- | sources | 1 |
3 files changed, 423 insertions, 0 deletions
@@ -0,0 +1 @@ +/odd_models-2.0.29.tar.gz diff --git a/python-odd-models.spec b/python-odd-models.spec new file mode 100644 index 0000000..e02ca69 --- /dev/null +++ b/python-odd-models.spec @@ -0,0 +1,421 @@ +%global _empty_manifest_terminate_build 0 +Name: python-odd-models +Version: 2.0.29 +Release: 1 +Summary: Open Data Discovery Models +License: Apache-2.0 +URL: https://github.com/opendatadiscovery/odd-models-package +Source0: https://mirrors.aliyun.com/pypi/web/packages/39/05/a1006b2bb97ec15dc074e73f0d130cf7a2835bf9a999eee855e870e51210/odd_models-2.0.29.tar.gz +BuildArch: noarch + +Requires: python3-pydantic +Requires: python3-sqlparse +Requires: python3-requests +Requires: python3-sql-metadata +Requires: python3-loguru +Requires: python3-oddrn-generator +Requires: python3-funcy + +%description +[](https://badge.fury.io/py/odd-models) + +# OpenDataDiscovery Models package +Has some useful tools for working with OpenDataDiscovery. Such as: +1. Generated Python models from OpenDataDiscovery specification. +2. API Client for working with OpenDataDiscovery. +3. API for manual discovering data entities. + +### Installation +```bash +pip install odd-models +``` + +### Models using example +**odd-models.models** package provides automatically generated Python model by OpenDataDiscovery specification. +It can be used for creating data entities for uploading them into the Platform. + +Code example ([full code](./examples/postgres_models.py)): +```python +from oddrn_generator import PostgresqlGenerator +from odd_models.models import DataEntity, DataSet, DataSetField, DataSetFieldType, DataEntityType, Type, MetadataExtension +generator = PostgresqlGenerator(host_settings="localhost", databases="my_database", schemas="public") +DataEntity( + oddrn=generator.get_oddrn_by_path("tables", "my_table"), + name="my_table", + type=DataEntityType.TABLE, + metadata=[MetadataExtension(schema_url="https://example.com/schema.json", metadata={"env": "DEV"})], + dataset=DataSet( + field_list=[ + DataSetField( + oddrn=generator.get_oddrn_by_path("tables_columns", "name"), + name="name", + type=DataSetFieldType( + type=Type.TYPE_STRING, + logical_type='str', + is_nullable=False + ), + ) + ] + ) +) +``` + + +### HTTP Client for OpenDataDiscovery +___ +**odd-models.client** package provides API client for OpenDataDiscovery API. +Client provides an API for working with OpenDataDiscovery Platform. +It has various methods for working with data sources, data entities, management etc. + +Code example([full code](./examples/client.py)): + +```python +from odd_models.api_client.v2.odd_api_client import Client +from examples.postgres_models import data_entity_list + +client = Client(host="http://localhost:8080") +client.auth(name="dev_aws_token", description="Token for dev AWS account data sources") +client.ingest_data_entity_list(data_entity_list) +``` + +### Manual Discovery API +___ +When there is no programmatic way to discover data sources and data entities, **odd-models.discovery** package provides API for manual discovery of data sources and data entities. + +Code example([full code](./examples/lambda_discovery.py)): + +```python +from odd_models.discovery import DataSource +from odd_models.discovery.data_assets import AWSLambda, S3Artifact +from odd_models.discovery.data_assets.data_asset_list import DataAssetsList + +with DataSource("//cloud/aws/dev") as data_source: + validation_lambda = AWSLambda.from_params( + region="eu-central-1", account="0123456789", function_name="validation" + ) + input_artifact = S3Artifact.from_url("s3://bucket/folder/test_data.csv") + + results = S3Artifact.from_url("s3://bucket/folder/test_result.csv") + metrics = S3Artifact.from_url("s3://bucket/folder/test_metrics.json") + + input_artifact >> validation_lambda >> DataAssetsList([results, metrics]) + + data_source.add_data_asset(validation_lambda) +``` + +# Development + +### Installation +```bash +# Install dependencies +poetry install + +# Activate virtual environment +poetry shell +``` + +### Generating models +```bash +# Generate models. Will generate models pydantic into odd_models/models +make generate_models + +# Generate api client. Will generate api client into odd_models/api_client +make generate_client +``` + +### Tests +```bash +pytest . +``` + +### Docker build +```bash +docker build -t odd-models . +``` + +%package -n python3-odd-models +Summary: Open Data Discovery Models +Provides: python-odd-models +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-odd-models +[](https://badge.fury.io/py/odd-models) + +# OpenDataDiscovery Models package +Has some useful tools for working with OpenDataDiscovery. Such as: +1. Generated Python models from OpenDataDiscovery specification. +2. API Client for working with OpenDataDiscovery. +3. API for manual discovering data entities. + +### Installation +```bash +pip install odd-models +``` + +### Models using example +**odd-models.models** package provides automatically generated Python model by OpenDataDiscovery specification. +It can be used for creating data entities for uploading them into the Platform. + +Code example ([full code](./examples/postgres_models.py)): +```python +from oddrn_generator import PostgresqlGenerator +from odd_models.models import DataEntity, DataSet, DataSetField, DataSetFieldType, DataEntityType, Type, MetadataExtension +generator = PostgresqlGenerator(host_settings="localhost", databases="my_database", schemas="public") +DataEntity( + oddrn=generator.get_oddrn_by_path("tables", "my_table"), + name="my_table", + type=DataEntityType.TABLE, + metadata=[MetadataExtension(schema_url="https://example.com/schema.json", metadata={"env": "DEV"})], + dataset=DataSet( + field_list=[ + DataSetField( + oddrn=generator.get_oddrn_by_path("tables_columns", "name"), + name="name", + type=DataSetFieldType( + type=Type.TYPE_STRING, + logical_type='str', + is_nullable=False + ), + ) + ] + ) +) +``` + + +### HTTP Client for OpenDataDiscovery +___ +**odd-models.client** package provides API client for OpenDataDiscovery API. +Client provides an API for working with OpenDataDiscovery Platform. +It has various methods for working with data sources, data entities, management etc. + +Code example([full code](./examples/client.py)): + +```python +from odd_models.api_client.v2.odd_api_client import Client +from examples.postgres_models import data_entity_list + +client = Client(host="http://localhost:8080") +client.auth(name="dev_aws_token", description="Token for dev AWS account data sources") +client.ingest_data_entity_list(data_entity_list) +``` + +### Manual Discovery API +___ +When there is no programmatic way to discover data sources and data entities, **odd-models.discovery** package provides API for manual discovery of data sources and data entities. + +Code example([full code](./examples/lambda_discovery.py)): + +```python +from odd_models.discovery import DataSource +from odd_models.discovery.data_assets import AWSLambda, S3Artifact +from odd_models.discovery.data_assets.data_asset_list import DataAssetsList + +with DataSource("//cloud/aws/dev") as data_source: + validation_lambda = AWSLambda.from_params( + region="eu-central-1", account="0123456789", function_name="validation" + ) + input_artifact = S3Artifact.from_url("s3://bucket/folder/test_data.csv") + + results = S3Artifact.from_url("s3://bucket/folder/test_result.csv") + metrics = S3Artifact.from_url("s3://bucket/folder/test_metrics.json") + + input_artifact >> validation_lambda >> DataAssetsList([results, metrics]) + + data_source.add_data_asset(validation_lambda) +``` + +# Development + +### Installation +```bash +# Install dependencies +poetry install + +# Activate virtual environment +poetry shell +``` + +### Generating models +```bash +# Generate models. Will generate models pydantic into odd_models/models +make generate_models + +# Generate api client. Will generate api client into odd_models/api_client +make generate_client +``` + +### Tests +```bash +pytest . +``` + +### Docker build +```bash +docker build -t odd-models . +``` + +%package help +Summary: Development documents and examples for odd-models +Provides: python3-odd-models-doc +%description help +[](https://badge.fury.io/py/odd-models) + +# OpenDataDiscovery Models package +Has some useful tools for working with OpenDataDiscovery. Such as: +1. Generated Python models from OpenDataDiscovery specification. +2. API Client for working with OpenDataDiscovery. +3. API for manual discovering data entities. + +### Installation +```bash +pip install odd-models +``` + +### Models using example +**odd-models.models** package provides automatically generated Python model by OpenDataDiscovery specification. +It can be used for creating data entities for uploading them into the Platform. + +Code example ([full code](./examples/postgres_models.py)): +```python +from oddrn_generator import PostgresqlGenerator +from odd_models.models import DataEntity, DataSet, DataSetField, DataSetFieldType, DataEntityType, Type, MetadataExtension +generator = PostgresqlGenerator(host_settings="localhost", databases="my_database", schemas="public") +DataEntity( + oddrn=generator.get_oddrn_by_path("tables", "my_table"), + name="my_table", + type=DataEntityType.TABLE, + metadata=[MetadataExtension(schema_url="https://example.com/schema.json", metadata={"env": "DEV"})], + dataset=DataSet( + field_list=[ + DataSetField( + oddrn=generator.get_oddrn_by_path("tables_columns", "name"), + name="name", + type=DataSetFieldType( + type=Type.TYPE_STRING, + logical_type='str', + is_nullable=False + ), + ) + ] + ) +) +``` + + +### HTTP Client for OpenDataDiscovery +___ +**odd-models.client** package provides API client for OpenDataDiscovery API. +Client provides an API for working with OpenDataDiscovery Platform. +It has various methods for working with data sources, data entities, management etc. + +Code example([full code](./examples/client.py)): + +```python +from odd_models.api_client.v2.odd_api_client import Client +from examples.postgres_models import data_entity_list + +client = Client(host="http://localhost:8080") +client.auth(name="dev_aws_token", description="Token for dev AWS account data sources") +client.ingest_data_entity_list(data_entity_list) +``` + +### Manual Discovery API +___ +When there is no programmatic way to discover data sources and data entities, **odd-models.discovery** package provides API for manual discovery of data sources and data entities. + +Code example([full code](./examples/lambda_discovery.py)): + +```python +from odd_models.discovery import DataSource +from odd_models.discovery.data_assets import AWSLambda, S3Artifact +from odd_models.discovery.data_assets.data_asset_list import DataAssetsList + +with DataSource("//cloud/aws/dev") as data_source: + validation_lambda = AWSLambda.from_params( + region="eu-central-1", account="0123456789", function_name="validation" + ) + input_artifact = S3Artifact.from_url("s3://bucket/folder/test_data.csv") + + results = S3Artifact.from_url("s3://bucket/folder/test_result.csv") + metrics = S3Artifact.from_url("s3://bucket/folder/test_metrics.json") + + input_artifact >> validation_lambda >> DataAssetsList([results, metrics]) + + data_source.add_data_asset(validation_lambda) +``` + +# Development + +### Installation +```bash +# Install dependencies +poetry install + +# Activate virtual environment +poetry shell +``` + +### Generating models +```bash +# Generate models. Will generate models pydantic into odd_models/models +make generate_models + +# Generate api client. Will generate api client into odd_models/api_client +make generate_client +``` + +### Tests +```bash +pytest . +``` + +### Docker build +```bash +docker build -t odd-models . +``` + +%prep +%autosetup -n odd_models-2.0.29 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "\"/%h/%f.gz\"\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-odd-models -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Thu Jun 08 2023 Python_Bot <Python_Bot@openeuler.org> - 2.0.29-1 +- Package Spec generated @@ -0,0 +1 @@ +6056e226cc50d904c59e76c745f06e95 odd_models-2.0.29.tar.gz |