diff options
Diffstat (limited to 'python-airbyte-cdk.spec')
| -rw-r--r-- | python-airbyte-cdk.spec | 526 |
1 files changed, 526 insertions, 0 deletions
diff --git a/python-airbyte-cdk.spec b/python-airbyte-cdk.spec new file mode 100644 index 0000000..ea8b7de --- /dev/null +++ b/python-airbyte-cdk.spec @@ -0,0 +1,526 @@ +%global _empty_manifest_terminate_build 0 +Name: python-airbyte-cdk +Version: 0.34.0 +Release: 1 +Summary: A framework for writing Airbyte Connectors. +License: MIT +URL: https://github.com/airbytehq/airbyte +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/32/da/272296b4116d6ce36c01881dd3f8fbd8bb9bccf79820ac9254408da75a13/airbyte-cdk-0.34.0.tar.gz +BuildArch: noarch + +Requires: python3-airbyte-protocol-models +Requires: python3-backoff +Requires: python3-dpath +Requires: python3-isodate +Requires: python3-jsonschema +Requires: python3-jsonref +Requires: python3-pendulum +Requires: python3-genson +Requires: python3-pydantic +Requires: python3-dateutil +Requires: python3-PyYAML +Requires: python3-requests +Requires: python3-requests-cache +Requires: python3-Deprecated +Requires: python3-Jinja2 +Requires: python3-cachetools +Requires: python3-freezegun +Requires: python3-MyPy +Requires: python3-pytest +Requires: python3-pytest-cov +Requires: python3-pytest-mock +Requires: python3-requests-mock +Requires: python3-pytest-httpserver +Requires: python3-Sphinx +Requires: python3-sphinx-rtd-theme + +%description +# Connector Development Kit \(Python\) + +The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for: + +* HTTP APIs \(REST APIs, GraphQL, etc..\) +* Singer Taps +* Generic Python sources \(anything not covered by the above\) + +The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate. + +This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding. + +## Getting Started + +Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run + +```text +cd airbyte-integrations/connector-templates/generator +./generate.sh +``` + +then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector. + +Additionally, you can follow [this tutorial](https://docs.airbyte.io/connector-development/tutorials/cdk-tutorial-python-http) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK. + +### Concepts & Documentation + +See the [concepts docs](docs/concepts/) for a tour through what the API offers. + +### Example Connectors + +**HTTP Connectors**: + +* [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py) +* [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py) +* [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py) + +**Singer connectors**: + +* [Salesforce](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce-singer/source_salesforce_singer/source.py) +* [Github](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github-singer/source_github_singer/source.py) + +**Simple Python connectors using the barebones `Source` abstraction**: + +* [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py) +* [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py) + +## Contributing + +### First time setup + +We assume `python` points to python >=3.8. + +Setup a virtual env: + +```text +python -m venv .venv +source .venv/bin/activate +pip install -e ".[dev]" # [dev] installs development-only dependencies +``` + +#### Iteration + +* Iterate on the code locally +* Run tests via `python -m pytest -s unit_tests` +* Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `.mypy.ini`. +* The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it! + +##### Autogenerated files +If the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run: +```commandline +SUB_BUILD=CDK ./gradlew format +``` +This will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on +pre-commit to the others, you can run the appropriate generation command i.e. `./gradlew generateComponentManifestClassFiles`. + +#### Testing + +All tests are located in the `unit_tests` directory. Run `python -m pytest --cov=airbyte_cdk unit_tests/` to run them. This also presents a test coverage report. + +#### Building and testing a connector with your local CDK + +When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways: +* Running a connector locally +* Building and running a source via Docker + +##### Installing your local CDK into a local Python connector + +In order to get a local Python connector running your local CDK, do the following. + +First, make sure you have your connector's virtual environment active: +```bash +# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory +source .venv/bin/activate + +# if you haven't installed dependencies for your connector already +pip install -e . +``` + +Then, navigate to the CDK and install it in editable mode: +```bash +cd ../../../airbyte-cdk/python +pip install -e . +``` + +You should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment. + +##### Building a Python connector in Docker with your local CDK installed + +You can build your connector image with the local CDK using +```bash +# from the airbytehq/airbyte base directory +CONNECTOR_TAG=<TAG_NAME> CONNECTOR_NAME=<CONNECTOR_NAME> sh airbyte-integrations/scripts/build-connector-image-with-local-cdk.sh +``` +Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected. + +##### Running Connector Acceptance Tests for a single connector in Docker with your local CDK installed + +To run acceptance tests for a single connectors using the local CDK, from the connector directory, run +```bash +LOCAL_CDK=1 sh acceptance-test-docker.sh +``` +To additionally fetch secrets required by CATs, set the `FETCH_SECRETS` environment variable. This requires you to have a Google Service Account, and the GCP_GSM_CREDENTIALS environment variable to be set, per the instructions [here](https://github.com/airbytehq/airbyte/tree/b03653a24ef16be641333380f3a4d178271df0ee/tools/ci_credentials). + +##### Running Connector Acceptance Tests for multiple connectors in Docker with your local CDK installed + +To run acceptance tests for multiple connectors using the local CDK, from the root of the `airbyte` repo, run +```bash +./airbyte-cdk/python/bin/run-cats-with-local-cdk.sh -c <connector1>,<connector2>,... +``` + +#### Publishing a new version to PyPi + +1. Open a PR +2. Once it is approved and **merged**, an Airbyte member must run the `Publish CDK Manually` workflow from master using `release-type=major|manor|patch` and setting the changelog message. + +## Coming Soon + +* Full OAuth 2.0 support \(including refresh token issuing flow via UI or CLI\) +* Airbyte Java HTTP CDK +* CDK for Async HTTP endpoints \(request-poll-wait style endpoints\) +* CDK for other protocols +* Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=) + + +%package -n python3-airbyte-cdk +Summary: A framework for writing Airbyte Connectors. +Provides: python-airbyte-cdk +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-airbyte-cdk +# Connector Development Kit \(Python\) + +The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for: + +* HTTP APIs \(REST APIs, GraphQL, etc..\) +* Singer Taps +* Generic Python sources \(anything not covered by the above\) + +The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate. + +This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding. + +## Getting Started + +Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run + +```text +cd airbyte-integrations/connector-templates/generator +./generate.sh +``` + +then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector. + +Additionally, you can follow [this tutorial](https://docs.airbyte.io/connector-development/tutorials/cdk-tutorial-python-http) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK. + +### Concepts & Documentation + +See the [concepts docs](docs/concepts/) for a tour through what the API offers. + +### Example Connectors + +**HTTP Connectors**: + +* [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py) +* [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py) +* [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py) + +**Singer connectors**: + +* [Salesforce](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce-singer/source_salesforce_singer/source.py) +* [Github](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github-singer/source_github_singer/source.py) + +**Simple Python connectors using the barebones `Source` abstraction**: + +* [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py) +* [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py) + +## Contributing + +### First time setup + +We assume `python` points to python >=3.8. + +Setup a virtual env: + +```text +python -m venv .venv +source .venv/bin/activate +pip install -e ".[dev]" # [dev] installs development-only dependencies +``` + +#### Iteration + +* Iterate on the code locally +* Run tests via `python -m pytest -s unit_tests` +* Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `.mypy.ini`. +* The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it! + +##### Autogenerated files +If the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run: +```commandline +SUB_BUILD=CDK ./gradlew format +``` +This will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on +pre-commit to the others, you can run the appropriate generation command i.e. `./gradlew generateComponentManifestClassFiles`. + +#### Testing + +All tests are located in the `unit_tests` directory. Run `python -m pytest --cov=airbyte_cdk unit_tests/` to run them. This also presents a test coverage report. + +#### Building and testing a connector with your local CDK + +When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways: +* Running a connector locally +* Building and running a source via Docker + +##### Installing your local CDK into a local Python connector + +In order to get a local Python connector running your local CDK, do the following. + +First, make sure you have your connector's virtual environment active: +```bash +# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory +source .venv/bin/activate + +# if you haven't installed dependencies for your connector already +pip install -e . +``` + +Then, navigate to the CDK and install it in editable mode: +```bash +cd ../../../airbyte-cdk/python +pip install -e . +``` + +You should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment. + +##### Building a Python connector in Docker with your local CDK installed + +You can build your connector image with the local CDK using +```bash +# from the airbytehq/airbyte base directory +CONNECTOR_TAG=<TAG_NAME> CONNECTOR_NAME=<CONNECTOR_NAME> sh airbyte-integrations/scripts/build-connector-image-with-local-cdk.sh +``` +Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected. + +##### Running Connector Acceptance Tests for a single connector in Docker with your local CDK installed + +To run acceptance tests for a single connectors using the local CDK, from the connector directory, run +```bash +LOCAL_CDK=1 sh acceptance-test-docker.sh +``` +To additionally fetch secrets required by CATs, set the `FETCH_SECRETS` environment variable. This requires you to have a Google Service Account, and the GCP_GSM_CREDENTIALS environment variable to be set, per the instructions [here](https://github.com/airbytehq/airbyte/tree/b03653a24ef16be641333380f3a4d178271df0ee/tools/ci_credentials). + +##### Running Connector Acceptance Tests for multiple connectors in Docker with your local CDK installed + +To run acceptance tests for multiple connectors using the local CDK, from the root of the `airbyte` repo, run +```bash +./airbyte-cdk/python/bin/run-cats-with-local-cdk.sh -c <connector1>,<connector2>,... +``` + +#### Publishing a new version to PyPi + +1. Open a PR +2. Once it is approved and **merged**, an Airbyte member must run the `Publish CDK Manually` workflow from master using `release-type=major|manor|patch` and setting the changelog message. + +## Coming Soon + +* Full OAuth 2.0 support \(including refresh token issuing flow via UI or CLI\) +* Airbyte Java HTTP CDK +* CDK for Async HTTP endpoints \(request-poll-wait style endpoints\) +* CDK for other protocols +* Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=) + + +%package help +Summary: Development documents and examples for airbyte-cdk +Provides: python3-airbyte-cdk-doc +%description help +# Connector Development Kit \(Python\) + +The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for: + +* HTTP APIs \(REST APIs, GraphQL, etc..\) +* Singer Taps +* Generic Python sources \(anything not covered by the above\) + +The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate. + +This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding. + +## Getting Started + +Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run + +```text +cd airbyte-integrations/connector-templates/generator +./generate.sh +``` + +then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector. + +Additionally, you can follow [this tutorial](https://docs.airbyte.io/connector-development/tutorials/cdk-tutorial-python-http) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK. + +### Concepts & Documentation + +See the [concepts docs](docs/concepts/) for a tour through what the API offers. + +### Example Connectors + +**HTTP Connectors**: + +* [Exchangerates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py) +* [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py) +* [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py) + +**Singer connectors**: + +* [Salesforce](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce-singer/source_salesforce_singer/source.py) +* [Github](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github-singer/source_github_singer/source.py) + +**Simple Python connectors using the barebones `Source` abstraction**: + +* [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py) +* [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py) + +## Contributing + +### First time setup + +We assume `python` points to python >=3.8. + +Setup a virtual env: + +```text +python -m venv .venv +source .venv/bin/activate +pip install -e ".[dev]" # [dev] installs development-only dependencies +``` + +#### Iteration + +* Iterate on the code locally +* Run tests via `python -m pytest -s unit_tests` +* Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `.mypy.ini`. +* The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it! + +##### Autogenerated files +If the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run: +```commandline +SUB_BUILD=CDK ./gradlew format +``` +This will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on +pre-commit to the others, you can run the appropriate generation command i.e. `./gradlew generateComponentManifestClassFiles`. + +#### Testing + +All tests are located in the `unit_tests` directory. Run `python -m pytest --cov=airbyte_cdk unit_tests/` to run them. This also presents a test coverage report. + +#### Building and testing a connector with your local CDK + +When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways: +* Running a connector locally +* Building and running a source via Docker + +##### Installing your local CDK into a local Python connector + +In order to get a local Python connector running your local CDK, do the following. + +First, make sure you have your connector's virtual environment active: +```bash +# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory +source .venv/bin/activate + +# if you haven't installed dependencies for your connector already +pip install -e . +``` + +Then, navigate to the CDK and install it in editable mode: +```bash +cd ../../../airbyte-cdk/python +pip install -e . +``` + +You should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment. + +##### Building a Python connector in Docker with your local CDK installed + +You can build your connector image with the local CDK using +```bash +# from the airbytehq/airbyte base directory +CONNECTOR_TAG=<TAG_NAME> CONNECTOR_NAME=<CONNECTOR_NAME> sh airbyte-integrations/scripts/build-connector-image-with-local-cdk.sh +``` +Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected. + +##### Running Connector Acceptance Tests for a single connector in Docker with your local CDK installed + +To run acceptance tests for a single connectors using the local CDK, from the connector directory, run +```bash +LOCAL_CDK=1 sh acceptance-test-docker.sh +``` +To additionally fetch secrets required by CATs, set the `FETCH_SECRETS` environment variable. This requires you to have a Google Service Account, and the GCP_GSM_CREDENTIALS environment variable to be set, per the instructions [here](https://github.com/airbytehq/airbyte/tree/b03653a24ef16be641333380f3a4d178271df0ee/tools/ci_credentials). + +##### Running Connector Acceptance Tests for multiple connectors in Docker with your local CDK installed + +To run acceptance tests for multiple connectors using the local CDK, from the root of the `airbyte` repo, run +```bash +./airbyte-cdk/python/bin/run-cats-with-local-cdk.sh -c <connector1>,<connector2>,... +``` + +#### Publishing a new version to PyPi + +1. Open a PR +2. Once it is approved and **merged**, an Airbyte member must run the `Publish CDK Manually` workflow from master using `release-type=major|manor|patch` and setting the changelog message. + +## Coming Soon + +* Full OAuth 2.0 support \(including refresh token issuing flow via UI or CLI\) +* Airbyte Java HTTP CDK +* CDK for Async HTTP endpoints \(request-poll-wait style endpoints\) +* CDK for other protocols +* Don't see a feature you need? [Create an issue and let us know how we can help!](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement&template=feature-request.md&title=) + + +%prep +%autosetup -n airbyte-cdk-0.34.0 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-airbyte-cdk -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Tue Apr 11 2023 Python_Bot <Python_Bot@openeuler.org> - 0.34.0-1 +- Package Spec generated |