automatic import of python-dbnd-airflow-monitor

3 files changed, 344 insertions, 0 deletions

diff --git a/.gitignore b/.gitignore
index e69de29..9d798ef 100644
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1 @@
+/dbnd-airflow-monitor-1.0.12.12.tar.gz
diff --git a/python-dbnd-airflow-monitor.spec b/python-dbnd-airflow-monitor.spec
new file mode 100644
index 0000000..4f46c67
--- /dev/null
+++ b/python-dbnd-airflow-monitor.spec
@@ -0,0 +1,342 @@
+%global _empty_manifest_terminate_build 0
+Name:		python-dbnd-airflow-monitor
+Version:	1.0.12.12
+Release:	1
+Summary:	Machine Learning Orchestration
+License:	Apache Software License
+URL:		https://github.com/databand-ai/dbnd
+Source0:	https://mirrors.nju.edu.cn/pypi/web/packages/34/82/528fd10a8a5639e6a1eb6a0d9ecec79300ffe4ab716fad22958b87703835/dbnd-airflow-monitor-1.0.12.12.tar.gz
+BuildArch:	noarch
+
+Requires:	python3-dbnd
+Requires:	python3-setuptools
+Requires:	python3-prometheus-client
+Requires:	python3-PyJWT
+Requires:	python3-cryptography
+Requires:	python3-google-auth
+Requires:	python3-requests
+Requires:	python3-requests-toolbelt
+Requires:	python3-tzlocal
+Requires:	python3-pytest
+Requires:	python3-mock
+Requires:	python3-sh
+
+%description
+# Databand Airflow Monitor
+
+Databand Airflow Monitor is a stand-alone module for Databand system, enables you to load data from Airflow server and import it into Databand system.
+This Databand side module is one of two components allows you to sync your Airflow data into Databand system.
+
+## Installation with setup tools
+
+```bash
+cd modules/dbnd-airflow-monitor
+pip install -e .
+```
+
+## Usage
+
+`dbnd airflow-monitor`
+
+### Important flags
+
+`--sync-history`: by default, airflow monitor's `since` value will be determined by last time it was running. use this flag to enable syncning from beginning
+
+### Configuration
+
+You can configure your syncing variables inside databand configuration system
+
+```cfg
+[airflow_monitor]
+interval = 10 ; Time in seconds to wait between each fetching cycle
+include_logs = True ; Whether or not to include logs (might be heavy)
+include_task_args = True ; Whether or not to include task arguments
+fetch_quantity = 100 ; Max number of tasks or dag runs to retrieve at each fetch
+fetch_period = 60 ; Time in minutes for window fetching size (start: since, end: since + period)
+dag_ids = ['ingest_data_dag', 'simple_dag'] ; Limit fetching to these specific dag ids
+
+## DB Fetcher
+### Pay attention, when using this system airflow version must be equal to databand's airflow version
+sql_alchemy_conn = sqlite:////usr/local/airflow/airflow.db ; When using fetcher=db, use this sql connection string
+local_dag_folder =  /usr/local/airflow/dags ; When using fetcher=db, this is the dag folder location
+```
+
+## Steps for Google Composer
+
+​
+After spinning new google composer, under PyPi packages add dbnd, and add `DBND__CORE__DATABAND_URL` env pointing to dnbd instance, copy plugin file to pluings folder (go to dags folder, one level up, and then plugins)
+​
+​
+For monitor to work you will need to setup service account (add relevant binding):
+(taken from here: https://medium.com/google-cloud/using-airflow-experimental-rest-api-on-google-cloud-platform-cloud-composer-and-iap-9bd0260f095a
+see Create a Service Account for POST Trigger section)
+​
+example with creating new SA:
+
+```bash
+export PROJECT=prefab-root-227507
+export SERVICE_ACCOUNT_NAME=dbnd-airflow-monitor
+gcloud iam service-accounts create $SERVICE_ACCOUNT_NAME --project $PROJECT
+# Give service account permissions to create tokens for iap requests.
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/iam.serviceAccountTokenCreator
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/iam.serviceAccountActor
+# Service account also needs to be authorized to use Composer.
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/composer.user
+# We need a service account key to trigger the dag.
+gcloud iam service-accounts keys create ~/$PROJECT-$SERVICE_ACCOUNT_NAME-key.json --iam-account=$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com
+export GOOGLE_APPLICATION_CREDENTIALS=~/$PROJECT-$SERVICE_ACCOUNT_NAME-key.json
+```
+
+​
+configure airflow monitor with composer fetcher, with url pointing to composer airflow instance and client id (same article, Getting Airflow Client ID section):
+Visit the Airflow URL https://YOUR_UNIQUE_ID.appspot.com (which you noted in the last step) in an incognito window, don’t login. At this first landing page for IAP Auth has client id in the url in the address bar:
+
+```
+https://accounts.google.com/signin/oauth/identifier?client_id=00000000000-xxxx0x0xx0xx00xxxx0x00xxx0xxxxx.apps.googleusercontent.com&...
+```
+
+## Integration Tests
+
+We have 2 tests:
+
+-   databand/integration-tests/airflow_monitor
+-   databand/integration-tests/airflow_monitor_stress
+
+To run them, go to the right dir and run inttest container:
+
+```
+cd databand/integration-tests/airflow_monitor
+docker-compose up inttest
+```
+
+
+%package -n python3-dbnd-airflow-monitor
+Summary:	Machine Learning Orchestration
+Provides:	python-dbnd-airflow-monitor
+BuildRequires:	python3-devel
+BuildRequires:	python3-setuptools
+BuildRequires:	python3-pip
+%description -n python3-dbnd-airflow-monitor
+# Databand Airflow Monitor
+
+Databand Airflow Monitor is a stand-alone module for Databand system, enables you to load data from Airflow server and import it into Databand system.
+This Databand side module is one of two components allows you to sync your Airflow data into Databand system.
+
+## Installation with setup tools
+
+```bash
+cd modules/dbnd-airflow-monitor
+pip install -e .
+```
+
+## Usage
+
+`dbnd airflow-monitor`
+
+### Important flags
+
+`--sync-history`: by default, airflow monitor's `since` value will be determined by last time it was running. use this flag to enable syncning from beginning
+
+### Configuration
+
+You can configure your syncing variables inside databand configuration system
+
+```cfg
+[airflow_monitor]
+interval = 10 ; Time in seconds to wait between each fetching cycle
+include_logs = True ; Whether or not to include logs (might be heavy)
+include_task_args = True ; Whether or not to include task arguments
+fetch_quantity = 100 ; Max number of tasks or dag runs to retrieve at each fetch
+fetch_period = 60 ; Time in minutes for window fetching size (start: since, end: since + period)
+dag_ids = ['ingest_data_dag', 'simple_dag'] ; Limit fetching to these specific dag ids
+
+## DB Fetcher
+### Pay attention, when using this system airflow version must be equal to databand's airflow version
+sql_alchemy_conn = sqlite:////usr/local/airflow/airflow.db ; When using fetcher=db, use this sql connection string
+local_dag_folder =  /usr/local/airflow/dags ; When using fetcher=db, this is the dag folder location
+```
+
+## Steps for Google Composer
+
+​
+After spinning new google composer, under PyPi packages add dbnd, and add `DBND__CORE__DATABAND_URL` env pointing to dnbd instance, copy plugin file to pluings folder (go to dags folder, one level up, and then plugins)
+​
+​
+For monitor to work you will need to setup service account (add relevant binding):
+(taken from here: https://medium.com/google-cloud/using-airflow-experimental-rest-api-on-google-cloud-platform-cloud-composer-and-iap-9bd0260f095a
+see Create a Service Account for POST Trigger section)
+​
+example with creating new SA:
+
+```bash
+export PROJECT=prefab-root-227507
+export SERVICE_ACCOUNT_NAME=dbnd-airflow-monitor
+gcloud iam service-accounts create $SERVICE_ACCOUNT_NAME --project $PROJECT
+# Give service account permissions to create tokens for iap requests.
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/iam.serviceAccountTokenCreator
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/iam.serviceAccountActor
+# Service account also needs to be authorized to use Composer.
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/composer.user
+# We need a service account key to trigger the dag.
+gcloud iam service-accounts keys create ~/$PROJECT-$SERVICE_ACCOUNT_NAME-key.json --iam-account=$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com
+export GOOGLE_APPLICATION_CREDENTIALS=~/$PROJECT-$SERVICE_ACCOUNT_NAME-key.json
+```
+
+​
+configure airflow monitor with composer fetcher, with url pointing to composer airflow instance and client id (same article, Getting Airflow Client ID section):
+Visit the Airflow URL https://YOUR_UNIQUE_ID.appspot.com (which you noted in the last step) in an incognito window, don’t login. At this first landing page for IAP Auth has client id in the url in the address bar:
+
+```
+https://accounts.google.com/signin/oauth/identifier?client_id=00000000000-xxxx0x0xx0xx00xxxx0x00xxx0xxxxx.apps.googleusercontent.com&...
+```
+
+## Integration Tests
+
+We have 2 tests:
+
+-   databand/integration-tests/airflow_monitor
+-   databand/integration-tests/airflow_monitor_stress
+
+To run them, go to the right dir and run inttest container:
+
+```
+cd databand/integration-tests/airflow_monitor
+docker-compose up inttest
+```
+
+
+%package help
+Summary:	Development documents and examples for dbnd-airflow-monitor
+Provides:	python3-dbnd-airflow-monitor-doc
+%description help
+# Databand Airflow Monitor
+
+Databand Airflow Monitor is a stand-alone module for Databand system, enables you to load data from Airflow server and import it into Databand system.
+This Databand side module is one of two components allows you to sync your Airflow data into Databand system.
+
+## Installation with setup tools
+
+```bash
+cd modules/dbnd-airflow-monitor
+pip install -e .
+```
+
+## Usage
+
+`dbnd airflow-monitor`
+
+### Important flags
+
+`--sync-history`: by default, airflow monitor's `since` value will be determined by last time it was running. use this flag to enable syncning from beginning
+
+### Configuration
+
+You can configure your syncing variables inside databand configuration system
+
+```cfg
+[airflow_monitor]
+interval = 10 ; Time in seconds to wait between each fetching cycle
+include_logs = True ; Whether or not to include logs (might be heavy)
+include_task_args = True ; Whether or not to include task arguments
+fetch_quantity = 100 ; Max number of tasks or dag runs to retrieve at each fetch
+fetch_period = 60 ; Time in minutes for window fetching size (start: since, end: since + period)
+dag_ids = ['ingest_data_dag', 'simple_dag'] ; Limit fetching to these specific dag ids
+
+## DB Fetcher
+### Pay attention, when using this system airflow version must be equal to databand's airflow version
+sql_alchemy_conn = sqlite:////usr/local/airflow/airflow.db ; When using fetcher=db, use this sql connection string
+local_dag_folder =  /usr/local/airflow/dags ; When using fetcher=db, this is the dag folder location
+```
+
+## Steps for Google Composer
+
+​
+After spinning new google composer, under PyPi packages add dbnd, and add `DBND__CORE__DATABAND_URL` env pointing to dnbd instance, copy plugin file to pluings folder (go to dags folder, one level up, and then plugins)
+​
+​
+For monitor to work you will need to setup service account (add relevant binding):
+(taken from here: https://medium.com/google-cloud/using-airflow-experimental-rest-api-on-google-cloud-platform-cloud-composer-and-iap-9bd0260f095a
+see Create a Service Account for POST Trigger section)
+​
+example with creating new SA:
+
+```bash
+export PROJECT=prefab-root-227507
+export SERVICE_ACCOUNT_NAME=dbnd-airflow-monitor
+gcloud iam service-accounts create $SERVICE_ACCOUNT_NAME --project $PROJECT
+# Give service account permissions to create tokens for iap requests.
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/iam.serviceAccountTokenCreator
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/iam.serviceAccountActor
+# Service account also needs to be authorized to use Composer.
+gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com --role roles/composer.user
+# We need a service account key to trigger the dag.
+gcloud iam service-accounts keys create ~/$PROJECT-$SERVICE_ACCOUNT_NAME-key.json --iam-account=$SERVICE_ACCOUNT_NAME@$PROJECT.iam.gserviceaccount.com
+export GOOGLE_APPLICATION_CREDENTIALS=~/$PROJECT-$SERVICE_ACCOUNT_NAME-key.json
+```
+
+​
+configure airflow monitor with composer fetcher, with url pointing to composer airflow instance and client id (same article, Getting Airflow Client ID section):
+Visit the Airflow URL https://YOUR_UNIQUE_ID.appspot.com (which you noted in the last step) in an incognito window, don’t login. At this first landing page for IAP Auth has client id in the url in the address bar:
+
+```
+https://accounts.google.com/signin/oauth/identifier?client_id=00000000000-xxxx0x0xx0xx00xxxx0x00xxx0xxxxx.apps.googleusercontent.com&...
+```
+
+## Integration Tests
+
+We have 2 tests:
+
+-   databand/integration-tests/airflow_monitor
+-   databand/integration-tests/airflow_monitor_stress
+
+To run them, go to the right dir and run inttest container:
+
+```
+cd databand/integration-tests/airflow_monitor
+docker-compose up inttest
+```
+
+
+%prep
+%autosetup -n dbnd-airflow-monitor-1.0.12.12
+
+%build
+%py3_build
+
+%install
+%py3_install
+install -d -m755 %{buildroot}/%{_pkgdocdir}
+if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
+if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
+if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
+if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
+pushd %{buildroot}
+if [ -d usr/lib ]; then
+	find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+if [ -d usr/lib64 ]; then
+	find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+if [ -d usr/bin ]; then
+	find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+if [ -d usr/sbin ]; then
+	find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
+fi
+touch doclist.lst
+if [ -d usr/share/man ]; then
+	find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
+fi
+popd
+mv %{buildroot}/filelist.lst .
+mv %{buildroot}/doclist.lst .
+
+%files -n python3-dbnd-airflow-monitor -f filelist.lst
+%dir %{python3_sitelib}/*
+
+%files help -f doclist.lst
+%{_docdir}/*
+
+%changelog
+* Wed Apr 12 2023 Python_Bot <Python_Bot@openeuler.org> - 1.0.12.12-1
+- Package Spec generated
diff --git a/sources b/sources
new file mode 100644
index 0000000..3de967e
--- /dev/null
+++ b/sources
@@ -0,0 +1 @@
+ae3d1667b878ac8385bef500f073b1b8  dbnd-airflow-monitor-1.0.12.12.tar.gz