1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
|
%global _empty_manifest_terminate_build 0
Name: python-ibmcloudsql
Version: 0.5.13
Release: 1
Summary: Python client for interacting with IBM Cloud Data Engine service
License: Apache 2.0
URL: https://github.com/IBM-Cloud/sql-query-clients
Source0: https://mirrors.nju.edu.cn/pypi/web/packages/a5/8c/785034007f3da2e28d2819734efb978248c637000041218e696721ea3ce8/ibmcloudsql-0.5.13.tar.gz
BuildArch: noarch
%description
# ibmcloudsql
Allows you to run SQL statements in the IBM Cloud on data stored on object storage::
## Building and testing the library locally
### Set up Python environment
Run `source ./setup_env.sh` which creates and activates a clean virtual Python environment. It uses Python 2.7 by default. Adapt line 2 inside the script if you want a different version.
### Install the local code in your Python environment
Run `./_install.sh`.
### Test the library locally
1. Create a file `ibmcloudsql/test_credentials.py` with the following three lines and your according properties:
```
apikey='<your IBM Cloud API key>'
instance_crn='<your Data Engine instance CRN>'
result_location='<COS URI of default result location for your SQL result sets>'
...
```
see details in the template file
2. Run `python ibmcloudsql/test.py`.
### Packaging and publishing distribution
1. Make sure to increase `version=...` in `setup.py` before creating a new package.
2. Run `package.sh`. It will prompt for user and password that must be authorized for package `ibmcloudsql` on pypi.org.
## Example usage
```
import ibmcloudsql
my_ibmcloud_apikey = '<your api key here>'
my_instance_crn='<your ibm cloud Data Engine instance CRN here>'
my_target_cos_url='<Cloud Object Storage URL for the SQL result target. Format: cos://<endpoint>/<bucket>/[<prefix>]>'
sqlClient = SQLQuery(my_ibmcloud_apikey, my_instance_crn)
sqlClient.run_sql('SELECT * FROM cos://us-geo/sql/orders.parquet STORED AS PARQUET LIMIT 5 INTO {} STORED AS CSV'.format(my_target_cos_url)).head()
```
## Demo notebook
You can use IBM Watson Studio with the following [demo notebook](https://dataplatform.cloud.ibm.com/analytics/notebooks/v2/440b3665-367f-4fc9-86d8-4fe7eae13b18/view?access_token=3c1471a6970890fe28cadf118215df44e82c2472a83c4051e3ff80fe505448ed) that shows some elaborate examples of using various aspects of ibmcloudsql.
## SQLQuery method list
* `SQLQuery(api_key, instance_crn, target_cos_url=None, token=None, client_info='')` Constructor
* `logon(force=False, token=None)` Needs to be called before any other method below. It exchanges the `api_key` set at initialization for a temporary oauth token. The invocation is a No-Op if previous logon is less than 5 minutes ago. You can force logon anyway with optional paramater `force=True`. When you have inititialized the client without an `api_key` but instead specified a custom `token` then you can specify a fresh `token to logon method to update the client with that.
* `submit_sql(sql_text, pagesize=None)` Returns `jobId`as string. Optional pagesize parameter (in rows) for paginated result objects.
* `wait_for_job(jobId)` Waits for job to end and returns job completion state (either `completed` or `failed`)
* `get_result(jobId, pagenumber=None)` returns SQL result data frame for entire result or for specified page of results.
* `list_results(jobId)` Returns a data frame with the list of result objects written
* `delete_result(jobId)` Deletes all result set objects in cloud object storage for the given jobId
* `rename_exact_result(jobId)` Renames single partitioned query result to exact single object name without folder hierarchy.
* `get_job(jobId)` Returns details for the given SQL job as a json object
* `get_jobs()` Returns the list of recent 30 submitted SQL jobs with all details as a data frame
* `run_sql(sql_text)` Compound method that calls `submit_sql`, `wait_for_job` and `wait_for_job` in sequenceA
* `sql_ui_link()` Returns browser link for Data Engine web console for currently configured instance
* `get_cos_summary(cos_url)` Returns summary for stored number of objects and volume for a given cos url as a json
* `list_cos_objects(cos_url)` Returns a data frame with the list of objects found in the given cos url
* `export_job_history(cos_url)` Exports new jobs as parquet file to the given `cos_url`.
* `export_tags_for_cos_objects(cos_url, export_target_cos_file)` Exports all objects as a parquet file to the given `cos_url` that have tags configured along with the value for each tag.
## Exceptions
* `RateLimitedException(message)` raised when jobs can't be submitted due to 429 / Plan limit for concurrent queries has been reached
## Constructor options
* `api_key`: IAM API key. When this parameter is set to `None` then you must specify an own valid IAM otauth token in the parameter `token`.
* `instance_crn`: Data Engine instance CRN identifier
* `target_cos_url`: Optional default target URL. Don't use when you want to provide target URL in SQL statement text.
* `token`: Optional custom IAM oauth token. When you specify this then you must set `api_key` parameter to `None`.
* `client_info`: Optional string to identify your client application in IBM Cloud for PD reasons.
* `max_tries`: Optional integer to specify maximum attempts when dealing with request rate limit. Default value is `1`, which means it will through exception `RateLimitedException` when response status code is `429`. It will enable _exponential backoff_ when specifying any positive number greater than `1`. For instance, given `max_tries=5`, assuming it will get response status code `429` for 4 times until the 5th attempt will get response status code `201`, the wait time will be `2s`, `4s`, `8s` and `16s` for each attempts.
## Limitations
Data Engine Python SDK does not support Pyinstaller.
%package -n python3-ibmcloudsql
Summary: Python client for interacting with IBM Cloud Data Engine service
Provides: python-ibmcloudsql
BuildRequires: python3-devel
BuildRequires: python3-setuptools
BuildRequires: python3-pip
%description -n python3-ibmcloudsql
# ibmcloudsql
Allows you to run SQL statements in the IBM Cloud on data stored on object storage::
## Building and testing the library locally
### Set up Python environment
Run `source ./setup_env.sh` which creates and activates a clean virtual Python environment. It uses Python 2.7 by default. Adapt line 2 inside the script if you want a different version.
### Install the local code in your Python environment
Run `./_install.sh`.
### Test the library locally
1. Create a file `ibmcloudsql/test_credentials.py` with the following three lines and your according properties:
```
apikey='<your IBM Cloud API key>'
instance_crn='<your Data Engine instance CRN>'
result_location='<COS URI of default result location for your SQL result sets>'
...
```
see details in the template file
2. Run `python ibmcloudsql/test.py`.
### Packaging and publishing distribution
1. Make sure to increase `version=...` in `setup.py` before creating a new package.
2. Run `package.sh`. It will prompt for user and password that must be authorized for package `ibmcloudsql` on pypi.org.
## Example usage
```
import ibmcloudsql
my_ibmcloud_apikey = '<your api key here>'
my_instance_crn='<your ibm cloud Data Engine instance CRN here>'
my_target_cos_url='<Cloud Object Storage URL for the SQL result target. Format: cos://<endpoint>/<bucket>/[<prefix>]>'
sqlClient = SQLQuery(my_ibmcloud_apikey, my_instance_crn)
sqlClient.run_sql('SELECT * FROM cos://us-geo/sql/orders.parquet STORED AS PARQUET LIMIT 5 INTO {} STORED AS CSV'.format(my_target_cos_url)).head()
```
## Demo notebook
You can use IBM Watson Studio with the following [demo notebook](https://dataplatform.cloud.ibm.com/analytics/notebooks/v2/440b3665-367f-4fc9-86d8-4fe7eae13b18/view?access_token=3c1471a6970890fe28cadf118215df44e82c2472a83c4051e3ff80fe505448ed) that shows some elaborate examples of using various aspects of ibmcloudsql.
## SQLQuery method list
* `SQLQuery(api_key, instance_crn, target_cos_url=None, token=None, client_info='')` Constructor
* `logon(force=False, token=None)` Needs to be called before any other method below. It exchanges the `api_key` set at initialization for a temporary oauth token. The invocation is a No-Op if previous logon is less than 5 minutes ago. You can force logon anyway with optional paramater `force=True`. When you have inititialized the client without an `api_key` but instead specified a custom `token` then you can specify a fresh `token to logon method to update the client with that.
* `submit_sql(sql_text, pagesize=None)` Returns `jobId`as string. Optional pagesize parameter (in rows) for paginated result objects.
* `wait_for_job(jobId)` Waits for job to end and returns job completion state (either `completed` or `failed`)
* `get_result(jobId, pagenumber=None)` returns SQL result data frame for entire result or for specified page of results.
* `list_results(jobId)` Returns a data frame with the list of result objects written
* `delete_result(jobId)` Deletes all result set objects in cloud object storage for the given jobId
* `rename_exact_result(jobId)` Renames single partitioned query result to exact single object name without folder hierarchy.
* `get_job(jobId)` Returns details for the given SQL job as a json object
* `get_jobs()` Returns the list of recent 30 submitted SQL jobs with all details as a data frame
* `run_sql(sql_text)` Compound method that calls `submit_sql`, `wait_for_job` and `wait_for_job` in sequenceA
* `sql_ui_link()` Returns browser link for Data Engine web console for currently configured instance
* `get_cos_summary(cos_url)` Returns summary for stored number of objects and volume for a given cos url as a json
* `list_cos_objects(cos_url)` Returns a data frame with the list of objects found in the given cos url
* `export_job_history(cos_url)` Exports new jobs as parquet file to the given `cos_url`.
* `export_tags_for_cos_objects(cos_url, export_target_cos_file)` Exports all objects as a parquet file to the given `cos_url` that have tags configured along with the value for each tag.
## Exceptions
* `RateLimitedException(message)` raised when jobs can't be submitted due to 429 / Plan limit for concurrent queries has been reached
## Constructor options
* `api_key`: IAM API key. When this parameter is set to `None` then you must specify an own valid IAM otauth token in the parameter `token`.
* `instance_crn`: Data Engine instance CRN identifier
* `target_cos_url`: Optional default target URL. Don't use when you want to provide target URL in SQL statement text.
* `token`: Optional custom IAM oauth token. When you specify this then you must set `api_key` parameter to `None`.
* `client_info`: Optional string to identify your client application in IBM Cloud for PD reasons.
* `max_tries`: Optional integer to specify maximum attempts when dealing with request rate limit. Default value is `1`, which means it will through exception `RateLimitedException` when response status code is `429`. It will enable _exponential backoff_ when specifying any positive number greater than `1`. For instance, given `max_tries=5`, assuming it will get response status code `429` for 4 times until the 5th attempt will get response status code `201`, the wait time will be `2s`, `4s`, `8s` and `16s` for each attempts.
## Limitations
Data Engine Python SDK does not support Pyinstaller.
%package help
Summary: Development documents and examples for ibmcloudsql
Provides: python3-ibmcloudsql-doc
%description help
# ibmcloudsql
Allows you to run SQL statements in the IBM Cloud on data stored on object storage::
## Building and testing the library locally
### Set up Python environment
Run `source ./setup_env.sh` which creates and activates a clean virtual Python environment. It uses Python 2.7 by default. Adapt line 2 inside the script if you want a different version.
### Install the local code in your Python environment
Run `./_install.sh`.
### Test the library locally
1. Create a file `ibmcloudsql/test_credentials.py` with the following three lines and your according properties:
```
apikey='<your IBM Cloud API key>'
instance_crn='<your Data Engine instance CRN>'
result_location='<COS URI of default result location for your SQL result sets>'
...
```
see details in the template file
2. Run `python ibmcloudsql/test.py`.
### Packaging and publishing distribution
1. Make sure to increase `version=...` in `setup.py` before creating a new package.
2. Run `package.sh`. It will prompt for user and password that must be authorized for package `ibmcloudsql` on pypi.org.
## Example usage
```
import ibmcloudsql
my_ibmcloud_apikey = '<your api key here>'
my_instance_crn='<your ibm cloud Data Engine instance CRN here>'
my_target_cos_url='<Cloud Object Storage URL for the SQL result target. Format: cos://<endpoint>/<bucket>/[<prefix>]>'
sqlClient = SQLQuery(my_ibmcloud_apikey, my_instance_crn)
sqlClient.run_sql('SELECT * FROM cos://us-geo/sql/orders.parquet STORED AS PARQUET LIMIT 5 INTO {} STORED AS CSV'.format(my_target_cos_url)).head()
```
## Demo notebook
You can use IBM Watson Studio with the following [demo notebook](https://dataplatform.cloud.ibm.com/analytics/notebooks/v2/440b3665-367f-4fc9-86d8-4fe7eae13b18/view?access_token=3c1471a6970890fe28cadf118215df44e82c2472a83c4051e3ff80fe505448ed) that shows some elaborate examples of using various aspects of ibmcloudsql.
## SQLQuery method list
* `SQLQuery(api_key, instance_crn, target_cos_url=None, token=None, client_info='')` Constructor
* `logon(force=False, token=None)` Needs to be called before any other method below. It exchanges the `api_key` set at initialization for a temporary oauth token. The invocation is a No-Op if previous logon is less than 5 minutes ago. You can force logon anyway with optional paramater `force=True`. When you have inititialized the client without an `api_key` but instead specified a custom `token` then you can specify a fresh `token to logon method to update the client with that.
* `submit_sql(sql_text, pagesize=None)` Returns `jobId`as string. Optional pagesize parameter (in rows) for paginated result objects.
* `wait_for_job(jobId)` Waits for job to end and returns job completion state (either `completed` or `failed`)
* `get_result(jobId, pagenumber=None)` returns SQL result data frame for entire result or for specified page of results.
* `list_results(jobId)` Returns a data frame with the list of result objects written
* `delete_result(jobId)` Deletes all result set objects in cloud object storage for the given jobId
* `rename_exact_result(jobId)` Renames single partitioned query result to exact single object name without folder hierarchy.
* `get_job(jobId)` Returns details for the given SQL job as a json object
* `get_jobs()` Returns the list of recent 30 submitted SQL jobs with all details as a data frame
* `run_sql(sql_text)` Compound method that calls `submit_sql`, `wait_for_job` and `wait_for_job` in sequenceA
* `sql_ui_link()` Returns browser link for Data Engine web console for currently configured instance
* `get_cos_summary(cos_url)` Returns summary for stored number of objects and volume for a given cos url as a json
* `list_cos_objects(cos_url)` Returns a data frame with the list of objects found in the given cos url
* `export_job_history(cos_url)` Exports new jobs as parquet file to the given `cos_url`.
* `export_tags_for_cos_objects(cos_url, export_target_cos_file)` Exports all objects as a parquet file to the given `cos_url` that have tags configured along with the value for each tag.
## Exceptions
* `RateLimitedException(message)` raised when jobs can't be submitted due to 429 / Plan limit for concurrent queries has been reached
## Constructor options
* `api_key`: IAM API key. When this parameter is set to `None` then you must specify an own valid IAM otauth token in the parameter `token`.
* `instance_crn`: Data Engine instance CRN identifier
* `target_cos_url`: Optional default target URL. Don't use when you want to provide target URL in SQL statement text.
* `token`: Optional custom IAM oauth token. When you specify this then you must set `api_key` parameter to `None`.
* `client_info`: Optional string to identify your client application in IBM Cloud for PD reasons.
* `max_tries`: Optional integer to specify maximum attempts when dealing with request rate limit. Default value is `1`, which means it will through exception `RateLimitedException` when response status code is `429`. It will enable _exponential backoff_ when specifying any positive number greater than `1`. For instance, given `max_tries=5`, assuming it will get response status code `429` for 4 times until the 5th attempt will get response status code `201`, the wait time will be `2s`, `4s`, `8s` and `16s` for each attempts.
## Limitations
Data Engine Python SDK does not support Pyinstaller.
%prep
%autosetup -n ibmcloudsql-0.5.13
%build
%py3_build
%install
%py3_install
install -d -m755 %{buildroot}/%{_pkgdocdir}
if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
pushd %{buildroot}
if [ -d usr/lib ]; then
find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/lib64 ]; then
find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/bin ]; then
find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/sbin ]; then
find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
fi
touch doclist.lst
if [ -d usr/share/man ]; then
find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
fi
popd
mv %{buildroot}/filelist.lst .
mv %{buildroot}/doclist.lst .
%files -n python3-ibmcloudsql -f filelist.lst
%dir %{python3_sitelib}/*
%files help -f doclist.lst
%{_docdir}/*
%changelog
* Fri May 05 2023 Python_Bot <Python_Bot@openeuler.org> - 0.5.13-1
- Package Spec generated
|
