1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
|
%global _empty_manifest_terminate_build 0
Name: python-tdcsm
Version: 0.4.1.6
Release: 1
Summary: Teradata tools for CSMs (deprecated)
License: MIT License
URL: https://github.com/tdcoa/tdcsm
Source0: https://mirrors.aliyun.com/pypi/web/packages/9d/e9/529143b6b657f00dccfd47689531796208db4f09942fecb641f6d6c6eb79/tdcsm-0.4.1.6.tar.gz
BuildArch: noarch
Requires: python3-pandas
Requires: python3-numpy
Requires: python3-requests
Requires: python3-pyyaml
Requires: python3-teradatasql
Requires: python3-matplotlib
Requires: python3-seaborn
Requires: python3-pptx
Requires: python3-pydantic
Requires: python3-sklearn
Requires: python3-Pillow
Requires: python3-tk
%description
This library will perform 4 descrete steps:
(1) DOWNLOAD sql and csv from github respositories,
(2) PREPARE sql locally, including variable substitutions and csv merging,
(3) EXECUTE the prepared sql against customer site ids, and export any
indicated data sets to csv, then finally
(4) UPLOAD specifically indicated csv to Transcend, and call named stored procs
to merge uploaded temp tables with final target tables.
Each step is designed to be autonomous and can be run independently, assuming
all dependencies are met.
This allows CSMs to download and prepare SQL first, insepct and make manual
changes to reflect the particular needs of their customer, all on their local PC.
Once happy with the script, they can move the process over and execute on a
customer-owned laptop, where the process should pick-up seemlessly. Continuing
on, the CSM could execute on the customer's TD system, export CSVs, and move
back to the CSM's laptop where results can be uploaded to Transcend.
Sample Usage:
1 from tdcoa import tdcoa
2 coa = tdcoa()
3 coa.download_files()
4 coa.prepare_sql()
5 coa.execute_run()
6 coa.upload_to_transcend()
what stuff does, by line (python 3.6+):
Line 1 = import the class. Pretty standard python stuff.
Line 2 = instantiate the class. This will also setup the local running environment
in the same directory as your calling script (by default, but you can supply
an alternate path as a parameter). If they are missing, it will also create
default files such as secrets.yaml and config.yaml, which are critical for
the process. It will NOT overwrite existing files.
** if you are running for the first time, it is recommended you run just
line 1 & 2 first, and modify secrets.yaml and config.yaml as needed.
This makes subsequent tests more interesting, as credentials will work.
Line 3 = download any missing files or sql from GitHub. URL and file inventory are
both stored in the config.yaml. the 0000.*.sql files are example files.
While line 3 only needs to be run once, sql *will* be overwritten with
newer content, so it is recommended you update periodically.
Line 4 = iterates all .coa.sql and .csv files in in the 'sql' folder and preapres
the sql for later execution. This step includes several sustitution
steps: from secrets.yaml (never printed in logs), from config.yaml in
the substitution section, and from any embedded /*{{loop:myfile.csv}}
command. In the last case, the csv is opened and the process generates
one new sql per row in the file, substituting {{column_name}} with the
value for that row. All of these sql scripts are written out to the
'run' directory as defined in the config.yaml, awaiting the next step.
Line 5 = iterates thru all site id connection strings first, then thru all sql
files found in the 'run' directory *in alpha order*, and executes the sql.
all substitutions are done in the previous step, so besides secrets.yaml
replacements and certain runtime values like {{siteid}}, the sql will be
exactly what is run. There are three special commands, formatted in
comments:
/*{{save:MyFile.csv}}*/ = save the sql results to the named csv
/*{{load:db.Tablename}}*/ = load the above csv to the Transcend table
/*{{call:db.StoredProc}}*/ = call the stored proc after above load
The first special command will execute during step 5, exporting data from
each siteid. Note that it is possible to use {{substitution}} in these
commands, so /*{{save:{{siteid}}_export.csv}}*/ is valid, and will
produce a different csv for each site id you run against. Without this,
the process might run against many siteids, but each would overwrite the
same filename.
The last two special commands, load and call, are intended to run against
transcend, and so in step five are only written to 'upload_manifest.json'
and saved in the same 'output' folder, awaiting the next and final step.
Line 6 = for each line in the 'upload_manifest.json', perform the requested load
and subsequent stored proc call. The intention is to upload csv files
into Transcend Global Temporary Tables, then initiate a stored procedure
to complete whatever cleansing, validation, and data movement is required
to merge into the final target table. This keeps most of the business
logic as close to the data as possible.
DEBUGGING:
Missing tdcoa -- if you get an error stating you're missing tdcoa, then first, I'm
unclear on how you're reading this text. That aside, open up a command prompt and type:
pip install tdcoa
if that gives you errors, then maybe this wasn't meant to be. Call Stephen Hilton.
%package -n python3-tdcsm
Summary: Teradata tools for CSMs (deprecated)
Provides: python-tdcsm
BuildRequires: python3-devel
BuildRequires: python3-setuptools
BuildRequires: python3-pip
%description -n python3-tdcsm
This library will perform 4 descrete steps:
(1) DOWNLOAD sql and csv from github respositories,
(2) PREPARE sql locally, including variable substitutions and csv merging,
(3) EXECUTE the prepared sql against customer site ids, and export any
indicated data sets to csv, then finally
(4) UPLOAD specifically indicated csv to Transcend, and call named stored procs
to merge uploaded temp tables with final target tables.
Each step is designed to be autonomous and can be run independently, assuming
all dependencies are met.
This allows CSMs to download and prepare SQL first, insepct and make manual
changes to reflect the particular needs of their customer, all on their local PC.
Once happy with the script, they can move the process over and execute on a
customer-owned laptop, where the process should pick-up seemlessly. Continuing
on, the CSM could execute on the customer's TD system, export CSVs, and move
back to the CSM's laptop where results can be uploaded to Transcend.
Sample Usage:
1 from tdcoa import tdcoa
2 coa = tdcoa()
3 coa.download_files()
4 coa.prepare_sql()
5 coa.execute_run()
6 coa.upload_to_transcend()
what stuff does, by line (python 3.6+):
Line 1 = import the class. Pretty standard python stuff.
Line 2 = instantiate the class. This will also setup the local running environment
in the same directory as your calling script (by default, but you can supply
an alternate path as a parameter). If they are missing, it will also create
default files such as secrets.yaml and config.yaml, which are critical for
the process. It will NOT overwrite existing files.
** if you are running for the first time, it is recommended you run just
line 1 & 2 first, and modify secrets.yaml and config.yaml as needed.
This makes subsequent tests more interesting, as credentials will work.
Line 3 = download any missing files or sql from GitHub. URL and file inventory are
both stored in the config.yaml. the 0000.*.sql files are example files.
While line 3 only needs to be run once, sql *will* be overwritten with
newer content, so it is recommended you update periodically.
Line 4 = iterates all .coa.sql and .csv files in in the 'sql' folder and preapres
the sql for later execution. This step includes several sustitution
steps: from secrets.yaml (never printed in logs), from config.yaml in
the substitution section, and from any embedded /*{{loop:myfile.csv}}
command. In the last case, the csv is opened and the process generates
one new sql per row in the file, substituting {{column_name}} with the
value for that row. All of these sql scripts are written out to the
'run' directory as defined in the config.yaml, awaiting the next step.
Line 5 = iterates thru all site id connection strings first, then thru all sql
files found in the 'run' directory *in alpha order*, and executes the sql.
all substitutions are done in the previous step, so besides secrets.yaml
replacements and certain runtime values like {{siteid}}, the sql will be
exactly what is run. There are three special commands, formatted in
comments:
/*{{save:MyFile.csv}}*/ = save the sql results to the named csv
/*{{load:db.Tablename}}*/ = load the above csv to the Transcend table
/*{{call:db.StoredProc}}*/ = call the stored proc after above load
The first special command will execute during step 5, exporting data from
each siteid. Note that it is possible to use {{substitution}} in these
commands, so /*{{save:{{siteid}}_export.csv}}*/ is valid, and will
produce a different csv for each site id you run against. Without this,
the process might run against many siteids, but each would overwrite the
same filename.
The last two special commands, load and call, are intended to run against
transcend, and so in step five are only written to 'upload_manifest.json'
and saved in the same 'output' folder, awaiting the next and final step.
Line 6 = for each line in the 'upload_manifest.json', perform the requested load
and subsequent stored proc call. The intention is to upload csv files
into Transcend Global Temporary Tables, then initiate a stored procedure
to complete whatever cleansing, validation, and data movement is required
to merge into the final target table. This keeps most of the business
logic as close to the data as possible.
DEBUGGING:
Missing tdcoa -- if you get an error stating you're missing tdcoa, then first, I'm
unclear on how you're reading this text. That aside, open up a command prompt and type:
pip install tdcoa
if that gives you errors, then maybe this wasn't meant to be. Call Stephen Hilton.
%package help
Summary: Development documents and examples for tdcsm
Provides: python3-tdcsm-doc
%description help
This library will perform 4 descrete steps:
(1) DOWNLOAD sql and csv from github respositories,
(2) PREPARE sql locally, including variable substitutions and csv merging,
(3) EXECUTE the prepared sql against customer site ids, and export any
indicated data sets to csv, then finally
(4) UPLOAD specifically indicated csv to Transcend, and call named stored procs
to merge uploaded temp tables with final target tables.
Each step is designed to be autonomous and can be run independently, assuming
all dependencies are met.
This allows CSMs to download and prepare SQL first, insepct and make manual
changes to reflect the particular needs of their customer, all on their local PC.
Once happy with the script, they can move the process over and execute on a
customer-owned laptop, where the process should pick-up seemlessly. Continuing
on, the CSM could execute on the customer's TD system, export CSVs, and move
back to the CSM's laptop where results can be uploaded to Transcend.
Sample Usage:
1 from tdcoa import tdcoa
2 coa = tdcoa()
3 coa.download_files()
4 coa.prepare_sql()
5 coa.execute_run()
6 coa.upload_to_transcend()
what stuff does, by line (python 3.6+):
Line 1 = import the class. Pretty standard python stuff.
Line 2 = instantiate the class. This will also setup the local running environment
in the same directory as your calling script (by default, but you can supply
an alternate path as a parameter). If they are missing, it will also create
default files such as secrets.yaml and config.yaml, which are critical for
the process. It will NOT overwrite existing files.
** if you are running for the first time, it is recommended you run just
line 1 & 2 first, and modify secrets.yaml and config.yaml as needed.
This makes subsequent tests more interesting, as credentials will work.
Line 3 = download any missing files or sql from GitHub. URL and file inventory are
both stored in the config.yaml. the 0000.*.sql files are example files.
While line 3 only needs to be run once, sql *will* be overwritten with
newer content, so it is recommended you update periodically.
Line 4 = iterates all .coa.sql and .csv files in in the 'sql' folder and preapres
the sql for later execution. This step includes several sustitution
steps: from secrets.yaml (never printed in logs), from config.yaml in
the substitution section, and from any embedded /*{{loop:myfile.csv}}
command. In the last case, the csv is opened and the process generates
one new sql per row in the file, substituting {{column_name}} with the
value for that row. All of these sql scripts are written out to the
'run' directory as defined in the config.yaml, awaiting the next step.
Line 5 = iterates thru all site id connection strings first, then thru all sql
files found in the 'run' directory *in alpha order*, and executes the sql.
all substitutions are done in the previous step, so besides secrets.yaml
replacements and certain runtime values like {{siteid}}, the sql will be
exactly what is run. There are three special commands, formatted in
comments:
/*{{save:MyFile.csv}}*/ = save the sql results to the named csv
/*{{load:db.Tablename}}*/ = load the above csv to the Transcend table
/*{{call:db.StoredProc}}*/ = call the stored proc after above load
The first special command will execute during step 5, exporting data from
each siteid. Note that it is possible to use {{substitution}} in these
commands, so /*{{save:{{siteid}}_export.csv}}*/ is valid, and will
produce a different csv for each site id you run against. Without this,
the process might run against many siteids, but each would overwrite the
same filename.
The last two special commands, load and call, are intended to run against
transcend, and so in step five are only written to 'upload_manifest.json'
and saved in the same 'output' folder, awaiting the next and final step.
Line 6 = for each line in the 'upload_manifest.json', perform the requested load
and subsequent stored proc call. The intention is to upload csv files
into Transcend Global Temporary Tables, then initiate a stored procedure
to complete whatever cleansing, validation, and data movement is required
to merge into the final target table. This keeps most of the business
logic as close to the data as possible.
DEBUGGING:
Missing tdcoa -- if you get an error stating you're missing tdcoa, then first, I'm
unclear on how you're reading this text. That aside, open up a command prompt and type:
pip install tdcoa
if that gives you errors, then maybe this wasn't meant to be. Call Stephen Hilton.
%prep
%autosetup -n tdcsm-0.4.1.6
%build
%py3_build
%install
%py3_install
install -d -m755 %{buildroot}/%{_pkgdocdir}
if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
pushd %{buildroot}
if [ -d usr/lib ]; then
find usr/lib -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
if [ -d usr/lib64 ]; then
find usr/lib64 -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
if [ -d usr/bin ]; then
find usr/bin -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
if [ -d usr/sbin ]; then
find usr/sbin -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
touch doclist.lst
if [ -d usr/share/man ]; then
find usr/share/man -type f -printf "\"/%h/%f.gz\"\n" >> doclist.lst
fi
popd
mv %{buildroot}/filelist.lst .
mv %{buildroot}/doclist.lst .
%files -n python3-tdcsm -f filelist.lst
%dir %{python3_sitelib}/*
%files help -f doclist.lst
%{_docdir}/*
%changelog
* Thu Jun 08 2023 Python_Bot <Python_Bot@openeuler.org> - 0.4.1.6-1
- Package Spec generated
|
