1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
|
%global _empty_manifest_terminate_build 0
Name: python-dsub
Version: 0.4.9
Release: 1
Summary: A command-line tool that makes it easy to submit and run batch scripts in the cloud
License: Apache
URL: https://github.com/DataBiosphere/dsub
Source0: https://mirrors.aliyun.com/pypi/web/packages/7a/36/6e2013482f87a93f4334525435c6a8e1050b028899d9054050bf62d2d23c/dsub-0.4.9.tar.gz
BuildArch: noarch
Requires: python3-funcsigs
Requires: python3-google-api-core
Requires: python3-google-api-python-client
Requires: python3-google-auth-httplib2
Requires: python3-google-auth
Requires: python3-google-cloud-batch
Requires: python3-httplib2
Requires: python3-mock
Requires: python3-parameterized
Requires: python3-pyasn1-modules
Requires: python3-pyasn1
Requires: python3-dateutil
Requires: python3-pytz
Requires: python3-pyyaml
Requires: python3-rsa
Requires: python3-tabulate
Requires: python3-tenacity
Requires: python3-uritemplate
%description
my-job-name RUNNING 2
my-job-name SUCCESS 1
```
In this mode, dstat prints one line per (job name, task status) pair. You can
see at a glance how many tasks are finished, how many are still running, and
how many are failed/canceled.
### Deleting a job
The `ddel` command will delete running jobs.
By default, only jobs submitted by the current user will be deleted.
Use the `--users` flag to specify other users, or `'*'` for all users.
To delete a running job:
ddel --provider google-v2 --project my-cloud-project --jobs job-id
If the job is a batch job, all running tasks will be deleted.
To delete specific tasks:
ddel \
--provider google-v2 \
--project my-cloud-project \
--jobs job-id \
--tasks task-id1 task-id2
To delete all running jobs for the current user:
ddel --provider google-v2 --project my-cloud-project --jobs '*'
## Service Accounts and Scope (Google providers only)
When you run the `dsub` command with the `google-v2` or `google-cls-v2`
provider, there are two different sets of credentials to consider:
- Account submitting the `pipelines.run()` request to run your command/script on a VM
- Account accessing Cloud resources (such as files in GCS) when executing your command/script
The account used to submit the `pipelines.run()` request is typically your
end user credentials. You would have set this up by running:
gcloud auth application-default login
The account used on the VM is a [service account](https://cloud.google.com/iam/docs/service-accounts).
The image below illustrates this:
By default, `dsub` will use the [default Compute Engine service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account)
as the authorized service account on the VM instance. You can choose to specify
the email address of another service acount using `--service-account`.
By default, `dsub` will grant the following access scopes to the service account:
- https://www.googleapis.com/auth/bigquery
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.full_control
- https://www.googleapis.com/auth/genomics
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
In addition, [the API](https://cloud.google.com/life-sciences/docs/reference/rest/v2beta/projects.locations.pipelines/run#serviceaccount) will always add this scope:
- https://www.googleapis.com/auth/cloud-platform
You can choose to specify scopes using `--scopes`.
### Recommendations for service accounts
While it is straightforward to use the default service account, this account also
has broad privileges granted to it by default. Following the
[Principle of Least Privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)
you may want to create and use a service account that has only sufficient privileges
granted in order to run your `dsub` command/script.
To create a new service account, follow the steps below:
1. Execute the `gcloud iam service-accounts create` command. The email address
of the service account will be `sa-name@project-id.iam.gserviceaccount.com`.
gcloud iam service-accounts create "sa-name"
2. Grant IAM access on buckets, etc. to the service account.
gsutil iam ch serviceAccount:sa-name@project-id.iam.gserviceaccount.com:roles/storage.objectAdmin gs://bucket-name
3. Update your `dsub` command to include `--service-account`
dsub \
--service-account sa-name@project-id.iam.gserviceaccount.com
## What next?
* See the examples:
* [Custom scripts](https://github.com/DataBiosphere/dsub/tree/main/examples/custom_scripts)
* [Decompress files](https://github.com/DataBiosphere/dsub/tree/main/examples/decompress)
* [FastQC](https://github.com/DataBiosphere/dsub/tree/main/examples/fastqc)
* [Samtools index](https://github.com/DataBiosphere/dsub/tree/main/examples/samtools)
* See more documentation for:
* [Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/main/docs/code.md)
* [Input and Output File Handling](https://github.com/DataBiosphere/dsub/blob/main/docs/input_output.md)
* [Logging](https://github.com/DataBiosphere/dsub/blob/main/docs/logging.md)
* [Compute Resources](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_resources.md)
* [Compute Quotas](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_quotas.md)
* [Job Control](https://github.com/DataBiosphere/dsub/blob/main/docs/job_control.md)
* [Retries](https://github.com/DataBiosphere/dsub/blob/main/docs/retries.md)
* [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md)
* [Backend providers](https://github.com/DataBiosphere/dsub/blob/main/docs/providers/README.md)
%package -n python3-dsub
Summary: A command-line tool that makes it easy to submit and run batch scripts in the cloud
Provides: python-dsub
BuildRequires: python3-devel
BuildRequires: python3-setuptools
BuildRequires: python3-pip
%description -n python3-dsub
my-job-name RUNNING 2
my-job-name SUCCESS 1
```
In this mode, dstat prints one line per (job name, task status) pair. You can
see at a glance how many tasks are finished, how many are still running, and
how many are failed/canceled.
### Deleting a job
The `ddel` command will delete running jobs.
By default, only jobs submitted by the current user will be deleted.
Use the `--users` flag to specify other users, or `'*'` for all users.
To delete a running job:
ddel --provider google-v2 --project my-cloud-project --jobs job-id
If the job is a batch job, all running tasks will be deleted.
To delete specific tasks:
ddel \
--provider google-v2 \
--project my-cloud-project \
--jobs job-id \
--tasks task-id1 task-id2
To delete all running jobs for the current user:
ddel --provider google-v2 --project my-cloud-project --jobs '*'
## Service Accounts and Scope (Google providers only)
When you run the `dsub` command with the `google-v2` or `google-cls-v2`
provider, there are two different sets of credentials to consider:
- Account submitting the `pipelines.run()` request to run your command/script on a VM
- Account accessing Cloud resources (such as files in GCS) when executing your command/script
The account used to submit the `pipelines.run()` request is typically your
end user credentials. You would have set this up by running:
gcloud auth application-default login
The account used on the VM is a [service account](https://cloud.google.com/iam/docs/service-accounts).
The image below illustrates this:
By default, `dsub` will use the [default Compute Engine service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account)
as the authorized service account on the VM instance. You can choose to specify
the email address of another service acount using `--service-account`.
By default, `dsub` will grant the following access scopes to the service account:
- https://www.googleapis.com/auth/bigquery
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.full_control
- https://www.googleapis.com/auth/genomics
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
In addition, [the API](https://cloud.google.com/life-sciences/docs/reference/rest/v2beta/projects.locations.pipelines/run#serviceaccount) will always add this scope:
- https://www.googleapis.com/auth/cloud-platform
You can choose to specify scopes using `--scopes`.
### Recommendations for service accounts
While it is straightforward to use the default service account, this account also
has broad privileges granted to it by default. Following the
[Principle of Least Privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)
you may want to create and use a service account that has only sufficient privileges
granted in order to run your `dsub` command/script.
To create a new service account, follow the steps below:
1. Execute the `gcloud iam service-accounts create` command. The email address
of the service account will be `sa-name@project-id.iam.gserviceaccount.com`.
gcloud iam service-accounts create "sa-name"
2. Grant IAM access on buckets, etc. to the service account.
gsutil iam ch serviceAccount:sa-name@project-id.iam.gserviceaccount.com:roles/storage.objectAdmin gs://bucket-name
3. Update your `dsub` command to include `--service-account`
dsub \
--service-account sa-name@project-id.iam.gserviceaccount.com
## What next?
* See the examples:
* [Custom scripts](https://github.com/DataBiosphere/dsub/tree/main/examples/custom_scripts)
* [Decompress files](https://github.com/DataBiosphere/dsub/tree/main/examples/decompress)
* [FastQC](https://github.com/DataBiosphere/dsub/tree/main/examples/fastqc)
* [Samtools index](https://github.com/DataBiosphere/dsub/tree/main/examples/samtools)
* See more documentation for:
* [Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/main/docs/code.md)
* [Input and Output File Handling](https://github.com/DataBiosphere/dsub/blob/main/docs/input_output.md)
* [Logging](https://github.com/DataBiosphere/dsub/blob/main/docs/logging.md)
* [Compute Resources](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_resources.md)
* [Compute Quotas](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_quotas.md)
* [Job Control](https://github.com/DataBiosphere/dsub/blob/main/docs/job_control.md)
* [Retries](https://github.com/DataBiosphere/dsub/blob/main/docs/retries.md)
* [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md)
* [Backend providers](https://github.com/DataBiosphere/dsub/blob/main/docs/providers/README.md)
%package help
Summary: Development documents and examples for dsub
Provides: python3-dsub-doc
%description help
my-job-name RUNNING 2
my-job-name SUCCESS 1
```
In this mode, dstat prints one line per (job name, task status) pair. You can
see at a glance how many tasks are finished, how many are still running, and
how many are failed/canceled.
### Deleting a job
The `ddel` command will delete running jobs.
By default, only jobs submitted by the current user will be deleted.
Use the `--users` flag to specify other users, or `'*'` for all users.
To delete a running job:
ddel --provider google-v2 --project my-cloud-project --jobs job-id
If the job is a batch job, all running tasks will be deleted.
To delete specific tasks:
ddel \
--provider google-v2 \
--project my-cloud-project \
--jobs job-id \
--tasks task-id1 task-id2
To delete all running jobs for the current user:
ddel --provider google-v2 --project my-cloud-project --jobs '*'
## Service Accounts and Scope (Google providers only)
When you run the `dsub` command with the `google-v2` or `google-cls-v2`
provider, there are two different sets of credentials to consider:
- Account submitting the `pipelines.run()` request to run your command/script on a VM
- Account accessing Cloud resources (such as files in GCS) when executing your command/script
The account used to submit the `pipelines.run()` request is typically your
end user credentials. You would have set this up by running:
gcloud auth application-default login
The account used on the VM is a [service account](https://cloud.google.com/iam/docs/service-accounts).
The image below illustrates this:
By default, `dsub` will use the [default Compute Engine service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account)
as the authorized service account on the VM instance. You can choose to specify
the email address of another service acount using `--service-account`.
By default, `dsub` will grant the following access scopes to the service account:
- https://www.googleapis.com/auth/bigquery
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.full_control
- https://www.googleapis.com/auth/genomics
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
In addition, [the API](https://cloud.google.com/life-sciences/docs/reference/rest/v2beta/projects.locations.pipelines/run#serviceaccount) will always add this scope:
- https://www.googleapis.com/auth/cloud-platform
You can choose to specify scopes using `--scopes`.
### Recommendations for service accounts
While it is straightforward to use the default service account, this account also
has broad privileges granted to it by default. Following the
[Principle of Least Privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)
you may want to create and use a service account that has only sufficient privileges
granted in order to run your `dsub` command/script.
To create a new service account, follow the steps below:
1. Execute the `gcloud iam service-accounts create` command. The email address
of the service account will be `sa-name@project-id.iam.gserviceaccount.com`.
gcloud iam service-accounts create "sa-name"
2. Grant IAM access on buckets, etc. to the service account.
gsutil iam ch serviceAccount:sa-name@project-id.iam.gserviceaccount.com:roles/storage.objectAdmin gs://bucket-name
3. Update your `dsub` command to include `--service-account`
dsub \
--service-account sa-name@project-id.iam.gserviceaccount.com
## What next?
* See the examples:
* [Custom scripts](https://github.com/DataBiosphere/dsub/tree/main/examples/custom_scripts)
* [Decompress files](https://github.com/DataBiosphere/dsub/tree/main/examples/decompress)
* [FastQC](https://github.com/DataBiosphere/dsub/tree/main/examples/fastqc)
* [Samtools index](https://github.com/DataBiosphere/dsub/tree/main/examples/samtools)
* See more documentation for:
* [Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/main/docs/code.md)
* [Input and Output File Handling](https://github.com/DataBiosphere/dsub/blob/main/docs/input_output.md)
* [Logging](https://github.com/DataBiosphere/dsub/blob/main/docs/logging.md)
* [Compute Resources](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_resources.md)
* [Compute Quotas](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_quotas.md)
* [Job Control](https://github.com/DataBiosphere/dsub/blob/main/docs/job_control.md)
* [Retries](https://github.com/DataBiosphere/dsub/blob/main/docs/retries.md)
* [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md)
* [Backend providers](https://github.com/DataBiosphere/dsub/blob/main/docs/providers/README.md)
%prep
%autosetup -n dsub-0.4.9
%build
%py3_build
%install
%py3_install
install -d -m755 %{buildroot}/%{_pkgdocdir}
if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
pushd %{buildroot}
if [ -d usr/lib ]; then
find usr/lib -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
if [ -d usr/lib64 ]; then
find usr/lib64 -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
if [ -d usr/bin ]; then
find usr/bin -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
if [ -d usr/sbin ]; then
find usr/sbin -type f -printf "\"/%h/%f\"\n" >> filelist.lst
fi
touch doclist.lst
if [ -d usr/share/man ]; then
find usr/share/man -type f -printf "\"/%h/%f.gz\"\n" >> doclist.lst
fi
popd
mv %{buildroot}/filelist.lst .
mv %{buildroot}/doclist.lst .
%files -n python3-dsub -f filelist.lst
%dir %{python3_sitelib}/*
%files help -f doclist.lst
%{_docdir}/*
%changelog
* Fri Jun 09 2023 Python_Bot <Python_Bot@openeuler.org> - 0.4.9-1
- Package Spec generated
|
