%global _empty_manifest_terminate_build 0
Name:		python-tx-parallex
Version:	0.1.11
Release:	1
Summary:	A job queue with data dependencies
License:	MIT
URL:		https://github.com/RENCI/tx-parallex
Source0:	https://mirrors.nju.edu.cn/pypi/web/packages/cf/72/6c2cb7b4f74ffd3e485e3f7cd49f8302bd1e5f09fcf44c72a383bb74140f/tx-parallex-0.1.11.tar.gz
BuildArch:	noarch

Requires:	python3-more-itertools
Requires:	python3-jsonschema
Requires:	python3-pyyaml
Requires:	python3-tx-functional
Requires:	python3-graph-theory
Requires:	python3-terminaltables
Requires:	python3-jsonpickle
Requires:	python3-pyarrow

%description
[![Build Status](https://travis-ci.com/RENCI/tx-parallex.svg?branch=master)](https://travis-ci.com/RENCI/tx-parallex)

# `parallex`


## System Requirements
Python >= 3.8
## install

Default object store
```
pip install tx-parallex
```

Plasma store https://arrow.apache.org/
```
pip install tx-parallex[arrow]
```
## Install from source
1. Clone the repo
2. Easy install instructions:
```
# Create a virtual environment called 'px'
conda create -n px python=3.8
# start-up the environment you just created
conda activate px
# install the rest of the tx-parallex pre-requirements
pip install -r requirements.txt
```
3. Test
```
# run the tests, a number of test 'specs'
PYTHONPATH=src pytest -x -vv --full-trace -s --timeout 60
# deactivate the environment (if desired)
conda deactivate
```
## set log level
set environment variable LOG_LEVEL to one of Python's logging library setLevel.

## Introduction
A queue with dependencies

## Usage

```
from tx.parallex import run_python

ret = run_python(number_of_workers = 4, pyf = "spec.py", dataf = "data.yml")
```

## Spec

`tx-parallex` specs can be written in YAML or a Python-like DSL. The Python-like DSL is translated to YAML by `tx-parallex`. Each object in a spec specifies a task. When the task is executed, it is given a dict called `data`. The pipeline will return a dictionary.

### YAML
Assuming you have a function `sqr` defined in module `math` which returns the square of its argument.

```
def sqr(x):
  return x * x
```

#### `let`
The `let` task sets `data` for its subtask. It adds a new var value pair into `data` within the scope of its subtask, and executes that task.

Syntax:
```
type: let
var: <var>
obj: <value>
sub: <subtask>
```

Example:
```
type: let
var: a
obj:
  data: 1
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: a
```



### `map`
The `map` task reads a list `coll` from `data` and applies a subtask to each member of the list. The members will be assigned to `var` in `data` passed to those tasks

Syntax:
```
type: map
coll: <value>
var: <variable name>
sub: <subtask>
```

`<value>` is an object of the form:

Reference an entry in `data` or the name of a task
```
"name": <variable name>
```

Constant
```
"data": <constant>
```

Example:
```
type: map
coll: 
  data:
  - 1
  - 2
  - 3
var: a
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: a
```

### `cond`
The `cond` task reads a boolean value and if it is true then it executes the `then` task otherwise it executes the `else` task.

Syntax:
```
type: cond
on: <value>
then: <subtask>
else: <subtask>
```

Example:
```
type: cond
on: 
  data:
    true
then:
  type: ret
  obj:
    data: 1
else:
  type: ret
  obj:
    data: 0
```



### `python`

You can use any Python module.

The `python` task runs a Python function. It reads parameters from `data`. The return value must be pickleable.

Syntax:
```
type: python
name: <name>
mod: <module>
func: <function>
params: <parameters>
```

`<parameters>` is an object of the form:
```
<param> : <value>
...
<param> : <value>
```
where `<param>` can be either name or position.

Example:
```
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
```
### `top`

The `top` task toplogically sorts subtasks based on their dependencies and ensure the tasks are executed in parallel in the order compatible with those dependencies. 

Syntax:
```
type: top
sub: <subtasks>
```

It reads the `name` properties of subtasks that are not in data.

Example:
```
type: top
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: y
```

### `seq`

The `seq` task forces all subtasks to be run sequentially. 

Syntax:
```
type: top
sub: <subtasks>
```

It reads the `name` properties of subtasks that are not in data.

Example:
```
type: seq
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: y
```

### `ret`
`ret` specify a value. The pipeline will return a dictionary. When a task appears under a `map` task, it is prefix with the index of the element in that collection as following 

```
<index>
```
For nested maps, the indices will be chained together as followings
```
<index>. ... .<index>
```

Syntax:
```
type: ret
obj: <value>
```

Example:
```
type: ret
obj: 
    name: z
```

## Python
A dsl block contains a subset of Python.

* There is a semantic difference from python. Any assignment in block is not visiable outside of the block.
* Assignment within a block are unordered
* return statement


Available syntax:

### import

```
from <module> import *
from <module> import <func>, ..., <func>
```

import names from module

`<module>` absolute module names

### assignment
```
<var> = <const>
```
where
```
<const> = <integer> | <number> | <boolean> | <string> | <list> | <dict>
```

This translates to `let`.

Example:
```
a = 1
y = sqr(x=a)
yield y
```

### function application

```
<var> = [<module>.]<func>(<param>=<expr>, ...) | <expr>
```

This translate to `python`.
where `<var>` is `name`
`<expr>` is

```
<expr> = <expr> if <expr> else <expr> | <expr> <binop> <expr> | <expr> <boolop> <expr> | <expr> <compare> <expr> | <unaryop> <expr> | <var> | <const>
```

`<binop>`, `<boolop>` and `<compare>` and `<unaryop>` are python BinOp, BoolOp, Compare, and UnaryOp. `<expr>` is translated to a set of assignments, `name`, or `data` depending on its content.

Example:
```
y = math.sqr(1)
z = math.sqr(y)
return z
```

### parallel for

```
for <var> in <expr>:
    ...
```
This translates to `map`.

Example:

```
for a in [1, 2, 3]:
  y = math.sqr(a)
  yield y
```

### if
```
if <expr>:
    ...
else:
    ...
```

This translates to `cond`.

Example:
```
if z:
    yield 1
else:
    yield 0
```

The semantics of if is different from python, variables inside if is not visible outside
### with
```
with Seq:
    ...
```
This translates to `seq`. 

Example:
```
with Seq:
    y = math.sqr(1)
    return y
```

### yield
```
yield <expr>
```
This translates to `ret`. 

Example:
```
y = math.sqr(1)
return y
```

## Data

data can be arbitrary yaml





%package -n python3-tx-parallex
Summary:	A job queue with data dependencies
Provides:	python-tx-parallex
BuildRequires:	python3-devel
BuildRequires:	python3-setuptools
BuildRequires:	python3-pip
%description -n python3-tx-parallex
[![Build Status](https://travis-ci.com/RENCI/tx-parallex.svg?branch=master)](https://travis-ci.com/RENCI/tx-parallex)

# `parallex`


## System Requirements
Python >= 3.8
## install

Default object store
```
pip install tx-parallex
```

Plasma store https://arrow.apache.org/
```
pip install tx-parallex[arrow]
```
## Install from source
1. Clone the repo
2. Easy install instructions:
```
# Create a virtual environment called 'px'
conda create -n px python=3.8
# start-up the environment you just created
conda activate px
# install the rest of the tx-parallex pre-requirements
pip install -r requirements.txt
```
3. Test
```
# run the tests, a number of test 'specs'
PYTHONPATH=src pytest -x -vv --full-trace -s --timeout 60
# deactivate the environment (if desired)
conda deactivate
```
## set log level
set environment variable LOG_LEVEL to one of Python's logging library setLevel.

## Introduction
A queue with dependencies

## Usage

```
from tx.parallex import run_python

ret = run_python(number_of_workers = 4, pyf = "spec.py", dataf = "data.yml")
```

## Spec

`tx-parallex` specs can be written in YAML or a Python-like DSL. The Python-like DSL is translated to YAML by `tx-parallex`. Each object in a spec specifies a task. When the task is executed, it is given a dict called `data`. The pipeline will return a dictionary.

### YAML
Assuming you have a function `sqr` defined in module `math` which returns the square of its argument.

```
def sqr(x):
  return x * x
```

#### `let`
The `let` task sets `data` for its subtask. It adds a new var value pair into `data` within the scope of its subtask, and executes that task.

Syntax:
```
type: let
var: <var>
obj: <value>
sub: <subtask>
```

Example:
```
type: let
var: a
obj:
  data: 1
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: a
```



### `map`
The `map` task reads a list `coll` from `data` and applies a subtask to each member of the list. The members will be assigned to `var` in `data` passed to those tasks

Syntax:
```
type: map
coll: <value>
var: <variable name>
sub: <subtask>
```

`<value>` is an object of the form:

Reference an entry in `data` or the name of a task
```
"name": <variable name>
```

Constant
```
"data": <constant>
```

Example:
```
type: map
coll: 
  data:
  - 1
  - 2
  - 3
var: a
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: a
```

### `cond`
The `cond` task reads a boolean value and if it is true then it executes the `then` task otherwise it executes the `else` task.

Syntax:
```
type: cond
on: <value>
then: <subtask>
else: <subtask>
```

Example:
```
type: cond
on: 
  data:
    true
then:
  type: ret
  obj:
    data: 1
else:
  type: ret
  obj:
    data: 0
```



### `python`

You can use any Python module.

The `python` task runs a Python function. It reads parameters from `data`. The return value must be pickleable.

Syntax:
```
type: python
name: <name>
mod: <module>
func: <function>
params: <parameters>
```

`<parameters>` is an object of the form:
```
<param> : <value>
...
<param> : <value>
```
where `<param>` can be either name or position.

Example:
```
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
```
### `top`

The `top` task toplogically sorts subtasks based on their dependencies and ensure the tasks are executed in parallel in the order compatible with those dependencies. 

Syntax:
```
type: top
sub: <subtasks>
```

It reads the `name` properties of subtasks that are not in data.

Example:
```
type: top
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: y
```

### `seq`

The `seq` task forces all subtasks to be run sequentially. 

Syntax:
```
type: top
sub: <subtasks>
```

It reads the `name` properties of subtasks that are not in data.

Example:
```
type: seq
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: y
```

### `ret`
`ret` specify a value. The pipeline will return a dictionary. When a task appears under a `map` task, it is prefix with the index of the element in that collection as following 

```
<index>
```
For nested maps, the indices will be chained together as followings
```
<index>. ... .<index>
```

Syntax:
```
type: ret
obj: <value>
```

Example:
```
type: ret
obj: 
    name: z
```

## Python
A dsl block contains a subset of Python.

* There is a semantic difference from python. Any assignment in block is not visiable outside of the block.
* Assignment within a block are unordered
* return statement


Available syntax:

### import

```
from <module> import *
from <module> import <func>, ..., <func>
```

import names from module

`<module>` absolute module names

### assignment
```
<var> = <const>
```
where
```
<const> = <integer> | <number> | <boolean> | <string> | <list> | <dict>
```

This translates to `let`.

Example:
```
a = 1
y = sqr(x=a)
yield y
```

### function application

```
<var> = [<module>.]<func>(<param>=<expr>, ...) | <expr>
```

This translate to `python`.
where `<var>` is `name`
`<expr>` is

```
<expr> = <expr> if <expr> else <expr> | <expr> <binop> <expr> | <expr> <boolop> <expr> | <expr> <compare> <expr> | <unaryop> <expr> | <var> | <const>
```

`<binop>`, `<boolop>` and `<compare>` and `<unaryop>` are python BinOp, BoolOp, Compare, and UnaryOp. `<expr>` is translated to a set of assignments, `name`, or `data` depending on its content.

Example:
```
y = math.sqr(1)
z = math.sqr(y)
return z
```

### parallel for

```
for <var> in <expr>:
    ...
```
This translates to `map`.

Example:

```
for a in [1, 2, 3]:
  y = math.sqr(a)
  yield y
```

### if
```
if <expr>:
    ...
else:
    ...
```

This translates to `cond`.

Example:
```
if z:
    yield 1
else:
    yield 0
```

The semantics of if is different from python, variables inside if is not visible outside
### with
```
with Seq:
    ...
```
This translates to `seq`. 

Example:
```
with Seq:
    y = math.sqr(1)
    return y
```

### yield
```
yield <expr>
```
This translates to `ret`. 

Example:
```
y = math.sqr(1)
return y
```

## Data

data can be arbitrary yaml





%package help
Summary:	Development documents and examples for tx-parallex
Provides:	python3-tx-parallex-doc
%description help
[![Build Status](https://travis-ci.com/RENCI/tx-parallex.svg?branch=master)](https://travis-ci.com/RENCI/tx-parallex)

# `parallex`


## System Requirements
Python >= 3.8
## install

Default object store
```
pip install tx-parallex
```

Plasma store https://arrow.apache.org/
```
pip install tx-parallex[arrow]
```
## Install from source
1. Clone the repo
2. Easy install instructions:
```
# Create a virtual environment called 'px'
conda create -n px python=3.8
# start-up the environment you just created
conda activate px
# install the rest of the tx-parallex pre-requirements
pip install -r requirements.txt
```
3. Test
```
# run the tests, a number of test 'specs'
PYTHONPATH=src pytest -x -vv --full-trace -s --timeout 60
# deactivate the environment (if desired)
conda deactivate
```
## set log level
set environment variable LOG_LEVEL to one of Python's logging library setLevel.

## Introduction
A queue with dependencies

## Usage

```
from tx.parallex import run_python

ret = run_python(number_of_workers = 4, pyf = "spec.py", dataf = "data.yml")
```

## Spec

`tx-parallex` specs can be written in YAML or a Python-like DSL. The Python-like DSL is translated to YAML by `tx-parallex`. Each object in a spec specifies a task. When the task is executed, it is given a dict called `data`. The pipeline will return a dictionary.

### YAML
Assuming you have a function `sqr` defined in module `math` which returns the square of its argument.

```
def sqr(x):
  return x * x
```

#### `let`
The `let` task sets `data` for its subtask. It adds a new var value pair into `data` within the scope of its subtask, and executes that task.

Syntax:
```
type: let
var: <var>
obj: <value>
sub: <subtask>
```

Example:
```
type: let
var: a
obj:
  data: 1
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: a
```



### `map`
The `map` task reads a list `coll` from `data` and applies a subtask to each member of the list. The members will be assigned to `var` in `data` passed to those tasks

Syntax:
```
type: map
coll: <value>
var: <variable name>
sub: <subtask>
```

`<value>` is an object of the form:

Reference an entry in `data` or the name of a task
```
"name": <variable name>
```

Constant
```
"data": <constant>
```

Example:
```
type: map
coll: 
  data:
  - 1
  - 2
  - 3
var: a
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: a
```

### `cond`
The `cond` task reads a boolean value and if it is true then it executes the `then` task otherwise it executes the `else` task.

Syntax:
```
type: cond
on: <value>
then: <subtask>
else: <subtask>
```

Example:
```
type: cond
on: 
  data:
    true
then:
  type: ret
  obj:
    data: 1
else:
  type: ret
  obj:
    data: 0
```



### `python`

You can use any Python module.

The `python` task runs a Python function. It reads parameters from `data`. The return value must be pickleable.

Syntax:
```
type: python
name: <name>
mod: <module>
func: <function>
params: <parameters>
```

`<parameters>` is an object of the form:
```
<param> : <value>
...
<param> : <value>
```
where `<param>` can be either name or position.

Example:
```
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
```
### `top`

The `top` task toplogically sorts subtasks based on their dependencies and ensure the tasks are executed in parallel in the order compatible with those dependencies. 

Syntax:
```
type: top
sub: <subtasks>
```

It reads the `name` properties of subtasks that are not in data.

Example:
```
type: top
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: y
```

### `seq`

The `seq` task forces all subtasks to be run sequentially. 

Syntax:
```
type: top
sub: <subtasks>
```

It reads the `name` properties of subtasks that are not in data.

Example:
```
type: seq
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: y
```

### `ret`
`ret` specify a value. The pipeline will return a dictionary. When a task appears under a `map` task, it is prefix with the index of the element in that collection as following 

```
<index>
```
For nested maps, the indices will be chained together as followings
```
<index>. ... .<index>
```

Syntax:
```
type: ret
obj: <value>
```

Example:
```
type: ret
obj: 
    name: z
```

## Python
A dsl block contains a subset of Python.

* There is a semantic difference from python. Any assignment in block is not visiable outside of the block.
* Assignment within a block are unordered
* return statement


Available syntax:

### import

```
from <module> import *
from <module> import <func>, ..., <func>
```

import names from module

`<module>` absolute module names

### assignment
```
<var> = <const>
```
where
```
<const> = <integer> | <number> | <boolean> | <string> | <list> | <dict>
```

This translates to `let`.

Example:
```
a = 1
y = sqr(x=a)
yield y
```

### function application

```
<var> = [<module>.]<func>(<param>=<expr>, ...) | <expr>
```

This translate to `python`.
where `<var>` is `name`
`<expr>` is

```
<expr> = <expr> if <expr> else <expr> | <expr> <binop> <expr> | <expr> <boolop> <expr> | <expr> <compare> <expr> | <unaryop> <expr> | <var> | <const>
```

`<binop>`, `<boolop>` and `<compare>` and `<unaryop>` are python BinOp, BoolOp, Compare, and UnaryOp. `<expr>` is translated to a set of assignments, `name`, or `data` depending on its content.

Example:
```
y = math.sqr(1)
z = math.sqr(y)
return z
```

### parallel for

```
for <var> in <expr>:
    ...
```
This translates to `map`.

Example:

```
for a in [1, 2, 3]:
  y = math.sqr(a)
  yield y
```

### if
```
if <expr>:
    ...
else:
    ...
```

This translates to `cond`.

Example:
```
if z:
    yield 1
else:
    yield 0
```

The semantics of if is different from python, variables inside if is not visible outside
### with
```
with Seq:
    ...
```
This translates to `seq`. 

Example:
```
with Seq:
    y = math.sqr(1)
    return y
```

### yield
```
yield <expr>
```
This translates to `ret`. 

Example:
```
y = math.sqr(1)
return y
```

## Data

data can be arbitrary yaml





%prep
%autosetup -n tx-parallex-0.1.11

%build
%py3_build

%install
%py3_install
install -d -m755 %{buildroot}/%{_pkgdocdir}
if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
pushd %{buildroot}
if [ -d usr/lib ]; then
	find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/lib64 ]; then
	find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/bin ]; then
	find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/sbin ]; then
	find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
fi
touch doclist.lst
if [ -d usr/share/man ]; then
	find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
fi
popd
mv %{buildroot}/filelist.lst .
mv %{buildroot}/doclist.lst .

%files -n python3-tx-parallex -f filelist.lst
%dir %{python3_sitelib}/*

%files help -f doclist.lst
%{_docdir}/*

%changelog
* Tue May 30 2023 Python_Bot <Python_Bot@openeuler.org> - 0.1.11-1
- Package Spec generated