diff options
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | python-dnn-cool.spec | 1092 | ||||
| -rw-r--r-- | sources | 1 |
3 files changed, 1094 insertions, 0 deletions
@@ -0,0 +1 @@ +/dnn_cool-0.4.0.tar.gz diff --git a/python-dnn-cool.spec b/python-dnn-cool.spec new file mode 100644 index 0000000..935a744 --- /dev/null +++ b/python-dnn-cool.spec @@ -0,0 +1,1092 @@ +%global _empty_manifest_terminate_build 0 +Name: python-dnn-cool +Version: 0.4.0 +Release: 1 +Summary: DNN.Cool: Multi-task learning for Deep Neural Networks (DNN). +License: MIT +URL: https://github.com/hristo-vrigazov/dnn.cool +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/b6/d7/7897f57c34e468a48f01fc91fa16ffb786802192b606ce37a03855621e95/dnn_cool-0.4.0.tar.gz +BuildArch: noarch + + +%description +## `dnn_cool`: Deep Neural Networks for Conditional objective oriented learning + +WARNING: API is not yet stable, expect breaking changes in 0.x versions! + +To install, just do: + +```bash +pip install dnn_cool +``` + +* [Introduction](#introduction): What is `dnn_cool` in a nutshell? +* [Examples](#examples): a simple step-by-step example. +* [Features](#features): a list of the utilities that `dnn_cool` provides for you +* [Customization](#customization): Learn how to add new tasks, modify them, etc. +* [Inspiration](#inspiration): list of papers and videos which inspired this library + +To see the predefined tasks for this release, see [list of predefined tasks](#list-of-predefined-tasks) + +### Introduction + +A framework for multi-task learning in Pytorch, where you may precondition tasks and compose them into bigger tasks. +Many complex neural networks can be trivially implemented with `dnn_cool`. +For example, creating a neural network that does classification and localization is as simple as: + +```python +@project.add_flow +def localize_flow(flow, x, out): + out += flow.obj_exists(x.features) + out += flow.obj_x(x.features) | out.obj_exists + out += flow.obj_y(x.features) | out.obj_exists + out += flow.obj_w(x.features) | out.obj_exists + out += flow.obj_h(x.features) | out.obj_exists + out += flow.obj_class(x.features) | out.obj_exists + return out +``` + +If for example you want to classify first if the camera is blocked and then do localization **given that the camera +is not blocked**, you could do: + +```python +@project.add_flow +def full_flow(flow, x, out): + out += flow.camera_blocked(x.cam_features) + out += flow.localize_flow(x.localization_features) | (~out.camera_blocked) + return out +``` + +Based on these "task flows" as we call them, `dnn_cool` provides a bunch of [features](#features). +Currently, this is the list of the predefined tasks (they are all located in `dnn_cool.task_flow`): + +##### List of predefined tasks + +In the current release, the following tasks are available out of the box: + +* `BinaryClassificationTask` - sigmoid activation, thresholding decoder, binary cross entropy loss function. In the +examples above, `camera_blocked` and `obj_exists` are `BinaryClassificationTask`s. +* `ClassificationTask` - softmax activation, sorting classes decoder, categorical cross entropy loss. In the example +above, `obj_class` is a `ClassificationTask` +* `MultilabelClassificationTask` - sigmoid activation, thresholding decoder, binary cross entropy loss function. +* `BoundedRegressionTask` - sigmoid activation, rescaling decoder, mean squared error loss function. In the examples +above, `obj_x`, `obj_y`, `obj_w`, `obj_h` are bounded regression tasks. +* `MaskedLanguageModelingTask` - softmax activation, sorting decoder, cross entropy per token loss. +* `TaskFlow` - a composite task, that contains a list of children tasks. We saw 2 task flows above. + +### Examples + +#### Quick Imagenet example + +We just have to add a `ClassificationTask` named `classifier` and add the flow below: + +```python +@project.add_flow() +def imagenet_model(flow, x, out): + out += flow.classifier(x.features) + return out +``` + +That's great! But what if there is not an object always? Then we have to first check if an object exists. Let's +add a `BinaryClassificationTask` and use it as a precondition to classifier. + +```python +@project.add_flow() +def imagenet_model(flow, x, out): + out += flow.object_exists(x.features) + out += flow.classifier(x.features) | out.object_exists + return out +``` + +But what if we also want to localize the object? Then we have to add new tasks that regress the bounding box. Let's +call them `object_x`, `object_y`, `object_w`, `object_h` and make them a `BoundedRegressionTask`. To avoid +preconditioning all tasks on `object_exists`, let's group them first. Then we modify the +flow: + +```python +@project.add_flow() +def object_flow(flow, x, out): + out += flow.classifier(x.features) + out += flow.object_x(x.features) + out += flow.object_y(x.features) + out += flow.object_w(x.features) + out += flow.object_h(x.features) + return out + +@project.add_flow() +def imagenet_flow(flow, x, out): + out += flow.object_exists(x.features) + out += flow.object_flow(x.features) | out.object_exists + return out +``` + +But what if the camera is blocked? Then there is no need to do anything, so let's create a new flow +that executes our `imagenet_flow` only when the camera is not blocked. + +```python +def full_flow(flow, x, out): + out += flow.camera_blocked(x.features) + out += flow.imagenet_flow(x.features) | (~out.camera_blocked) + return out +``` + +But what if for example we want to check if the object is a kite, and if it is, to classify its color? +Then we would have to modify our `object_flow` as follows: + +```python +@project.add_flow() +def object_flow(flow, x, out): + out += flow.classifier(x.features) + out += flow.object_x(x.features) + out += flow.object_y(x.features) + out += flow.object_w(x.features) + out += flow.object_h(x.features) + out += flow.is_kite(x.features) + out += flow.color(x.features) | out.is_kite + return out +``` + +I think you can see what `dnn_cool` is meant to do! :) + +To see a full walkthrough on a synthetic dataset, check out the [Colab notebook](https://colab.research.google.com/drive/1fEidcOszTI9JXptbuU5GGC-O_yxb6hxO?usp=sharing) +or the [markdown write-up](./story.md). + +### Features + +Main features are: + +* [Task precondition](#task-preconditioning) +* [Missing values handling](#missing-values) +* [Task composition](#task-composition) +* [Tensorboard metrics logging](#tensorboard-logging) +* [Task interpretations](#task-interpretation) +* [Task evaluation](#task-evaluation) +* [Task threshold tuning](#task-threshold-tuning) +* [Dataset generation](#dataset-generation) +* [Tree explanations](#tree-explanations) +* [Memory balancing for dataparallel](#memory-balancing) + +##### Task preconditioning + +Use the `|` for task preconditioning (think of `P(A|B)` notation). Preconditioning - ` A | B` means that: + +* Include the ground truth for `B` in the input batch when training +* When training, update the weights of the `A` only when `B` is satisfied in the ground truth. +* When training, compute the loss function for `A` only when `B` is satisfied in the ground truth +* When training, compute the metrics for `A` only when `B` is satisfied in the ground truth. +* When tuning threshold for `A`, optimize only on values for which `B` is satisfied in the ground truth. +* When doing inference, compute the metrics for `A` only when the precondition is satisfied according to the decoded +result of the `B` task +* When generating tree explanation in inference mode, do not show the branch for `A` if `B` is not +satisfied. +* When computing results interpretation, include only loss terms when the precondition is satisfied. + +Usually, you have to keep track of all this stuff manually, which makes adding new preconditions very difficult. +`dnn_cool` makes this stuff easy, so that you can chain a long list of preconditions without worrying you forgot +something. + +##### Missing values + +Sometimes for an input you don't have labels for all tasks. With `dnn_cool`, you can just mark the missing label and +`dnn_cool` will update only the weights of the tasks for which labels are available. + +This feature has the awesome property that you don't need a single dataset with all tasks labeled, you can +have different datasets for different tasks and it will work. For example, you can train a single object detection +neural network that trains its classifier head on ImageNet, and its detection head on COCO. + +##### Task composition + +You can group tasks in a task flow (we already saw 2 above - `localize_flow` and `full_flow`). You can use this to +organize things better, for example when you want to precondition a whole task flow. For example: + +```python +@project.add_flow +def face_regression(flow, x, out): + out += flow.face_x1(x.face_localization) + out += flow.face_y1(x.face_localization) + out += flow.face_w(x.face_localization) + out += flow.face_h(x.face_localization) + out += flow.facial_characteristics(x.features) + return out +``` + +##### Tensorboard logging + +`dnn_cool` logs the metrics per task in Tensorboard, e.g: + + + +##### Task interpretation + +Also, the best and worst inputs per task are logged in the Tensorboard, for example if the input is an image: + +  + +##### Task evaluation + +Per-task evaluation information is available, to pinpoint the exact problem in your network. An example +evaluation dataframe: + +| | task_path | metric_name | metric_res | num_samples | +|---:|:---------------------------------------------------------|:--------------------|-------------:|--------------:| +| 0 | camera_blocked | accuracy | 0.980326 | 996 | +| 1 | camera_blocked | f1_score | 0.974368 | 996 | +| 2 | camera_blocked | precision | 0.946635 | 996 | +| 3 | camera_blocked | recall | 0.960107 | 996 | +| 4 | door_open | accuracy | 0.921215 | 902 | +| 5 | door_open | f1_score | 0.966859 | 902 | +| 6 | door_open | precision | 0.976749 | 902 | +| 7 | door_open | recall | 0.939038 | 902 | +| 8 | door_locked | accuracy | 0.983039 | 201 | +| 9 | door_locked | f1_score | 0.948372 | 201 | +| 10 | door_locked | precision | 0.982583 | 201 | +| 11 | door_locked | recall | 0.934788 | 201 | +| 12 | person_present | accuracy | 0.999166 | 701 | +| 13 | person_present | f1_score | 0.937541 | 701 | +| 14 | person_present | precision | 0.927337 | 701 | +| 15 | person_present | recall | 0.963428 | 701 | +| 16 | person_regression.face_regression.face_x1 | mean_absolute_error | 0.0137292 | 611 | +| 17 | person_regression.face_regression.face_y1 | mean_absolute_error | 0.0232761 | 611 | +| 18 | person_regression.face_regression.face_w | mean_absolute_error | 0.00740503 | 611 | +| 19 | person_regression.face_regression.face_h | mean_absolute_error | 0.0101 | 611 | +| 20 | person_regression.face_regression.facial_characteristics | accuracy | 0.932624 | 611 | +| 21 | person_regression.body_regression.body_x1 | mean_absolute_error | 0.00830785 | 611 | +| 22 | person_regression.body_regression.body_y1 | mean_absolute_error | 0.0151234 | 611 | +| 23 | person_regression.body_regression.body_w | mean_absolute_error | 0.0130214 | 611 | +| 24 | person_regression.body_regression.body_h | mean_absolute_error | 0.0101 | 611 | +| 25 | person_regression.body_regression.shirt_type | accuracy_1 | 0.979934 | 611 | +| 26 | person_regression.body_regression.shirt_type | accuracy_3 | 0.993334 | 611 | +| 27 | person_regression.body_regression.shirt_type | accuracy_5 | 0.990526 | 611 | +| 28 | person_regression.body_regression.shirt_type | f1_score | 0.928516 | 611 | +| 29 | person_regression.body_regression.shirt_type | precision | 0.959826 | 611 | +| 30 | person_regression.body_regression.shirt_type | recall | 0.968146 | 611 | + +##### Task threshold tuning + +Many tasks need to tune their threshold. Just call `flow.get_decoder().tune()` and you will get optimized thresholds +for the metric you define. + +##### Dataset generation + +As noted above, `dnn_cool` will automatically trace the tasks used as a precondition and include the ground truth for +them under the key `gt`. + +##### Tree explanations + +Examples: + +``` +├── inp 1 +│ └── camera_blocked | decoded: [False], activated: [0.], logits: [-117.757324] +│ └── door_open | decoded: [ True], activated: [1.], logits: [41.11258] +│ └── person_present | decoded: [ True], activated: [1.], logits: [60.38873] +│ └── person_regression +│ ├── body_regression +│ │ ├── body_h | decoded: [29.672623], activated: [0.46363473], logits: [-0.14571853] +│ │ ├── body_w | decoded: [12.86382], activated: [0.20099719], logits: [-1.3800735] +│ │ ├── body_x1 | decoded: [21.34288], activated: [0.3334825], logits: [-0.69247603] +│ │ ├── body_y1 | decoded: [18.468979], activated: [0.2885778], logits: [-0.9023013] +│ │ └── shirt_type | decoded: [6 1 0 4 2 5 3], activated: [4.1331367e-23 3.5493638e-17 3.1328378e-26 5.6903808e-30 2.4471377e-25 + 2.8071076e-29 1.0000000e+00], logits: [-20.549513 -6.88627 -27.734364 -36.34787 -25.6788 -34.751904 + 30.990908] +│ └── face_regression +│ ├── face_h | decoded: [11.265154], activated: [0.17601803], logits: [-1.5435623] +│ ├── face_w | decoded: [12.225838], activated: [0.19102871], logits: [-1.4433397] +│ ├── face_x1 | decoded: [21.98834], activated: [0.34356782], logits: [-0.64743483] +│ ├── face_y1 | decoded: [3.2855165], activated: [0.0513362], logits: [-2.9166584] +│ └── facial_characteristics | decoded: [ True False True], activated: [9.9999940e-01 1.2074912e-12 9.9999833e-01], logits: [ 14.240071 -27.442476 13.27557 ] + +``` + +but if the model thinks the camera is blocked, then the explanation would be: + +``` +├── inp 2 +│ └── camera_blocked | decoded: [ True], activated: [1.], logits: [76.367676] +``` + +##### Memory balancing + +When using [nn.DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html), the computation of +the loss function is done on the main GPU, which leads to dramatically unbalanced memory usage if your outputs are big +and you have a lot of metrics (e.g segmentation masks, language modeling, etc). `dnn_cool` gives you a +convenient way to balance the memory in such situations - just a single `balance_dataparallel_memory = True` handles +this case for you by first reducing all metrics on their respective device, and then additionally aggregating +the results that were reduced on each device automatically. Here's an example memory usage: + + +Before: + + + +After: + + + +### Customization + +Since `flow.torch()` returns a normal `nn.Module`, you can use any library you are used to. If you use +[Catalyst](https://github.com/catalyst-team/catalyst), `dnn_cool` provides a bunch of useful callbacks. Creating +a new task is as simple as creating a new instance of this dataclass: + +```python +@dataclass +class Task(ITask): + name: str + labels: Any + loss: nn.Module + per_sample_loss: nn.Module + available_func: Callable + inputs: Any + activation: Optional[nn.Module] + decoder: Decoder + module: nn.Module + metrics: Tuple[str, TorchMetric] +``` + +Alternatively, you can subclass `ITask` and implement its inferface. + +### Inspiration + +* [Andrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and Prediction](https://www.youtube.com/watch?v=IHH47nZ7FZU) +* [PyTorch at Tesla - Andrej Karpathy, Tesla](https://www.youtube.com/watch?v=oBklltKXtDE) +* [Multitask learning - Andrew Ng](https://www.youtube.com/watch?v=UdXfsAr4Gjw) + +%package -n python3-dnn-cool +Summary: DNN.Cool: Multi-task learning for Deep Neural Networks (DNN). +Provides: python-dnn-cool +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-dnn-cool +## `dnn_cool`: Deep Neural Networks for Conditional objective oriented learning + +WARNING: API is not yet stable, expect breaking changes in 0.x versions! + +To install, just do: + +```bash +pip install dnn_cool +``` + +* [Introduction](#introduction): What is `dnn_cool` in a nutshell? +* [Examples](#examples): a simple step-by-step example. +* [Features](#features): a list of the utilities that `dnn_cool` provides for you +* [Customization](#customization): Learn how to add new tasks, modify them, etc. +* [Inspiration](#inspiration): list of papers and videos which inspired this library + +To see the predefined tasks for this release, see [list of predefined tasks](#list-of-predefined-tasks) + +### Introduction + +A framework for multi-task learning in Pytorch, where you may precondition tasks and compose them into bigger tasks. +Many complex neural networks can be trivially implemented with `dnn_cool`. +For example, creating a neural network that does classification and localization is as simple as: + +```python +@project.add_flow +def localize_flow(flow, x, out): + out += flow.obj_exists(x.features) + out += flow.obj_x(x.features) | out.obj_exists + out += flow.obj_y(x.features) | out.obj_exists + out += flow.obj_w(x.features) | out.obj_exists + out += flow.obj_h(x.features) | out.obj_exists + out += flow.obj_class(x.features) | out.obj_exists + return out +``` + +If for example you want to classify first if the camera is blocked and then do localization **given that the camera +is not blocked**, you could do: + +```python +@project.add_flow +def full_flow(flow, x, out): + out += flow.camera_blocked(x.cam_features) + out += flow.localize_flow(x.localization_features) | (~out.camera_blocked) + return out +``` + +Based on these "task flows" as we call them, `dnn_cool` provides a bunch of [features](#features). +Currently, this is the list of the predefined tasks (they are all located in `dnn_cool.task_flow`): + +##### List of predefined tasks + +In the current release, the following tasks are available out of the box: + +* `BinaryClassificationTask` - sigmoid activation, thresholding decoder, binary cross entropy loss function. In the +examples above, `camera_blocked` and `obj_exists` are `BinaryClassificationTask`s. +* `ClassificationTask` - softmax activation, sorting classes decoder, categorical cross entropy loss. In the example +above, `obj_class` is a `ClassificationTask` +* `MultilabelClassificationTask` - sigmoid activation, thresholding decoder, binary cross entropy loss function. +* `BoundedRegressionTask` - sigmoid activation, rescaling decoder, mean squared error loss function. In the examples +above, `obj_x`, `obj_y`, `obj_w`, `obj_h` are bounded regression tasks. +* `MaskedLanguageModelingTask` - softmax activation, sorting decoder, cross entropy per token loss. +* `TaskFlow` - a composite task, that contains a list of children tasks. We saw 2 task flows above. + +### Examples + +#### Quick Imagenet example + +We just have to add a `ClassificationTask` named `classifier` and add the flow below: + +```python +@project.add_flow() +def imagenet_model(flow, x, out): + out += flow.classifier(x.features) + return out +``` + +That's great! But what if there is not an object always? Then we have to first check if an object exists. Let's +add a `BinaryClassificationTask` and use it as a precondition to classifier. + +```python +@project.add_flow() +def imagenet_model(flow, x, out): + out += flow.object_exists(x.features) + out += flow.classifier(x.features) | out.object_exists + return out +``` + +But what if we also want to localize the object? Then we have to add new tasks that regress the bounding box. Let's +call them `object_x`, `object_y`, `object_w`, `object_h` and make them a `BoundedRegressionTask`. To avoid +preconditioning all tasks on `object_exists`, let's group them first. Then we modify the +flow: + +```python +@project.add_flow() +def object_flow(flow, x, out): + out += flow.classifier(x.features) + out += flow.object_x(x.features) + out += flow.object_y(x.features) + out += flow.object_w(x.features) + out += flow.object_h(x.features) + return out + +@project.add_flow() +def imagenet_flow(flow, x, out): + out += flow.object_exists(x.features) + out += flow.object_flow(x.features) | out.object_exists + return out +``` + +But what if the camera is blocked? Then there is no need to do anything, so let's create a new flow +that executes our `imagenet_flow` only when the camera is not blocked. + +```python +def full_flow(flow, x, out): + out += flow.camera_blocked(x.features) + out += flow.imagenet_flow(x.features) | (~out.camera_blocked) + return out +``` + +But what if for example we want to check if the object is a kite, and if it is, to classify its color? +Then we would have to modify our `object_flow` as follows: + +```python +@project.add_flow() +def object_flow(flow, x, out): + out += flow.classifier(x.features) + out += flow.object_x(x.features) + out += flow.object_y(x.features) + out += flow.object_w(x.features) + out += flow.object_h(x.features) + out += flow.is_kite(x.features) + out += flow.color(x.features) | out.is_kite + return out +``` + +I think you can see what `dnn_cool` is meant to do! :) + +To see a full walkthrough on a synthetic dataset, check out the [Colab notebook](https://colab.research.google.com/drive/1fEidcOszTI9JXptbuU5GGC-O_yxb6hxO?usp=sharing) +or the [markdown write-up](./story.md). + +### Features + +Main features are: + +* [Task precondition](#task-preconditioning) +* [Missing values handling](#missing-values) +* [Task composition](#task-composition) +* [Tensorboard metrics logging](#tensorboard-logging) +* [Task interpretations](#task-interpretation) +* [Task evaluation](#task-evaluation) +* [Task threshold tuning](#task-threshold-tuning) +* [Dataset generation](#dataset-generation) +* [Tree explanations](#tree-explanations) +* [Memory balancing for dataparallel](#memory-balancing) + +##### Task preconditioning + +Use the `|` for task preconditioning (think of `P(A|B)` notation). Preconditioning - ` A | B` means that: + +* Include the ground truth for `B` in the input batch when training +* When training, update the weights of the `A` only when `B` is satisfied in the ground truth. +* When training, compute the loss function for `A` only when `B` is satisfied in the ground truth +* When training, compute the metrics for `A` only when `B` is satisfied in the ground truth. +* When tuning threshold for `A`, optimize only on values for which `B` is satisfied in the ground truth. +* When doing inference, compute the metrics for `A` only when the precondition is satisfied according to the decoded +result of the `B` task +* When generating tree explanation in inference mode, do not show the branch for `A` if `B` is not +satisfied. +* When computing results interpretation, include only loss terms when the precondition is satisfied. + +Usually, you have to keep track of all this stuff manually, which makes adding new preconditions very difficult. +`dnn_cool` makes this stuff easy, so that you can chain a long list of preconditions without worrying you forgot +something. + +##### Missing values + +Sometimes for an input you don't have labels for all tasks. With `dnn_cool`, you can just mark the missing label and +`dnn_cool` will update only the weights of the tasks for which labels are available. + +This feature has the awesome property that you don't need a single dataset with all tasks labeled, you can +have different datasets for different tasks and it will work. For example, you can train a single object detection +neural network that trains its classifier head on ImageNet, and its detection head on COCO. + +##### Task composition + +You can group tasks in a task flow (we already saw 2 above - `localize_flow` and `full_flow`). You can use this to +organize things better, for example when you want to precondition a whole task flow. For example: + +```python +@project.add_flow +def face_regression(flow, x, out): + out += flow.face_x1(x.face_localization) + out += flow.face_y1(x.face_localization) + out += flow.face_w(x.face_localization) + out += flow.face_h(x.face_localization) + out += flow.facial_characteristics(x.features) + return out +``` + +##### Tensorboard logging + +`dnn_cool` logs the metrics per task in Tensorboard, e.g: + + + +##### Task interpretation + +Also, the best and worst inputs per task are logged in the Tensorboard, for example if the input is an image: + +  + +##### Task evaluation + +Per-task evaluation information is available, to pinpoint the exact problem in your network. An example +evaluation dataframe: + +| | task_path | metric_name | metric_res | num_samples | +|---:|:---------------------------------------------------------|:--------------------|-------------:|--------------:| +| 0 | camera_blocked | accuracy | 0.980326 | 996 | +| 1 | camera_blocked | f1_score | 0.974368 | 996 | +| 2 | camera_blocked | precision | 0.946635 | 996 | +| 3 | camera_blocked | recall | 0.960107 | 996 | +| 4 | door_open | accuracy | 0.921215 | 902 | +| 5 | door_open | f1_score | 0.966859 | 902 | +| 6 | door_open | precision | 0.976749 | 902 | +| 7 | door_open | recall | 0.939038 | 902 | +| 8 | door_locked | accuracy | 0.983039 | 201 | +| 9 | door_locked | f1_score | 0.948372 | 201 | +| 10 | door_locked | precision | 0.982583 | 201 | +| 11 | door_locked | recall | 0.934788 | 201 | +| 12 | person_present | accuracy | 0.999166 | 701 | +| 13 | person_present | f1_score | 0.937541 | 701 | +| 14 | person_present | precision | 0.927337 | 701 | +| 15 | person_present | recall | 0.963428 | 701 | +| 16 | person_regression.face_regression.face_x1 | mean_absolute_error | 0.0137292 | 611 | +| 17 | person_regression.face_regression.face_y1 | mean_absolute_error | 0.0232761 | 611 | +| 18 | person_regression.face_regression.face_w | mean_absolute_error | 0.00740503 | 611 | +| 19 | person_regression.face_regression.face_h | mean_absolute_error | 0.0101 | 611 | +| 20 | person_regression.face_regression.facial_characteristics | accuracy | 0.932624 | 611 | +| 21 | person_regression.body_regression.body_x1 | mean_absolute_error | 0.00830785 | 611 | +| 22 | person_regression.body_regression.body_y1 | mean_absolute_error | 0.0151234 | 611 | +| 23 | person_regression.body_regression.body_w | mean_absolute_error | 0.0130214 | 611 | +| 24 | person_regression.body_regression.body_h | mean_absolute_error | 0.0101 | 611 | +| 25 | person_regression.body_regression.shirt_type | accuracy_1 | 0.979934 | 611 | +| 26 | person_regression.body_regression.shirt_type | accuracy_3 | 0.993334 | 611 | +| 27 | person_regression.body_regression.shirt_type | accuracy_5 | 0.990526 | 611 | +| 28 | person_regression.body_regression.shirt_type | f1_score | 0.928516 | 611 | +| 29 | person_regression.body_regression.shirt_type | precision | 0.959826 | 611 | +| 30 | person_regression.body_regression.shirt_type | recall | 0.968146 | 611 | + +##### Task threshold tuning + +Many tasks need to tune their threshold. Just call `flow.get_decoder().tune()` and you will get optimized thresholds +for the metric you define. + +##### Dataset generation + +As noted above, `dnn_cool` will automatically trace the tasks used as a precondition and include the ground truth for +them under the key `gt`. + +##### Tree explanations + +Examples: + +``` +├── inp 1 +│ └── camera_blocked | decoded: [False], activated: [0.], logits: [-117.757324] +│ └── door_open | decoded: [ True], activated: [1.], logits: [41.11258] +│ └── person_present | decoded: [ True], activated: [1.], logits: [60.38873] +│ └── person_regression +│ ├── body_regression +│ │ ├── body_h | decoded: [29.672623], activated: [0.46363473], logits: [-0.14571853] +│ │ ├── body_w | decoded: [12.86382], activated: [0.20099719], logits: [-1.3800735] +│ │ ├── body_x1 | decoded: [21.34288], activated: [0.3334825], logits: [-0.69247603] +│ │ ├── body_y1 | decoded: [18.468979], activated: [0.2885778], logits: [-0.9023013] +│ │ └── shirt_type | decoded: [6 1 0 4 2 5 3], activated: [4.1331367e-23 3.5493638e-17 3.1328378e-26 5.6903808e-30 2.4471377e-25 + 2.8071076e-29 1.0000000e+00], logits: [-20.549513 -6.88627 -27.734364 -36.34787 -25.6788 -34.751904 + 30.990908] +│ └── face_regression +│ ├── face_h | decoded: [11.265154], activated: [0.17601803], logits: [-1.5435623] +│ ├── face_w | decoded: [12.225838], activated: [0.19102871], logits: [-1.4433397] +│ ├── face_x1 | decoded: [21.98834], activated: [0.34356782], logits: [-0.64743483] +│ ├── face_y1 | decoded: [3.2855165], activated: [0.0513362], logits: [-2.9166584] +│ └── facial_characteristics | decoded: [ True False True], activated: [9.9999940e-01 1.2074912e-12 9.9999833e-01], logits: [ 14.240071 -27.442476 13.27557 ] + +``` + +but if the model thinks the camera is blocked, then the explanation would be: + +``` +├── inp 2 +│ └── camera_blocked | decoded: [ True], activated: [1.], logits: [76.367676] +``` + +##### Memory balancing + +When using [nn.DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html), the computation of +the loss function is done on the main GPU, which leads to dramatically unbalanced memory usage if your outputs are big +and you have a lot of metrics (e.g segmentation masks, language modeling, etc). `dnn_cool` gives you a +convenient way to balance the memory in such situations - just a single `balance_dataparallel_memory = True` handles +this case for you by first reducing all metrics on their respective device, and then additionally aggregating +the results that were reduced on each device automatically. Here's an example memory usage: + + +Before: + + + +After: + + + +### Customization + +Since `flow.torch()` returns a normal `nn.Module`, you can use any library you are used to. If you use +[Catalyst](https://github.com/catalyst-team/catalyst), `dnn_cool` provides a bunch of useful callbacks. Creating +a new task is as simple as creating a new instance of this dataclass: + +```python +@dataclass +class Task(ITask): + name: str + labels: Any + loss: nn.Module + per_sample_loss: nn.Module + available_func: Callable + inputs: Any + activation: Optional[nn.Module] + decoder: Decoder + module: nn.Module + metrics: Tuple[str, TorchMetric] +``` + +Alternatively, you can subclass `ITask` and implement its inferface. + +### Inspiration + +* [Andrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and Prediction](https://www.youtube.com/watch?v=IHH47nZ7FZU) +* [PyTorch at Tesla - Andrej Karpathy, Tesla](https://www.youtube.com/watch?v=oBklltKXtDE) +* [Multitask learning - Andrew Ng](https://www.youtube.com/watch?v=UdXfsAr4Gjw) + +%package help +Summary: Development documents and examples for dnn-cool +Provides: python3-dnn-cool-doc +%description help +## `dnn_cool`: Deep Neural Networks for Conditional objective oriented learning + +WARNING: API is not yet stable, expect breaking changes in 0.x versions! + +To install, just do: + +```bash +pip install dnn_cool +``` + +* [Introduction](#introduction): What is `dnn_cool` in a nutshell? +* [Examples](#examples): a simple step-by-step example. +* [Features](#features): a list of the utilities that `dnn_cool` provides for you +* [Customization](#customization): Learn how to add new tasks, modify them, etc. +* [Inspiration](#inspiration): list of papers and videos which inspired this library + +To see the predefined tasks for this release, see [list of predefined tasks](#list-of-predefined-tasks) + +### Introduction + +A framework for multi-task learning in Pytorch, where you may precondition tasks and compose them into bigger tasks. +Many complex neural networks can be trivially implemented with `dnn_cool`. +For example, creating a neural network that does classification and localization is as simple as: + +```python +@project.add_flow +def localize_flow(flow, x, out): + out += flow.obj_exists(x.features) + out += flow.obj_x(x.features) | out.obj_exists + out += flow.obj_y(x.features) | out.obj_exists + out += flow.obj_w(x.features) | out.obj_exists + out += flow.obj_h(x.features) | out.obj_exists + out += flow.obj_class(x.features) | out.obj_exists + return out +``` + +If for example you want to classify first if the camera is blocked and then do localization **given that the camera +is not blocked**, you could do: + +```python +@project.add_flow +def full_flow(flow, x, out): + out += flow.camera_blocked(x.cam_features) + out += flow.localize_flow(x.localization_features) | (~out.camera_blocked) + return out +``` + +Based on these "task flows" as we call them, `dnn_cool` provides a bunch of [features](#features). +Currently, this is the list of the predefined tasks (they are all located in `dnn_cool.task_flow`): + +##### List of predefined tasks + +In the current release, the following tasks are available out of the box: + +* `BinaryClassificationTask` - sigmoid activation, thresholding decoder, binary cross entropy loss function. In the +examples above, `camera_blocked` and `obj_exists` are `BinaryClassificationTask`s. +* `ClassificationTask` - softmax activation, sorting classes decoder, categorical cross entropy loss. In the example +above, `obj_class` is a `ClassificationTask` +* `MultilabelClassificationTask` - sigmoid activation, thresholding decoder, binary cross entropy loss function. +* `BoundedRegressionTask` - sigmoid activation, rescaling decoder, mean squared error loss function. In the examples +above, `obj_x`, `obj_y`, `obj_w`, `obj_h` are bounded regression tasks. +* `MaskedLanguageModelingTask` - softmax activation, sorting decoder, cross entropy per token loss. +* `TaskFlow` - a composite task, that contains a list of children tasks. We saw 2 task flows above. + +### Examples + +#### Quick Imagenet example + +We just have to add a `ClassificationTask` named `classifier` and add the flow below: + +```python +@project.add_flow() +def imagenet_model(flow, x, out): + out += flow.classifier(x.features) + return out +``` + +That's great! But what if there is not an object always? Then we have to first check if an object exists. Let's +add a `BinaryClassificationTask` and use it as a precondition to classifier. + +```python +@project.add_flow() +def imagenet_model(flow, x, out): + out += flow.object_exists(x.features) + out += flow.classifier(x.features) | out.object_exists + return out +``` + +But what if we also want to localize the object? Then we have to add new tasks that regress the bounding box. Let's +call them `object_x`, `object_y`, `object_w`, `object_h` and make them a `BoundedRegressionTask`. To avoid +preconditioning all tasks on `object_exists`, let's group them first. Then we modify the +flow: + +```python +@project.add_flow() +def object_flow(flow, x, out): + out += flow.classifier(x.features) + out += flow.object_x(x.features) + out += flow.object_y(x.features) + out += flow.object_w(x.features) + out += flow.object_h(x.features) + return out + +@project.add_flow() +def imagenet_flow(flow, x, out): + out += flow.object_exists(x.features) + out += flow.object_flow(x.features) | out.object_exists + return out +``` + +But what if the camera is blocked? Then there is no need to do anything, so let's create a new flow +that executes our `imagenet_flow` only when the camera is not blocked. + +```python +def full_flow(flow, x, out): + out += flow.camera_blocked(x.features) + out += flow.imagenet_flow(x.features) | (~out.camera_blocked) + return out +``` + +But what if for example we want to check if the object is a kite, and if it is, to classify its color? +Then we would have to modify our `object_flow` as follows: + +```python +@project.add_flow() +def object_flow(flow, x, out): + out += flow.classifier(x.features) + out += flow.object_x(x.features) + out += flow.object_y(x.features) + out += flow.object_w(x.features) + out += flow.object_h(x.features) + out += flow.is_kite(x.features) + out += flow.color(x.features) | out.is_kite + return out +``` + +I think you can see what `dnn_cool` is meant to do! :) + +To see a full walkthrough on a synthetic dataset, check out the [Colab notebook](https://colab.research.google.com/drive/1fEidcOszTI9JXptbuU5GGC-O_yxb6hxO?usp=sharing) +or the [markdown write-up](./story.md). + +### Features + +Main features are: + +* [Task precondition](#task-preconditioning) +* [Missing values handling](#missing-values) +* [Task composition](#task-composition) +* [Tensorboard metrics logging](#tensorboard-logging) +* [Task interpretations](#task-interpretation) +* [Task evaluation](#task-evaluation) +* [Task threshold tuning](#task-threshold-tuning) +* [Dataset generation](#dataset-generation) +* [Tree explanations](#tree-explanations) +* [Memory balancing for dataparallel](#memory-balancing) + +##### Task preconditioning + +Use the `|` for task preconditioning (think of `P(A|B)` notation). Preconditioning - ` A | B` means that: + +* Include the ground truth for `B` in the input batch when training +* When training, update the weights of the `A` only when `B` is satisfied in the ground truth. +* When training, compute the loss function for `A` only when `B` is satisfied in the ground truth +* When training, compute the metrics for `A` only when `B` is satisfied in the ground truth. +* When tuning threshold for `A`, optimize only on values for which `B` is satisfied in the ground truth. +* When doing inference, compute the metrics for `A` only when the precondition is satisfied according to the decoded +result of the `B` task +* When generating tree explanation in inference mode, do not show the branch for `A` if `B` is not +satisfied. +* When computing results interpretation, include only loss terms when the precondition is satisfied. + +Usually, you have to keep track of all this stuff manually, which makes adding new preconditions very difficult. +`dnn_cool` makes this stuff easy, so that you can chain a long list of preconditions without worrying you forgot +something. + +##### Missing values + +Sometimes for an input you don't have labels for all tasks. With `dnn_cool`, you can just mark the missing label and +`dnn_cool` will update only the weights of the tasks for which labels are available. + +This feature has the awesome property that you don't need a single dataset with all tasks labeled, you can +have different datasets for different tasks and it will work. For example, you can train a single object detection +neural network that trains its classifier head on ImageNet, and its detection head on COCO. + +##### Task composition + +You can group tasks in a task flow (we already saw 2 above - `localize_flow` and `full_flow`). You can use this to +organize things better, for example when you want to precondition a whole task flow. For example: + +```python +@project.add_flow +def face_regression(flow, x, out): + out += flow.face_x1(x.face_localization) + out += flow.face_y1(x.face_localization) + out += flow.face_w(x.face_localization) + out += flow.face_h(x.face_localization) + out += flow.facial_characteristics(x.features) + return out +``` + +##### Tensorboard logging + +`dnn_cool` logs the metrics per task in Tensorboard, e.g: + + + +##### Task interpretation + +Also, the best and worst inputs per task are logged in the Tensorboard, for example if the input is an image: + +  + +##### Task evaluation + +Per-task evaluation information is available, to pinpoint the exact problem in your network. An example +evaluation dataframe: + +| | task_path | metric_name | metric_res | num_samples | +|---:|:---------------------------------------------------------|:--------------------|-------------:|--------------:| +| 0 | camera_blocked | accuracy | 0.980326 | 996 | +| 1 | camera_blocked | f1_score | 0.974368 | 996 | +| 2 | camera_blocked | precision | 0.946635 | 996 | +| 3 | camera_blocked | recall | 0.960107 | 996 | +| 4 | door_open | accuracy | 0.921215 | 902 | +| 5 | door_open | f1_score | 0.966859 | 902 | +| 6 | door_open | precision | 0.976749 | 902 | +| 7 | door_open | recall | 0.939038 | 902 | +| 8 | door_locked | accuracy | 0.983039 | 201 | +| 9 | door_locked | f1_score | 0.948372 | 201 | +| 10 | door_locked | precision | 0.982583 | 201 | +| 11 | door_locked | recall | 0.934788 | 201 | +| 12 | person_present | accuracy | 0.999166 | 701 | +| 13 | person_present | f1_score | 0.937541 | 701 | +| 14 | person_present | precision | 0.927337 | 701 | +| 15 | person_present | recall | 0.963428 | 701 | +| 16 | person_regression.face_regression.face_x1 | mean_absolute_error | 0.0137292 | 611 | +| 17 | person_regression.face_regression.face_y1 | mean_absolute_error | 0.0232761 | 611 | +| 18 | person_regression.face_regression.face_w | mean_absolute_error | 0.00740503 | 611 | +| 19 | person_regression.face_regression.face_h | mean_absolute_error | 0.0101 | 611 | +| 20 | person_regression.face_regression.facial_characteristics | accuracy | 0.932624 | 611 | +| 21 | person_regression.body_regression.body_x1 | mean_absolute_error | 0.00830785 | 611 | +| 22 | person_regression.body_regression.body_y1 | mean_absolute_error | 0.0151234 | 611 | +| 23 | person_regression.body_regression.body_w | mean_absolute_error | 0.0130214 | 611 | +| 24 | person_regression.body_regression.body_h | mean_absolute_error | 0.0101 | 611 | +| 25 | person_regression.body_regression.shirt_type | accuracy_1 | 0.979934 | 611 | +| 26 | person_regression.body_regression.shirt_type | accuracy_3 | 0.993334 | 611 | +| 27 | person_regression.body_regression.shirt_type | accuracy_5 | 0.990526 | 611 | +| 28 | person_regression.body_regression.shirt_type | f1_score | 0.928516 | 611 | +| 29 | person_regression.body_regression.shirt_type | precision | 0.959826 | 611 | +| 30 | person_regression.body_regression.shirt_type | recall | 0.968146 | 611 | + +##### Task threshold tuning + +Many tasks need to tune their threshold. Just call `flow.get_decoder().tune()` and you will get optimized thresholds +for the metric you define. + +##### Dataset generation + +As noted above, `dnn_cool` will automatically trace the tasks used as a precondition and include the ground truth for +them under the key `gt`. + +##### Tree explanations + +Examples: + +``` +├── inp 1 +│ └── camera_blocked | decoded: [False], activated: [0.], logits: [-117.757324] +│ └── door_open | decoded: [ True], activated: [1.], logits: [41.11258] +│ └── person_present | decoded: [ True], activated: [1.], logits: [60.38873] +│ └── person_regression +│ ├── body_regression +│ │ ├── body_h | decoded: [29.672623], activated: [0.46363473], logits: [-0.14571853] +│ │ ├── body_w | decoded: [12.86382], activated: [0.20099719], logits: [-1.3800735] +│ │ ├── body_x1 | decoded: [21.34288], activated: [0.3334825], logits: [-0.69247603] +│ │ ├── body_y1 | decoded: [18.468979], activated: [0.2885778], logits: [-0.9023013] +│ │ └── shirt_type | decoded: [6 1 0 4 2 5 3], activated: [4.1331367e-23 3.5493638e-17 3.1328378e-26 5.6903808e-30 2.4471377e-25 + 2.8071076e-29 1.0000000e+00], logits: [-20.549513 -6.88627 -27.734364 -36.34787 -25.6788 -34.751904 + 30.990908] +│ └── face_regression +│ ├── face_h | decoded: [11.265154], activated: [0.17601803], logits: [-1.5435623] +│ ├── face_w | decoded: [12.225838], activated: [0.19102871], logits: [-1.4433397] +│ ├── face_x1 | decoded: [21.98834], activated: [0.34356782], logits: [-0.64743483] +│ ├── face_y1 | decoded: [3.2855165], activated: [0.0513362], logits: [-2.9166584] +│ └── facial_characteristics | decoded: [ True False True], activated: [9.9999940e-01 1.2074912e-12 9.9999833e-01], logits: [ 14.240071 -27.442476 13.27557 ] + +``` + +but if the model thinks the camera is blocked, then the explanation would be: + +``` +├── inp 2 +│ └── camera_blocked | decoded: [ True], activated: [1.], logits: [76.367676] +``` + +##### Memory balancing + +When using [nn.DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html), the computation of +the loss function is done on the main GPU, which leads to dramatically unbalanced memory usage if your outputs are big +and you have a lot of metrics (e.g segmentation masks, language modeling, etc). `dnn_cool` gives you a +convenient way to balance the memory in such situations - just a single `balance_dataparallel_memory = True` handles +this case for you by first reducing all metrics on their respective device, and then additionally aggregating +the results that were reduced on each device automatically. Here's an example memory usage: + + +Before: + + + +After: + + + +### Customization + +Since `flow.torch()` returns a normal `nn.Module`, you can use any library you are used to. If you use +[Catalyst](https://github.com/catalyst-team/catalyst), `dnn_cool` provides a bunch of useful callbacks. Creating +a new task is as simple as creating a new instance of this dataclass: + +```python +@dataclass +class Task(ITask): + name: str + labels: Any + loss: nn.Module + per_sample_loss: nn.Module + available_func: Callable + inputs: Any + activation: Optional[nn.Module] + decoder: Decoder + module: nn.Module + metrics: Tuple[str, TorchMetric] +``` + +Alternatively, you can subclass `ITask` and implement its inferface. + +### Inspiration + +* [Andrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and Prediction](https://www.youtube.com/watch?v=IHH47nZ7FZU) +* [PyTorch at Tesla - Andrej Karpathy, Tesla](https://www.youtube.com/watch?v=oBklltKXtDE) +* [Multitask learning - Andrew Ng](https://www.youtube.com/watch?v=UdXfsAr4Gjw) + +%prep +%autosetup -n dnn-cool-0.4.0 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-dnn-cool -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Wed May 10 2023 Python_Bot <Python_Bot@openeuler.org> - 0.4.0-1 +- Package Spec generated @@ -0,0 +1 @@ +52ccf820465908a2a9b5e183ba85580c dnn_cool-0.4.0.tar.gz |