diff options
Diffstat (limited to 'python-ck.spec')
| -rw-r--r-- | python-ck.spec | 960 |
1 files changed, 960 insertions, 0 deletions
diff --git a/python-ck.spec b/python-ck.spec new file mode 100644 index 0000000..1e7d19d --- /dev/null +++ b/python-ck.spec @@ -0,0 +1,960 @@ +%global _empty_manifest_terminate_build 0 +Name: python-ck +Version: 2.6.3 +Release: 1 +Summary: Collective Knowledge - a lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows based on FAIR principles +License: Apache 2.0 +URL: https://github.com/mlcommons/ck +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/f4/29/b4c314545e53f113816b2965ce6b457272dff06d7ba4a346ad1196cb60cb/ck-2.6.3.tar.gz +BuildArch: noarch + + +%description +*Note that the 1st generation of the CK framework was discontinued in summer 2022 + after the [2nd generation of this framework (CM)](https://github.com/mlcommons/ck) + was released by the [open taskforce on education and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md) + at [MLCommons](https://mlcommons.org).* + + +# Collective Knowledge framework (CK) + +[](https://pepy.tech/project/ck) +[](https://badge.fury.io/py/ck) +[](https://pypi.org/project/ck) + +[](https://travis-ci.com/ctuning/ck) +[](https://ci.appveyor.com/project/gfursin/ck) +[](https://coveralls.io/github/ctuning/ck) + +[](https://ck.readthedocs.io/en/latest/?badge=latest) +[](https://colab.research.google.com/drive/1Fp6uxCqTazmCSSl8v-nY93VVmcOoLiXi?usp=sharing) +[](https://colab.research.google.com/drive/1a761nKgoHlJAy6gOXh-c9H4WkLV8nzRU?usp=sharing) + + +## News + + + +* **2022 July 17:** We have pre-released CK2-based MLOps and DevOps automation scripts at https://github.com/mlcommons/ck/tree/master/cm-mlops/script + +* **2022 May:** We started developing the 2nd generation of the CK framework (aka CM): https://github.com/mlcommons/ck/tree/master/cm + +* **2022 April 3:** We presented the CK concept to bridge the growing gap between ML Systems research and production + at the HPCA'22 workshop on [benchmarking deep learning systems](https://sites.google.com/g.harvard.edu/mlperf-bench-hpca22/home). + +* **2022 March:** We presented the [CK concept to enable collaborative and reproducible ML Systems R&D](https://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=73126) + at the SIAM'22 workshop on "Research Challenges and Opportunities within Software Productivity, Sustainability, and Reproducibility" + +* **2022 March:** we've released the first prototype of [the Collective Mind toolkit (CK2)](https://github.com/mlcommons/ck/tree/master/ck2) + based on your feedback and our practical experience [reproducing 150+ ML and Systems papers and validating them in the real world](https://www.youtube.com/watch?v=7zpeIVwICa4). + +## Motivation + +While Machine Learning is becoming more and more important in everyday life, designing efficient ML Systems and deploying them in the real world +is becoming increasingly challenging, time consuming and costly. +Researchers and engineers must keep pace with rapidly evolving software stacks and a Cambrian explosion of hardware platforms from the cloud to the edge. +Such platforms have their own specific libraries, frameworks, APIs and specifications and often require repetitive, tedious and ad-hoc optimization +of the whole model/software/hardware stack to trade off accuracy, latency, throughout, power consumption, size and costs depending on user requirements and constraints. + +### The CK framework + +*The Collective Knowledge framework (CK)* is our attempt to develop a common plug&play infrastructure that can be used +by the community similar to Wikipedia to learn how to solve above challenges and make it easier to co-design, +benchmark, optimize and deploy Machine Learning Systems in the real world across continuously evolving software, +hardware and data sets (see our [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4) for more details): + +* CK aims at providing a simple playground with minimal software dependencies to help researchers and practitioners share their knowledge + in the form of reusable automation recipes with a unified Python API, CLI and meta description: + - [Stable CK automation recipes](https://github.com/mlcommons/ck/tree/master/ck/repo/module) + - [MLPerf™ benchmark automation recipes](https://github.com/mlcommons/ck/tree/master/ck-mlops/repo/module) + +* CK helps to organize software projects and Git repositories as a database of above automation recipes + and related artifacts based on [FAIR principles](https://www.nature.com/articles/sdata201618) + as described in our [journal article](https://arxiv.org/pdf/2011.01149.pdf) ([shorter pre-print]( https://arxiv.org/abs/2006.07161 )). + See examples of CK-compatible GitHub repositories: + - [MLPerf/MLOps automation](https://github.com/mlcommons/ck-mlops) + - [ACM REQUEST tournament for collaborative and reproducible ML/SW/HW co-design](https://github.com/ctuning/ck-request) + +### Community developments + +We collaborated with the community to reproduce [150+ ML and Systems papers](https://cTuning.org/ae) +and implement the following reusable automation recipes in the CK format: + +* Portable meta package manager to automatically detect, install or rebuild various ML artifacts + (ML models, data sets, frameworks, libraries, etc) across different platform and operating systems including Linux, Windows, MacOS and Android: + - [ML artifact detection plugins](https://github.com/mlcommons/ck-mlops/tree/main/soft) + - [ML meta package installation plugins](https://github.com/mlcommons/ck-mlops/tree/main/package) + - OS descriptions: [Linux/MacOS/Android](https://github.com/mlcommons/ck-mlops/tree/main/os) ; [Windows](https://github.com/ctuning/ck-win/tree/main/os) + +* Portable manager for Python virtual environments: [CK repo](https://github.com/mlcommons/ck-venv). + +* Portable workflows to support collaborative, reproducible and cross-platform benchmarking: + - [ML Systems benchmarking](https://github.com/mlcommons/ck-mlops/tree/main/program) + - [Compiler benchmarking](https://github.com/ctuning/ctuning-programs/tree/master/program) + +* Portable workflows to automate MLPerf™ benchmark: + - [End-to-end submission suite used by multiple organizations to automate the submission of MLPerf inference benchmark](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/README.md) + - MLPerf inference v1.1 results: [MLCommons press-release](https://mlcommons.org/en/news/mlperf-inference-v11), + [Datacenter results](https://mlcommons.org/en/inference-datacenter-11), + [Edge results](https://mlcommons.org/en/inference-edge-11) + - [Reproducibility studies for MLPerf inference benchmark v1.1 automated by CK](https://github.com/mlcommons/ck/tree/master/docs/mlperf-automation/reproduce#reproducibility-reports-mlperf-inference-benchmark-v11) + - [Design space exploration of ML/SW/HW stacks and customizable visualization](https://cknowledge.io/result/crowd-benchmarking-mlperf-inference-classification-mobilenets-all) + + +Please contact [Grigori Fursin](https://www.linkedin.com/in/grigorifursin) if you are interested to join this community effort! + +### Tutorials + +* [CK automations for unified benchmarking]( https://colab.research.google.com/drive/1a761nKgoHlJAy6gOXh-c9H4WkLV8nzRU?usp=sharing ) +* [CK-based MLPerf inference benchmark automation example]( https://colab.research.google.com/drive/1Fp6uxCqTazmCSSl8v-nY93VVmcOoLiXi?usp=sharing ) + * [CK-based MLPerf inference vision benchmark v1.1 automation (TVM)]( https://colab.research.google.com/drive/1aywGlyD1ZRDtQTrQARVgL1882JcvmFK-?usp=sharing ) + * [CK-based MLPerf inference vision benchmark v1.1 automation (ONNX)]( https://colab.research.google.com/drive/1ij1rWoqje5-Sn6UsdFj1OzYakudI2RIS?usp=sharing ) +* [CK basics]( https://colab.research.google.com/drive/15lQgxuTSkEPqi4plaat1_v2gJcfIrATF?usp=sharing ) + +## Releases + +### Development version + +We are developing the 2nd generation of the CK framework (aka CM) based on your feedback: +* [CK2(CM) framework](https://github.com/mlcommons/ck/tree/master/cm) +* [CK2(CM) MLOps and DevOps automation scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops) + +### Stable versions + +The latest version of the CK automation suite supported by MLCommons™: +* [CK framework v2.6.1 (Apache 2.0 license)](https://github.com/mlcommons/ck/releases/tag/V2.6.1) +* [CK automation suite for MLPerf™ and ML/SW/HW co-design](https://github.com/mlcommons/ck-mlops) + + +## Current projects +* [Automating MLPerf(tm) inference benchmark and packing ML models, data sets and frameworks as CK components with a unified API and meta description](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/README.md) +* Developing customizable dashboards for MLPerf™ to help end-users select ML/SW/HW stacks on a Pareto frontier: [aggregated MLPerf™ results]( https://cknowledge.io/?q="mlperf-inference-all" ) +* Providing a common format to share artifacts at ML, systems and other conferences: [video](https://youtu.be/DIkZxraTmGM), [Artifact Evaluation](https://cTuning.org/ae) +* Redesigning CK together with the community based on user feedback: [incubator](https://github.com/mlcommons/ck/tree/master/incubator) +* [Other real-world use cases](https://cKnowledge.org/partners.html) from MLPerf™, Qualcomm, Arm, General Motors, IBM, the Raspberry Pi foundation, ACM and other great partners; + +## Documentation + +* [Online CK documentation]( https://ck.readthedocs.io ) + * [Why CK?]( https://ck.readthedocs.io/en/latest/src/introduction.html ) + * [CK Basics](https://michel-steuwer.github.io/About-CK) + * [Try CK]( https://ck.readthedocs.io/en/latest/src/first-steps.html ) +* [Publications](https://github.com/mlcommons/ck/wiki/Publications) + +## Installation + +Follow [this guide](https://ck.readthedocs.io/en/latest/src/installation.html) +to install CK framework on your platform. + +CK supports the following platforms: + +| | As a host platform | As a target platform | +|---------------|:------------------:|:--------------------:| +| Generic Linux | ✓ | ✓ | +| Linux (Arm) | ✓ | ✓ | +| Raspberry Pi | ✓ | ✓ | +| MacOS | ✓ | ± | +| Windows | ✓ | ✓ | +| Android | ± | ✓ | +| iOS | TBD | TBD | +| Bare-metal (edge devices) | - | ± | + +## Examples + +### Portable CK workflow (native environment without Docker) + +Here we show how to pull a GitHub repo in the CK format +and use a unified CK interface to compile and run +any program (image corner detection in our case) +with any compatible data set on any compatible platform: + +```bash +python3 -m pip install ck + +ck pull repo:mlcommons@ck-mlops + +ck ls program:*susan* + +ck search dataset --tags=jpeg + +ck pull repo:ctuning-datasets-min + +ck search dataset --tags=jpeg + +ck detect soft:compiler.gcc +ck detect soft:compiler.llvm + +ck show env --tags=compiler + +ck compile program:image-corner-detection --speed + +ck run program:image-corner-detection --repeat=1 --env.MY_ENV=123 --env.TEST=xyz +``` + +You can check output of this program in the following directory: +```bash +cd `ck find program:image-corner-detection`/tmp +ls + +processed-image.pgm +``` + +You can now view this image with detected corners. + + +Check [CK docs](https://ck.readthedocs.io/en/latest/src/introduction.html) for further details. + +### MLPerf™ benchmark workflows + +* [Current coverage](https://github.com/mlcommons/ck/tree/master/docs/mlperf-automation#readme) +* [MLPerf inference v1.1 workflows](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/reproduce/README.md#reproducibility-reports-mlperf-inference-benchmark-v11) + +### Portable CK workflows inside containers + +We have prepared adaptive CK containers to demonstrate MLOps capabilities: +* https://github.com/mlcommons/ck-mlops/tree/main/docker + +You can run them as follows: + +```bash +ck pull repo:mlcommons@ck-mlops +ck build docker:ck-template-mlperf --tag=ubuntu-20.04 +ck run docker:ck-template-mlperf --tag=ubuntu-20.04 +``` + +### Portable workflow example with virtual CK environments + +You can create multiple [virtual CK environments](https://github.com/mlcommons/ck-venv) with templates +to automatically install different CK packages and workflows, for example for MLPerf™ inference: + +``` +ck pull repo:mlcommons@ck-venv +ck create venv:test --template=mlperf-inference-main +ck ls venv +ck activate venv:test + +ck pull repo:mlcommons@ck-mlops +ck install package --ask --tags=dataset,coco,val,2017,full +ck show env + +``` + +### Integration with web services and CI platforms + +All CK modules, automation actions and workflows are accessible as a micro-service with a unified JSON I/O API +to make it easier to integrate them with web services and CI platforms as described +[here](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/tools/continuous-integration.md). + + + + +### Other use cases + +* [List of various use cases]( https://ck.readthedocs.io/en/latest/src/introduction.html#ck-showroom ) + + + + + + + +## CK portal + +We have developed the [cKnowledge.io portal](https://cKnowledge.io) to help the community +organize and find all the CK workflows and components similar to PyPI: + +* [Search CK components](https://cKnowledge.io) +* [Browse CK components](https://cKnowledge.io/browse) +* [Find reproduced results from papers]( https://cKnowledge.io/reproduced-results ) +* [Test CK workflows to benchmark and optimize ML Systems]( https://cKnowledge.io/demo ) + + + +## Containers to test CK automation recipes and workflows + +The community provides Docker containers to test CK and components using different ML/SW/HW stacks (DSE). + +* A set of Docker containers to test the basic CK functionality + using some MLPerf inference benchmark workflows: + https://github.com/mlcommons/ck-mlops/tree/main/docker/test-ck + + +## Contributions + +Users can extend the CK functionality via [CK modules](https://github.com/mlcommons/ck/tree/master/ck/repo/module) +or external [GitHub reposities](https://cKnowledge.io/repos) in the CK format +as described [here](https://ck.readthedocs.io/en/latest/src/typical-usage.html). + +Please check [this documentation](https://ck.readthedocs.io/en/latest/src/how-to-contribute.html) +if you want to extend the CK core functionality and [modules](https://github.com/mlcommons/ck/tree/master/ck/repo/module). + +Note, that we plan to [redesign the CK framework](https://github.com/mlcommons/ck/projects/1) +to be more pythonic (we wrote the first prototype without OO to be able +to port it to bare-metal devices in C but eventually we decided to drop this idea). + +Please contact [Grigori Fursin](mailto:grigori@octoml.ai) to join this community effort. + + + +## Contacts + +* [Grigori Fursin](https://fursin.net) - author +* [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh) - maintainer + +## Acknowledgments + +We would like to thank all [contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) +and [collaborators](https://cKnowledge.org/partners.html) for their support, fruitful discussions, +and useful feedback! See more acknowledgments in the [CK journal article](https://arxiv.org/abs/2011.01149) +and [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4). + +%package -n python3-ck +Summary: Collective Knowledge - a lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows based on FAIR principles +Provides: python-ck +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-ck +*Note that the 1st generation of the CK framework was discontinued in summer 2022 + after the [2nd generation of this framework (CM)](https://github.com/mlcommons/ck) + was released by the [open taskforce on education and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md) + at [MLCommons](https://mlcommons.org).* + + +# Collective Knowledge framework (CK) + +[](https://pepy.tech/project/ck) +[](https://badge.fury.io/py/ck) +[](https://pypi.org/project/ck) + +[](https://travis-ci.com/ctuning/ck) +[](https://ci.appveyor.com/project/gfursin/ck) +[](https://coveralls.io/github/ctuning/ck) + +[](https://ck.readthedocs.io/en/latest/?badge=latest) +[](https://colab.research.google.com/drive/1Fp6uxCqTazmCSSl8v-nY93VVmcOoLiXi?usp=sharing) +[](https://colab.research.google.com/drive/1a761nKgoHlJAy6gOXh-c9H4WkLV8nzRU?usp=sharing) + + +## News + + + +* **2022 July 17:** We have pre-released CK2-based MLOps and DevOps automation scripts at https://github.com/mlcommons/ck/tree/master/cm-mlops/script + +* **2022 May:** We started developing the 2nd generation of the CK framework (aka CM): https://github.com/mlcommons/ck/tree/master/cm + +* **2022 April 3:** We presented the CK concept to bridge the growing gap between ML Systems research and production + at the HPCA'22 workshop on [benchmarking deep learning systems](https://sites.google.com/g.harvard.edu/mlperf-bench-hpca22/home). + +* **2022 March:** We presented the [CK concept to enable collaborative and reproducible ML Systems R&D](https://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=73126) + at the SIAM'22 workshop on "Research Challenges and Opportunities within Software Productivity, Sustainability, and Reproducibility" + +* **2022 March:** we've released the first prototype of [the Collective Mind toolkit (CK2)](https://github.com/mlcommons/ck/tree/master/ck2) + based on your feedback and our practical experience [reproducing 150+ ML and Systems papers and validating them in the real world](https://www.youtube.com/watch?v=7zpeIVwICa4). + +## Motivation + +While Machine Learning is becoming more and more important in everyday life, designing efficient ML Systems and deploying them in the real world +is becoming increasingly challenging, time consuming and costly. +Researchers and engineers must keep pace with rapidly evolving software stacks and a Cambrian explosion of hardware platforms from the cloud to the edge. +Such platforms have their own specific libraries, frameworks, APIs and specifications and often require repetitive, tedious and ad-hoc optimization +of the whole model/software/hardware stack to trade off accuracy, latency, throughout, power consumption, size and costs depending on user requirements and constraints. + +### The CK framework + +*The Collective Knowledge framework (CK)* is our attempt to develop a common plug&play infrastructure that can be used +by the community similar to Wikipedia to learn how to solve above challenges and make it easier to co-design, +benchmark, optimize and deploy Machine Learning Systems in the real world across continuously evolving software, +hardware and data sets (see our [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4) for more details): + +* CK aims at providing a simple playground with minimal software dependencies to help researchers and practitioners share their knowledge + in the form of reusable automation recipes with a unified Python API, CLI and meta description: + - [Stable CK automation recipes](https://github.com/mlcommons/ck/tree/master/ck/repo/module) + - [MLPerf™ benchmark automation recipes](https://github.com/mlcommons/ck/tree/master/ck-mlops/repo/module) + +* CK helps to organize software projects and Git repositories as a database of above automation recipes + and related artifacts based on [FAIR principles](https://www.nature.com/articles/sdata201618) + as described in our [journal article](https://arxiv.org/pdf/2011.01149.pdf) ([shorter pre-print]( https://arxiv.org/abs/2006.07161 )). + See examples of CK-compatible GitHub repositories: + - [MLPerf/MLOps automation](https://github.com/mlcommons/ck-mlops) + - [ACM REQUEST tournament for collaborative and reproducible ML/SW/HW co-design](https://github.com/ctuning/ck-request) + +### Community developments + +We collaborated with the community to reproduce [150+ ML and Systems papers](https://cTuning.org/ae) +and implement the following reusable automation recipes in the CK format: + +* Portable meta package manager to automatically detect, install or rebuild various ML artifacts + (ML models, data sets, frameworks, libraries, etc) across different platform and operating systems including Linux, Windows, MacOS and Android: + - [ML artifact detection plugins](https://github.com/mlcommons/ck-mlops/tree/main/soft) + - [ML meta package installation plugins](https://github.com/mlcommons/ck-mlops/tree/main/package) + - OS descriptions: [Linux/MacOS/Android](https://github.com/mlcommons/ck-mlops/tree/main/os) ; [Windows](https://github.com/ctuning/ck-win/tree/main/os) + +* Portable manager for Python virtual environments: [CK repo](https://github.com/mlcommons/ck-venv). + +* Portable workflows to support collaborative, reproducible and cross-platform benchmarking: + - [ML Systems benchmarking](https://github.com/mlcommons/ck-mlops/tree/main/program) + - [Compiler benchmarking](https://github.com/ctuning/ctuning-programs/tree/master/program) + +* Portable workflows to automate MLPerf™ benchmark: + - [End-to-end submission suite used by multiple organizations to automate the submission of MLPerf inference benchmark](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/README.md) + - MLPerf inference v1.1 results: [MLCommons press-release](https://mlcommons.org/en/news/mlperf-inference-v11), + [Datacenter results](https://mlcommons.org/en/inference-datacenter-11), + [Edge results](https://mlcommons.org/en/inference-edge-11) + - [Reproducibility studies for MLPerf inference benchmark v1.1 automated by CK](https://github.com/mlcommons/ck/tree/master/docs/mlperf-automation/reproduce#reproducibility-reports-mlperf-inference-benchmark-v11) + - [Design space exploration of ML/SW/HW stacks and customizable visualization](https://cknowledge.io/result/crowd-benchmarking-mlperf-inference-classification-mobilenets-all) + + +Please contact [Grigori Fursin](https://www.linkedin.com/in/grigorifursin) if you are interested to join this community effort! + +### Tutorials + +* [CK automations for unified benchmarking]( https://colab.research.google.com/drive/1a761nKgoHlJAy6gOXh-c9H4WkLV8nzRU?usp=sharing ) +* [CK-based MLPerf inference benchmark automation example]( https://colab.research.google.com/drive/1Fp6uxCqTazmCSSl8v-nY93VVmcOoLiXi?usp=sharing ) + * [CK-based MLPerf inference vision benchmark v1.1 automation (TVM)]( https://colab.research.google.com/drive/1aywGlyD1ZRDtQTrQARVgL1882JcvmFK-?usp=sharing ) + * [CK-based MLPerf inference vision benchmark v1.1 automation (ONNX)]( https://colab.research.google.com/drive/1ij1rWoqje5-Sn6UsdFj1OzYakudI2RIS?usp=sharing ) +* [CK basics]( https://colab.research.google.com/drive/15lQgxuTSkEPqi4plaat1_v2gJcfIrATF?usp=sharing ) + +## Releases + +### Development version + +We are developing the 2nd generation of the CK framework (aka CM) based on your feedback: +* [CK2(CM) framework](https://github.com/mlcommons/ck/tree/master/cm) +* [CK2(CM) MLOps and DevOps automation scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops) + +### Stable versions + +The latest version of the CK automation suite supported by MLCommons™: +* [CK framework v2.6.1 (Apache 2.0 license)](https://github.com/mlcommons/ck/releases/tag/V2.6.1) +* [CK automation suite for MLPerf™ and ML/SW/HW co-design](https://github.com/mlcommons/ck-mlops) + + +## Current projects +* [Automating MLPerf(tm) inference benchmark and packing ML models, data sets and frameworks as CK components with a unified API and meta description](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/README.md) +* Developing customizable dashboards for MLPerf™ to help end-users select ML/SW/HW stacks on a Pareto frontier: [aggregated MLPerf™ results]( https://cknowledge.io/?q="mlperf-inference-all" ) +* Providing a common format to share artifacts at ML, systems and other conferences: [video](https://youtu.be/DIkZxraTmGM), [Artifact Evaluation](https://cTuning.org/ae) +* Redesigning CK together with the community based on user feedback: [incubator](https://github.com/mlcommons/ck/tree/master/incubator) +* [Other real-world use cases](https://cKnowledge.org/partners.html) from MLPerf™, Qualcomm, Arm, General Motors, IBM, the Raspberry Pi foundation, ACM and other great partners; + +## Documentation + +* [Online CK documentation]( https://ck.readthedocs.io ) + * [Why CK?]( https://ck.readthedocs.io/en/latest/src/introduction.html ) + * [CK Basics](https://michel-steuwer.github.io/About-CK) + * [Try CK]( https://ck.readthedocs.io/en/latest/src/first-steps.html ) +* [Publications](https://github.com/mlcommons/ck/wiki/Publications) + +## Installation + +Follow [this guide](https://ck.readthedocs.io/en/latest/src/installation.html) +to install CK framework on your platform. + +CK supports the following platforms: + +| | As a host platform | As a target platform | +|---------------|:------------------:|:--------------------:| +| Generic Linux | ✓ | ✓ | +| Linux (Arm) | ✓ | ✓ | +| Raspberry Pi | ✓ | ✓ | +| MacOS | ✓ | ± | +| Windows | ✓ | ✓ | +| Android | ± | ✓ | +| iOS | TBD | TBD | +| Bare-metal (edge devices) | - | ± | + +## Examples + +### Portable CK workflow (native environment without Docker) + +Here we show how to pull a GitHub repo in the CK format +and use a unified CK interface to compile and run +any program (image corner detection in our case) +with any compatible data set on any compatible platform: + +```bash +python3 -m pip install ck + +ck pull repo:mlcommons@ck-mlops + +ck ls program:*susan* + +ck search dataset --tags=jpeg + +ck pull repo:ctuning-datasets-min + +ck search dataset --tags=jpeg + +ck detect soft:compiler.gcc +ck detect soft:compiler.llvm + +ck show env --tags=compiler + +ck compile program:image-corner-detection --speed + +ck run program:image-corner-detection --repeat=1 --env.MY_ENV=123 --env.TEST=xyz +``` + +You can check output of this program in the following directory: +```bash +cd `ck find program:image-corner-detection`/tmp +ls + +processed-image.pgm +``` + +You can now view this image with detected corners. + + +Check [CK docs](https://ck.readthedocs.io/en/latest/src/introduction.html) for further details. + +### MLPerf™ benchmark workflows + +* [Current coverage](https://github.com/mlcommons/ck/tree/master/docs/mlperf-automation#readme) +* [MLPerf inference v1.1 workflows](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/reproduce/README.md#reproducibility-reports-mlperf-inference-benchmark-v11) + +### Portable CK workflows inside containers + +We have prepared adaptive CK containers to demonstrate MLOps capabilities: +* https://github.com/mlcommons/ck-mlops/tree/main/docker + +You can run them as follows: + +```bash +ck pull repo:mlcommons@ck-mlops +ck build docker:ck-template-mlperf --tag=ubuntu-20.04 +ck run docker:ck-template-mlperf --tag=ubuntu-20.04 +``` + +### Portable workflow example with virtual CK environments + +You can create multiple [virtual CK environments](https://github.com/mlcommons/ck-venv) with templates +to automatically install different CK packages and workflows, for example for MLPerf™ inference: + +``` +ck pull repo:mlcommons@ck-venv +ck create venv:test --template=mlperf-inference-main +ck ls venv +ck activate venv:test + +ck pull repo:mlcommons@ck-mlops +ck install package --ask --tags=dataset,coco,val,2017,full +ck show env + +``` + +### Integration with web services and CI platforms + +All CK modules, automation actions and workflows are accessible as a micro-service with a unified JSON I/O API +to make it easier to integrate them with web services and CI platforms as described +[here](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/tools/continuous-integration.md). + + + + +### Other use cases + +* [List of various use cases]( https://ck.readthedocs.io/en/latest/src/introduction.html#ck-showroom ) + + + + + + + +## CK portal + +We have developed the [cKnowledge.io portal](https://cKnowledge.io) to help the community +organize and find all the CK workflows and components similar to PyPI: + +* [Search CK components](https://cKnowledge.io) +* [Browse CK components](https://cKnowledge.io/browse) +* [Find reproduced results from papers]( https://cKnowledge.io/reproduced-results ) +* [Test CK workflows to benchmark and optimize ML Systems]( https://cKnowledge.io/demo ) + + + +## Containers to test CK automation recipes and workflows + +The community provides Docker containers to test CK and components using different ML/SW/HW stacks (DSE). + +* A set of Docker containers to test the basic CK functionality + using some MLPerf inference benchmark workflows: + https://github.com/mlcommons/ck-mlops/tree/main/docker/test-ck + + +## Contributions + +Users can extend the CK functionality via [CK modules](https://github.com/mlcommons/ck/tree/master/ck/repo/module) +or external [GitHub reposities](https://cKnowledge.io/repos) in the CK format +as described [here](https://ck.readthedocs.io/en/latest/src/typical-usage.html). + +Please check [this documentation](https://ck.readthedocs.io/en/latest/src/how-to-contribute.html) +if you want to extend the CK core functionality and [modules](https://github.com/mlcommons/ck/tree/master/ck/repo/module). + +Note, that we plan to [redesign the CK framework](https://github.com/mlcommons/ck/projects/1) +to be more pythonic (we wrote the first prototype without OO to be able +to port it to bare-metal devices in C but eventually we decided to drop this idea). + +Please contact [Grigori Fursin](mailto:grigori@octoml.ai) to join this community effort. + + + +## Contacts + +* [Grigori Fursin](https://fursin.net) - author +* [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh) - maintainer + +## Acknowledgments + +We would like to thank all [contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) +and [collaborators](https://cKnowledge.org/partners.html) for their support, fruitful discussions, +and useful feedback! See more acknowledgments in the [CK journal article](https://arxiv.org/abs/2011.01149) +and [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4). + +%package help +Summary: Development documents and examples for ck +Provides: python3-ck-doc +%description help +*Note that the 1st generation of the CK framework was discontinued in summer 2022 + after the [2nd generation of this framework (CM)](https://github.com/mlcommons/ck) + was released by the [open taskforce on education and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md) + at [MLCommons](https://mlcommons.org).* + + +# Collective Knowledge framework (CK) + +[](https://pepy.tech/project/ck) +[](https://badge.fury.io/py/ck) +[](https://pypi.org/project/ck) + +[](https://travis-ci.com/ctuning/ck) +[](https://ci.appveyor.com/project/gfursin/ck) +[](https://coveralls.io/github/ctuning/ck) + +[](https://ck.readthedocs.io/en/latest/?badge=latest) +[](https://colab.research.google.com/drive/1Fp6uxCqTazmCSSl8v-nY93VVmcOoLiXi?usp=sharing) +[](https://colab.research.google.com/drive/1a761nKgoHlJAy6gOXh-c9H4WkLV8nzRU?usp=sharing) + + +## News + + + +* **2022 July 17:** We have pre-released CK2-based MLOps and DevOps automation scripts at https://github.com/mlcommons/ck/tree/master/cm-mlops/script + +* **2022 May:** We started developing the 2nd generation of the CK framework (aka CM): https://github.com/mlcommons/ck/tree/master/cm + +* **2022 April 3:** We presented the CK concept to bridge the growing gap between ML Systems research and production + at the HPCA'22 workshop on [benchmarking deep learning systems](https://sites.google.com/g.harvard.edu/mlperf-bench-hpca22/home). + +* **2022 March:** We presented the [CK concept to enable collaborative and reproducible ML Systems R&D](https://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=73126) + at the SIAM'22 workshop on "Research Challenges and Opportunities within Software Productivity, Sustainability, and Reproducibility" + +* **2022 March:** we've released the first prototype of [the Collective Mind toolkit (CK2)](https://github.com/mlcommons/ck/tree/master/ck2) + based on your feedback and our practical experience [reproducing 150+ ML and Systems papers and validating them in the real world](https://www.youtube.com/watch?v=7zpeIVwICa4). + +## Motivation + +While Machine Learning is becoming more and more important in everyday life, designing efficient ML Systems and deploying them in the real world +is becoming increasingly challenging, time consuming and costly. +Researchers and engineers must keep pace with rapidly evolving software stacks and a Cambrian explosion of hardware platforms from the cloud to the edge. +Such platforms have their own specific libraries, frameworks, APIs and specifications and often require repetitive, tedious and ad-hoc optimization +of the whole model/software/hardware stack to trade off accuracy, latency, throughout, power consumption, size and costs depending on user requirements and constraints. + +### The CK framework + +*The Collective Knowledge framework (CK)* is our attempt to develop a common plug&play infrastructure that can be used +by the community similar to Wikipedia to learn how to solve above challenges and make it easier to co-design, +benchmark, optimize and deploy Machine Learning Systems in the real world across continuously evolving software, +hardware and data sets (see our [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4) for more details): + +* CK aims at providing a simple playground with minimal software dependencies to help researchers and practitioners share their knowledge + in the form of reusable automation recipes with a unified Python API, CLI and meta description: + - [Stable CK automation recipes](https://github.com/mlcommons/ck/tree/master/ck/repo/module) + - [MLPerf™ benchmark automation recipes](https://github.com/mlcommons/ck/tree/master/ck-mlops/repo/module) + +* CK helps to organize software projects and Git repositories as a database of above automation recipes + and related artifacts based on [FAIR principles](https://www.nature.com/articles/sdata201618) + as described in our [journal article](https://arxiv.org/pdf/2011.01149.pdf) ([shorter pre-print]( https://arxiv.org/abs/2006.07161 )). + See examples of CK-compatible GitHub repositories: + - [MLPerf/MLOps automation](https://github.com/mlcommons/ck-mlops) + - [ACM REQUEST tournament for collaborative and reproducible ML/SW/HW co-design](https://github.com/ctuning/ck-request) + +### Community developments + +We collaborated with the community to reproduce [150+ ML and Systems papers](https://cTuning.org/ae) +and implement the following reusable automation recipes in the CK format: + +* Portable meta package manager to automatically detect, install or rebuild various ML artifacts + (ML models, data sets, frameworks, libraries, etc) across different platform and operating systems including Linux, Windows, MacOS and Android: + - [ML artifact detection plugins](https://github.com/mlcommons/ck-mlops/tree/main/soft) + - [ML meta package installation plugins](https://github.com/mlcommons/ck-mlops/tree/main/package) + - OS descriptions: [Linux/MacOS/Android](https://github.com/mlcommons/ck-mlops/tree/main/os) ; [Windows](https://github.com/ctuning/ck-win/tree/main/os) + +* Portable manager for Python virtual environments: [CK repo](https://github.com/mlcommons/ck-venv). + +* Portable workflows to support collaborative, reproducible and cross-platform benchmarking: + - [ML Systems benchmarking](https://github.com/mlcommons/ck-mlops/tree/main/program) + - [Compiler benchmarking](https://github.com/ctuning/ctuning-programs/tree/master/program) + +* Portable workflows to automate MLPerf™ benchmark: + - [End-to-end submission suite used by multiple organizations to automate the submission of MLPerf inference benchmark](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/README.md) + - MLPerf inference v1.1 results: [MLCommons press-release](https://mlcommons.org/en/news/mlperf-inference-v11), + [Datacenter results](https://mlcommons.org/en/inference-datacenter-11), + [Edge results](https://mlcommons.org/en/inference-edge-11) + - [Reproducibility studies for MLPerf inference benchmark v1.1 automated by CK](https://github.com/mlcommons/ck/tree/master/docs/mlperf-automation/reproduce#reproducibility-reports-mlperf-inference-benchmark-v11) + - [Design space exploration of ML/SW/HW stacks and customizable visualization](https://cknowledge.io/result/crowd-benchmarking-mlperf-inference-classification-mobilenets-all) + + +Please contact [Grigori Fursin](https://www.linkedin.com/in/grigorifursin) if you are interested to join this community effort! + +### Tutorials + +* [CK automations for unified benchmarking]( https://colab.research.google.com/drive/1a761nKgoHlJAy6gOXh-c9H4WkLV8nzRU?usp=sharing ) +* [CK-based MLPerf inference benchmark automation example]( https://colab.research.google.com/drive/1Fp6uxCqTazmCSSl8v-nY93VVmcOoLiXi?usp=sharing ) + * [CK-based MLPerf inference vision benchmark v1.1 automation (TVM)]( https://colab.research.google.com/drive/1aywGlyD1ZRDtQTrQARVgL1882JcvmFK-?usp=sharing ) + * [CK-based MLPerf inference vision benchmark v1.1 automation (ONNX)]( https://colab.research.google.com/drive/1ij1rWoqje5-Sn6UsdFj1OzYakudI2RIS?usp=sharing ) +* [CK basics]( https://colab.research.google.com/drive/15lQgxuTSkEPqi4plaat1_v2gJcfIrATF?usp=sharing ) + +## Releases + +### Development version + +We are developing the 2nd generation of the CK framework (aka CM) based on your feedback: +* [CK2(CM) framework](https://github.com/mlcommons/ck/tree/master/cm) +* [CK2(CM) MLOps and DevOps automation scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops) + +### Stable versions + +The latest version of the CK automation suite supported by MLCommons™: +* [CK framework v2.6.1 (Apache 2.0 license)](https://github.com/mlcommons/ck/releases/tag/V2.6.1) +* [CK automation suite for MLPerf™ and ML/SW/HW co-design](https://github.com/mlcommons/ck-mlops) + + +## Current projects +* [Automating MLPerf(tm) inference benchmark and packing ML models, data sets and frameworks as CK components with a unified API and meta description](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/README.md) +* Developing customizable dashboards for MLPerf™ to help end-users select ML/SW/HW stacks on a Pareto frontier: [aggregated MLPerf™ results]( https://cknowledge.io/?q="mlperf-inference-all" ) +* Providing a common format to share artifacts at ML, systems and other conferences: [video](https://youtu.be/DIkZxraTmGM), [Artifact Evaluation](https://cTuning.org/ae) +* Redesigning CK together with the community based on user feedback: [incubator](https://github.com/mlcommons/ck/tree/master/incubator) +* [Other real-world use cases](https://cKnowledge.org/partners.html) from MLPerf™, Qualcomm, Arm, General Motors, IBM, the Raspberry Pi foundation, ACM and other great partners; + +## Documentation + +* [Online CK documentation]( https://ck.readthedocs.io ) + * [Why CK?]( https://ck.readthedocs.io/en/latest/src/introduction.html ) + * [CK Basics](https://michel-steuwer.github.io/About-CK) + * [Try CK]( https://ck.readthedocs.io/en/latest/src/first-steps.html ) +* [Publications](https://github.com/mlcommons/ck/wiki/Publications) + +## Installation + +Follow [this guide](https://ck.readthedocs.io/en/latest/src/installation.html) +to install CK framework on your platform. + +CK supports the following platforms: + +| | As a host platform | As a target platform | +|---------------|:------------------:|:--------------------:| +| Generic Linux | ✓ | ✓ | +| Linux (Arm) | ✓ | ✓ | +| Raspberry Pi | ✓ | ✓ | +| MacOS | ✓ | ± | +| Windows | ✓ | ✓ | +| Android | ± | ✓ | +| iOS | TBD | TBD | +| Bare-metal (edge devices) | - | ± | + +## Examples + +### Portable CK workflow (native environment without Docker) + +Here we show how to pull a GitHub repo in the CK format +and use a unified CK interface to compile and run +any program (image corner detection in our case) +with any compatible data set on any compatible platform: + +```bash +python3 -m pip install ck + +ck pull repo:mlcommons@ck-mlops + +ck ls program:*susan* + +ck search dataset --tags=jpeg + +ck pull repo:ctuning-datasets-min + +ck search dataset --tags=jpeg + +ck detect soft:compiler.gcc +ck detect soft:compiler.llvm + +ck show env --tags=compiler + +ck compile program:image-corner-detection --speed + +ck run program:image-corner-detection --repeat=1 --env.MY_ENV=123 --env.TEST=xyz +``` + +You can check output of this program in the following directory: +```bash +cd `ck find program:image-corner-detection`/tmp +ls + +processed-image.pgm +``` + +You can now view this image with detected corners. + + +Check [CK docs](https://ck.readthedocs.io/en/latest/src/introduction.html) for further details. + +### MLPerf™ benchmark workflows + +* [Current coverage](https://github.com/mlcommons/ck/tree/master/docs/mlperf-automation#readme) +* [MLPerf inference v1.1 workflows](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/reproduce/README.md#reproducibility-reports-mlperf-inference-benchmark-v11) + +### Portable CK workflows inside containers + +We have prepared adaptive CK containers to demonstrate MLOps capabilities: +* https://github.com/mlcommons/ck-mlops/tree/main/docker + +You can run them as follows: + +```bash +ck pull repo:mlcommons@ck-mlops +ck build docker:ck-template-mlperf --tag=ubuntu-20.04 +ck run docker:ck-template-mlperf --tag=ubuntu-20.04 +``` + +### Portable workflow example with virtual CK environments + +You can create multiple [virtual CK environments](https://github.com/mlcommons/ck-venv) with templates +to automatically install different CK packages and workflows, for example for MLPerf™ inference: + +``` +ck pull repo:mlcommons@ck-venv +ck create venv:test --template=mlperf-inference-main +ck ls venv +ck activate venv:test + +ck pull repo:mlcommons@ck-mlops +ck install package --ask --tags=dataset,coco,val,2017,full +ck show env + +``` + +### Integration with web services and CI platforms + +All CK modules, automation actions and workflows are accessible as a micro-service with a unified JSON I/O API +to make it easier to integrate them with web services and CI platforms as described +[here](https://github.com/mlcommons/ck/blob/master/docs/mlperf-automation/tools/continuous-integration.md). + + + + +### Other use cases + +* [List of various use cases]( https://ck.readthedocs.io/en/latest/src/introduction.html#ck-showroom ) + + + + + + + +## CK portal + +We have developed the [cKnowledge.io portal](https://cKnowledge.io) to help the community +organize and find all the CK workflows and components similar to PyPI: + +* [Search CK components](https://cKnowledge.io) +* [Browse CK components](https://cKnowledge.io/browse) +* [Find reproduced results from papers]( https://cKnowledge.io/reproduced-results ) +* [Test CK workflows to benchmark and optimize ML Systems]( https://cKnowledge.io/demo ) + + + +## Containers to test CK automation recipes and workflows + +The community provides Docker containers to test CK and components using different ML/SW/HW stacks (DSE). + +* A set of Docker containers to test the basic CK functionality + using some MLPerf inference benchmark workflows: + https://github.com/mlcommons/ck-mlops/tree/main/docker/test-ck + + +## Contributions + +Users can extend the CK functionality via [CK modules](https://github.com/mlcommons/ck/tree/master/ck/repo/module) +or external [GitHub reposities](https://cKnowledge.io/repos) in the CK format +as described [here](https://ck.readthedocs.io/en/latest/src/typical-usage.html). + +Please check [this documentation](https://ck.readthedocs.io/en/latest/src/how-to-contribute.html) +if you want to extend the CK core functionality and [modules](https://github.com/mlcommons/ck/tree/master/ck/repo/module). + +Note, that we plan to [redesign the CK framework](https://github.com/mlcommons/ck/projects/1) +to be more pythonic (we wrote the first prototype without OO to be able +to port it to bare-metal devices in C but eventually we decided to drop this idea). + +Please contact [Grigori Fursin](mailto:grigori@octoml.ai) to join this community effort. + + + +## Contacts + +* [Grigori Fursin](https://fursin.net) - author +* [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh) - maintainer + +## Acknowledgments + +We would like to thank all [contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) +and [collaborators](https://cKnowledge.org/partners.html) for their support, fruitful discussions, +and useful feedback! See more acknowledgments in the [CK journal article](https://arxiv.org/abs/2011.01149) +and [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4). + +%prep +%autosetup -n ck-2.6.3 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-ck -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Wed May 10 2023 Python_Bot <Python_Bot@openeuler.org> - 2.6.3-1 +- Package Spec generated |