diff options
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | python-linux-aio.spec | 508 | ||||
| -rw-r--r-- | sources | 1 |
3 files changed, 510 insertions, 0 deletions
@@ -0,0 +1 @@ +/linux_aio-0.4.1.tar.gz diff --git a/python-linux-aio.spec b/python-linux-aio.spec new file mode 100644 index 0000000..02481b7 --- /dev/null +++ b/python-linux-aio.spec @@ -0,0 +1,508 @@ +%global _empty_manifest_terminate_build 0 +Name: python-linux-aio +Version: 0.4.1 +Release: 1 +Summary: Linux aio ABI wrapper +License: LGPLv3+ +URL: https://github.com/isac322/linux_aio +Source0: https://mirrors.aliyun.com/pypi/web/packages/0a/14/04041de341534654eb6d7475ab8b6c25e057f43e612e21f7134ac744ad42/linux_aio-0.4.1.tar.gz +BuildArch: noarch + +Requires: python3-linux-aio-bind + +%description +**\[english\]** | [\[한국어 (korean)\]](https://github.com/isac322/linux_aio/blob/master/README.kor.md) + +# linux_aio: Python wrapper for [Linux Kernel AIO](http://lse.sourceforge.net/io/aio.html) + +[](https://travis-ci.com/isac322/linux_aio) +[](https://pypi.org/project/linux-aio/) +[](https://codecov.io/gh/isac322/linux_aio) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://saythanks.io/to/isac322) + +Python wrapper module that uses Linux Kernel AIO directly + + +## What is Linux Kernel AIO? + +[Linux IO Models table](https://oxnz.github.io/2016/10/13/linux-aio/#io-models) + +In summary, it allows non-blocking and asynchronous use of blocking IO operations such as [read(2)](http://man7.org/linux/man-pages/man2/read.2.html) and [write(2)](http://man7.org/linux/man-pages/man2/write.2.html). + + +### Related documents + +- [Linux Asynchronous I/O](https://oxnz.github.io/2016/10/13/linux-aio/) +- [Linux Kernel AIO Design Notes](http://lse.sourceforge.net/io/aionotes.txt) +- [How to use the Linux AIO feature](https://github.com/littledan/linux-aio) (in C) + + +### **It is different from [POSIX AIO](http://man7.org/linux/man-pages/man7/aio.7.html)** + +The POSIX AIO APIs have the `aio_` prefix, but the Linux Kernel AIO has the `io_` prefix. + +There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally. +So, as you can see from the experiment below, it's much worse than using the blocking IO API. + + +## Implementation & Structure + +### Package `linux_aio` + +- Implemented based on [linux_aio_bind](https://pypi.org/project/linux-aio-bind) package which is low-level binding of Linux kernel AIO. +- Unlike [linux_aio_bind](https://pypi.org/project/linux-aio-bind), it can be used without knowledge of `ctypes` +- Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test). + + +## Example + +Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test). + + +## Notes & Limits + +- Obviously available only on Linux +- Because it is a wrapper, it brings the constraints of Linux. + - It can not be used for files used as a kernel interface. (e.g. `cgroup`) + - [Sometimes it works as Blocking.](https://stackoverflow.com/questions/34572559/asynchronous-io-io-submit-latency-in-ubuntu-linux) + - There are some things that have been solved through development after posting. + - Some features are being added because they are still under development. + - There are also some features that are not supported when the Linux version is low + - You need to check [Linux man pages (4.16)](http://man7.org/linux/man-pages/man2/io_submit.2.html) and its related API documentation + + +## Evaluation + +[Experiment script](https://gist.github.com/isac322/8606f5c464fa390cb88b47354981cdab) (requires python 3.7) + +### Setup + +- Distribution: Ubuntu Server 16.04.5 LTS +- Linux: 4.19.0 +- CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz +- MEM: total 64GB +- Storage: SK hynix SC300B SATA 512GB +- Python: 3.7.2 ([Ubuntu ppa](https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa)) + + +- Attempts to read a total of `1000` times in` 1ms` intervals. +- The file size varies from `1KB` to` 100KB`, but it is small. +- Experiment with increasing the number of files read at the same time +- Because we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop. + + +### Comparison target + +- [aiofiles](https://pypi.org/project/aiofiles/) - Uses Thread pool +- [aiofile](https://pypi.org/project/aiofile/) - Uses POSIX AIO +- [libaio](https://pypi.org/project/libaio/) - Uses [libaio](http://lse.sourceforge.net/io/aio.html) +- [python built-in open()](https://docs.python.org/3/library/functions.html#open) + + +**It is not a perfectly fair comparison.** + +`aiofiles` and `aiofile` are libraries that support [asyncio](https://docs.python.org/3/library/asyncio.html). Since `open()` is blocking, there is a disadvantage that you can not do any other work while IO is going on. `libaio` and `linux_aio` are non-blocking, but must be polled. + + +### Results + +**It may differ from environment to environment.** + +#### Runtime + +- Unit: second + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:-----: |:-----: |:-----: |:-----: | +| aiofiles | 1.681 | 3.318 | 5.354 | 9.768 | +| aiofile | 1.543 | 1.958 | 2.493 | 3.737 | +| libaio | 1.311 | 1.344 | 1.362 | 1.423 | +| open() | 1.252 | 1.322 | 1.375 | 1.481 | +| linux_aio | 1.305 | 1.327 | 1.353 | 1.431 | + +#### Threads + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:---: |:---: |:---: |:---: | +| aiofiles | 321 | 321 | 321 | 321 | +| aiofile | 3 | 8 | 15 | 26 | +| libaio | 1 | 1 | 1 | 1 | +| open() | 1 | 1 | 1 | 1 | +| linux_aio | 1 | 1 | 1 | 1 | + +#### Memory + +- Physical memory (Virtual memory) + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:-------------- |:-------------- |:-------------- |:-------------- | +| aiofiles | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | +| aiofile | 17MB (258MB) | 17MB (654MB) | 17MB (1080MB) | 18MB (1949MB) | +| libaio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | +| open() | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | +| linux_aio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | + +#### CPU Utilization + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |------: |------: |-------: |-------: | +| aiofiles | 42.8% | 85.0% | 102.2% | 113.2% | +| aiofile | 31.4% | 52.4% | 67.0% | 84.0% | +| libaio | 14.0% | 16.0% | 17.2% | 20.6% | +| open() | 13.4% | 17.6% | 21.0% | 26.2% | +| linux_aio | 13.0% | 15.0% | 16.0% | 21.0% | + + + +%package -n python3-linux-aio +Summary: Linux aio ABI wrapper +Provides: python-linux-aio +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-linux-aio +**\[english\]** | [\[한국어 (korean)\]](https://github.com/isac322/linux_aio/blob/master/README.kor.md) + +# linux_aio: Python wrapper for [Linux Kernel AIO](http://lse.sourceforge.net/io/aio.html) + +[](https://travis-ci.com/isac322/linux_aio) +[](https://pypi.org/project/linux-aio/) +[](https://codecov.io/gh/isac322/linux_aio) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://saythanks.io/to/isac322) + +Python wrapper module that uses Linux Kernel AIO directly + + +## What is Linux Kernel AIO? + +[Linux IO Models table](https://oxnz.github.io/2016/10/13/linux-aio/#io-models) + +In summary, it allows non-blocking and asynchronous use of blocking IO operations such as [read(2)](http://man7.org/linux/man-pages/man2/read.2.html) and [write(2)](http://man7.org/linux/man-pages/man2/write.2.html). + + +### Related documents + +- [Linux Asynchronous I/O](https://oxnz.github.io/2016/10/13/linux-aio/) +- [Linux Kernel AIO Design Notes](http://lse.sourceforge.net/io/aionotes.txt) +- [How to use the Linux AIO feature](https://github.com/littledan/linux-aio) (in C) + + +### **It is different from [POSIX AIO](http://man7.org/linux/man-pages/man7/aio.7.html)** + +The POSIX AIO APIs have the `aio_` prefix, but the Linux Kernel AIO has the `io_` prefix. + +There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally. +So, as you can see from the experiment below, it's much worse than using the blocking IO API. + + +## Implementation & Structure + +### Package `linux_aio` + +- Implemented based on [linux_aio_bind](https://pypi.org/project/linux-aio-bind) package which is low-level binding of Linux kernel AIO. +- Unlike [linux_aio_bind](https://pypi.org/project/linux-aio-bind), it can be used without knowledge of `ctypes` +- Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test). + + +## Example + +Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test). + + +## Notes & Limits + +- Obviously available only on Linux +- Because it is a wrapper, it brings the constraints of Linux. + - It can not be used for files used as a kernel interface. (e.g. `cgroup`) + - [Sometimes it works as Blocking.](https://stackoverflow.com/questions/34572559/asynchronous-io-io-submit-latency-in-ubuntu-linux) + - There are some things that have been solved through development after posting. + - Some features are being added because they are still under development. + - There are also some features that are not supported when the Linux version is low + - You need to check [Linux man pages (4.16)](http://man7.org/linux/man-pages/man2/io_submit.2.html) and its related API documentation + + +## Evaluation + +[Experiment script](https://gist.github.com/isac322/8606f5c464fa390cb88b47354981cdab) (requires python 3.7) + +### Setup + +- Distribution: Ubuntu Server 16.04.5 LTS +- Linux: 4.19.0 +- CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz +- MEM: total 64GB +- Storage: SK hynix SC300B SATA 512GB +- Python: 3.7.2 ([Ubuntu ppa](https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa)) + + +- Attempts to read a total of `1000` times in` 1ms` intervals. +- The file size varies from `1KB` to` 100KB`, but it is small. +- Experiment with increasing the number of files read at the same time +- Because we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop. + + +### Comparison target + +- [aiofiles](https://pypi.org/project/aiofiles/) - Uses Thread pool +- [aiofile](https://pypi.org/project/aiofile/) - Uses POSIX AIO +- [libaio](https://pypi.org/project/libaio/) - Uses [libaio](http://lse.sourceforge.net/io/aio.html) +- [python built-in open()](https://docs.python.org/3/library/functions.html#open) + + +**It is not a perfectly fair comparison.** + +`aiofiles` and `aiofile` are libraries that support [asyncio](https://docs.python.org/3/library/asyncio.html). Since `open()` is blocking, there is a disadvantage that you can not do any other work while IO is going on. `libaio` and `linux_aio` are non-blocking, but must be polled. + + +### Results + +**It may differ from environment to environment.** + +#### Runtime + +- Unit: second + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:-----: |:-----: |:-----: |:-----: | +| aiofiles | 1.681 | 3.318 | 5.354 | 9.768 | +| aiofile | 1.543 | 1.958 | 2.493 | 3.737 | +| libaio | 1.311 | 1.344 | 1.362 | 1.423 | +| open() | 1.252 | 1.322 | 1.375 | 1.481 | +| linux_aio | 1.305 | 1.327 | 1.353 | 1.431 | + +#### Threads + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:---: |:---: |:---: |:---: | +| aiofiles | 321 | 321 | 321 | 321 | +| aiofile | 3 | 8 | 15 | 26 | +| libaio | 1 | 1 | 1 | 1 | +| open() | 1 | 1 | 1 | 1 | +| linux_aio | 1 | 1 | 1 | 1 | + +#### Memory + +- Physical memory (Virtual memory) + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:-------------- |:-------------- |:-------------- |:-------------- | +| aiofiles | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | +| aiofile | 17MB (258MB) | 17MB (654MB) | 17MB (1080MB) | 18MB (1949MB) | +| libaio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | +| open() | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | +| linux_aio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | + +#### CPU Utilization + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |------: |------: |-------: |-------: | +| aiofiles | 42.8% | 85.0% | 102.2% | 113.2% | +| aiofile | 31.4% | 52.4% | 67.0% | 84.0% | +| libaio | 14.0% | 16.0% | 17.2% | 20.6% | +| open() | 13.4% | 17.6% | 21.0% | 26.2% | +| linux_aio | 13.0% | 15.0% | 16.0% | 21.0% | + + + +%package help +Summary: Development documents and examples for linux-aio +Provides: python3-linux-aio-doc +%description help +**\[english\]** | [\[한국어 (korean)\]](https://github.com/isac322/linux_aio/blob/master/README.kor.md) + +# linux_aio: Python wrapper for [Linux Kernel AIO](http://lse.sourceforge.net/io/aio.html) + +[](https://travis-ci.com/isac322/linux_aio) +[](https://pypi.org/project/linux-aio/) +[](https://codecov.io/gh/isac322/linux_aio) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://pypi.org/project/linux-aio/) +[](https://saythanks.io/to/isac322) + +Python wrapper module that uses Linux Kernel AIO directly + + +## What is Linux Kernel AIO? + +[Linux IO Models table](https://oxnz.github.io/2016/10/13/linux-aio/#io-models) + +In summary, it allows non-blocking and asynchronous use of blocking IO operations such as [read(2)](http://man7.org/linux/man-pages/man2/read.2.html) and [write(2)](http://man7.org/linux/man-pages/man2/write.2.html). + + +### Related documents + +- [Linux Asynchronous I/O](https://oxnz.github.io/2016/10/13/linux-aio/) +- [Linux Kernel AIO Design Notes](http://lse.sourceforge.net/io/aionotes.txt) +- [How to use the Linux AIO feature](https://github.com/littledan/linux-aio) (in C) + + +### **It is different from [POSIX AIO](http://man7.org/linux/man-pages/man7/aio.7.html)** + +The POSIX AIO APIs have the `aio_` prefix, but the Linux Kernel AIO has the `io_` prefix. + +There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally. +So, as you can see from the experiment below, it's much worse than using the blocking IO API. + + +## Implementation & Structure + +### Package `linux_aio` + +- Implemented based on [linux_aio_bind](https://pypi.org/project/linux-aio-bind) package which is low-level binding of Linux kernel AIO. +- Unlike [linux_aio_bind](https://pypi.org/project/linux-aio-bind), it can be used without knowledge of `ctypes` +- Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test). + + +## Example + +Examples can be found in the code in the [test directory](https://github.com/isac322/linux_aio/tree/master/test). + + +## Notes & Limits + +- Obviously available only on Linux +- Because it is a wrapper, it brings the constraints of Linux. + - It can not be used for files used as a kernel interface. (e.g. `cgroup`) + - [Sometimes it works as Blocking.](https://stackoverflow.com/questions/34572559/asynchronous-io-io-submit-latency-in-ubuntu-linux) + - There are some things that have been solved through development after posting. + - Some features are being added because they are still under development. + - There are also some features that are not supported when the Linux version is low + - You need to check [Linux man pages (4.16)](http://man7.org/linux/man-pages/man2/io_submit.2.html) and its related API documentation + + +## Evaluation + +[Experiment script](https://gist.github.com/isac322/8606f5c464fa390cb88b47354981cdab) (requires python 3.7) + +### Setup + +- Distribution: Ubuntu Server 16.04.5 LTS +- Linux: 4.19.0 +- CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz +- MEM: total 64GB +- Storage: SK hynix SC300B SATA 512GB +- Python: 3.7.2 ([Ubuntu ppa](https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa)) + + +- Attempts to read a total of `1000` times in` 1ms` intervals. +- The file size varies from `1KB` to` 100KB`, but it is small. +- Experiment with increasing the number of files read at the same time +- Because we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop. + + +### Comparison target + +- [aiofiles](https://pypi.org/project/aiofiles/) - Uses Thread pool +- [aiofile](https://pypi.org/project/aiofile/) - Uses POSIX AIO +- [libaio](https://pypi.org/project/libaio/) - Uses [libaio](http://lse.sourceforge.net/io/aio.html) +- [python built-in open()](https://docs.python.org/3/library/functions.html#open) + + +**It is not a perfectly fair comparison.** + +`aiofiles` and `aiofile` are libraries that support [asyncio](https://docs.python.org/3/library/asyncio.html). Since `open()` is blocking, there is a disadvantage that you can not do any other work while IO is going on. `libaio` and `linux_aio` are non-blocking, but must be polled. + + +### Results + +**It may differ from environment to environment.** + +#### Runtime + +- Unit: second + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:-----: |:-----: |:-----: |:-----: | +| aiofiles | 1.681 | 3.318 | 5.354 | 9.768 | +| aiofile | 1.543 | 1.958 | 2.493 | 3.737 | +| libaio | 1.311 | 1.344 | 1.362 | 1.423 | +| open() | 1.252 | 1.322 | 1.375 | 1.481 | +| linux_aio | 1.305 | 1.327 | 1.353 | 1.431 | + +#### Threads + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:---: |:---: |:---: |:---: | +| aiofiles | 321 | 321 | 321 | 321 | +| aiofile | 3 | 8 | 15 | 26 | +| libaio | 1 | 1 | 1 | 1 | +| open() | 1 | 1 | 1 | 1 | +| linux_aio | 1 | 1 | 1 | 1 | + +#### Memory + +- Physical memory (Virtual memory) + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |:-------------- |:-------------- |:-------------- |:-------------- | +| aiofiles | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | +| aiofile | 17MB (258MB) | 17MB (654MB) | 17MB (1080MB) | 18MB (1949MB) | +| libaio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | +| open() | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | +| linux_aio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | + +#### CPU Utilization + +| # of files | 1 | 6 | 12 | 24 | +|:---------: |------: |------: |-------: |-------: | +| aiofiles | 42.8% | 85.0% | 102.2% | 113.2% | +| aiofile | 31.4% | 52.4% | 67.0% | 84.0% | +| libaio | 14.0% | 16.0% | 17.2% | 20.6% | +| open() | 13.4% | 17.6% | 21.0% | 26.2% | +| linux_aio | 13.0% | 15.0% | 16.0% | 21.0% | + + + +%prep +%autosetup -n linux_aio-0.4.1 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "\"/%h/%f.gz\"\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-linux-aio -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Tue Jun 20 2023 Python_Bot <Python_Bot@openeuler.org> - 0.4.1-1 +- Package Spec generated @@ -0,0 +1 @@ +2cb3a174b0f105f0e4923a6c33a31378 linux_aio-0.4.1.tar.gz |