diff options
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | python-rusenttokenize.spec | 309 | ||||
| -rw-r--r-- | sources | 1 |
3 files changed, 311 insertions, 0 deletions
@@ -0,0 +1 @@ +/rusenttokenize-0.0.5.tar.gz diff --git a/python-rusenttokenize.spec b/python-rusenttokenize.spec new file mode 100644 index 0000000..37114ea --- /dev/null +++ b/python-rusenttokenize.spec @@ -0,0 +1,309 @@ +%global _empty_manifest_terminate_build 0 +Name: python-rusenttokenize +Version: 0.0.5 +Release: 1 +Summary: Rule-based sentence tokenizer for Russian language +License: Apache Software License +URL: https://github.com/deepmipt/ru_sentence_tokenizer +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/6d/76/1226e1ddc11ad492a191664a4926c607bcbf1e5b352134ca6f83c4af8205/rusenttokenize-0.0.5.tar.gz +BuildArch: noarch + + +%description +# ru_sent_tokenize +A simple and fast rule-based sentence segmentation. Tested on OpenCorpora and SynTagRus datasets. + +# Installation +``` +pip install rusenttokenize +``` + +# Running +```ipython +>>> from rusenttokenize import ru_sent_tokenize +>>> ru_sent_tokenize('Эта шоколадка за 400р. ничего из себя не представляла. Артём решил больше не ходить в этот магазин') +['Эта шоколадка за 400р. ничего из себя не представляла.', 'Артём решил больше не ходить в этот магазин'] +``` + +# Metrics + +The tokenizer has been tested on OpenCorpora and SynTagRus. There are two important metrics. + +Precision. First one is we took single sentences from the datasets and measured how many times tokenizer didn't split them. + +Recall. Second metric is we took two consecutive sentences from the datasets and joined each pair with a space characted. We measured how many times tokenizer correctly splitted a long sentence into two. + +<table> + <tr> + <th rowspan=2>tokenizer</th> + <th colspan=3>OpenCorpora</th> + <th colspan=3>SynTagRus</th> + </tr> + <tr> + <th>Precision</th> + <th>Recall</th> + <th>Execution Time (sec)</th> + <th>Precision</th> + <th>Recall</th> + <th>Execution Time (sec)</th> + </tr> + <tbody> + <tr> + <td>nltk.sent_tokenize</td> + <td>94.30</td> + <td>86.06</td> + <td>8.67</td> + <td>98.15</td> + <td>94.95</td> + <td>5.07</td> + </tr> + <tr> + <td>nltk.sent_tokenize(x, language='russian')</td> + <td>95.53</td> + <td>88.37</td> + <td>8.54</td> + <td>98.44</td> + <td>95.45</td> + <td>5.68</td> + </tr> + <tr> + <td>bureaucratic-labs.segmentator.split</td> + <td>97.16</td> + <td>88.62</td> + <td>359</td> + <td>96.79</td> + <td>92.55</td> + <td>210</td> + </tr> + <tr> + <td>ru_sent_tokenize</td> + <td>98.73</td> + <td>93.45</td> + <td>4.92</td> + <td>99.81</td> + <td>98.59</td> + <td>2.87</td> + </tr> + </tbody> +</table> + +[Notebook](https://github.com/deepmipt/ru_sentence_tokenizer/blob/master/metrics/calculate.ipynb) shows how the table above was calculated + + + +%package -n python3-rusenttokenize +Summary: Rule-based sentence tokenizer for Russian language +Provides: python-rusenttokenize +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-rusenttokenize +# ru_sent_tokenize +A simple and fast rule-based sentence segmentation. Tested on OpenCorpora and SynTagRus datasets. + +# Installation +``` +pip install rusenttokenize +``` + +# Running +```ipython +>>> from rusenttokenize import ru_sent_tokenize +>>> ru_sent_tokenize('Эта шоколадка за 400р. ничего из себя не представляла. Артём решил больше не ходить в этот магазин') +['Эта шоколадка за 400р. ничего из себя не представляла.', 'Артём решил больше не ходить в этот магазин'] +``` + +# Metrics + +The tokenizer has been tested on OpenCorpora and SynTagRus. There are two important metrics. + +Precision. First one is we took single sentences from the datasets and measured how many times tokenizer didn't split them. + +Recall. Second metric is we took two consecutive sentences from the datasets and joined each pair with a space characted. We measured how many times tokenizer correctly splitted a long sentence into two. + +<table> + <tr> + <th rowspan=2>tokenizer</th> + <th colspan=3>OpenCorpora</th> + <th colspan=3>SynTagRus</th> + </tr> + <tr> + <th>Precision</th> + <th>Recall</th> + <th>Execution Time (sec)</th> + <th>Precision</th> + <th>Recall</th> + <th>Execution Time (sec)</th> + </tr> + <tbody> + <tr> + <td>nltk.sent_tokenize</td> + <td>94.30</td> + <td>86.06</td> + <td>8.67</td> + <td>98.15</td> + <td>94.95</td> + <td>5.07</td> + </tr> + <tr> + <td>nltk.sent_tokenize(x, language='russian')</td> + <td>95.53</td> + <td>88.37</td> + <td>8.54</td> + <td>98.44</td> + <td>95.45</td> + <td>5.68</td> + </tr> + <tr> + <td>bureaucratic-labs.segmentator.split</td> + <td>97.16</td> + <td>88.62</td> + <td>359</td> + <td>96.79</td> + <td>92.55</td> + <td>210</td> + </tr> + <tr> + <td>ru_sent_tokenize</td> + <td>98.73</td> + <td>93.45</td> + <td>4.92</td> + <td>99.81</td> + <td>98.59</td> + <td>2.87</td> + </tr> + </tbody> +</table> + +[Notebook](https://github.com/deepmipt/ru_sentence_tokenizer/blob/master/metrics/calculate.ipynb) shows how the table above was calculated + + + +%package help +Summary: Development documents and examples for rusenttokenize +Provides: python3-rusenttokenize-doc +%description help +# ru_sent_tokenize +A simple and fast rule-based sentence segmentation. Tested on OpenCorpora and SynTagRus datasets. + +# Installation +``` +pip install rusenttokenize +``` + +# Running +```ipython +>>> from rusenttokenize import ru_sent_tokenize +>>> ru_sent_tokenize('Эта шоколадка за 400р. ничего из себя не представляла. Артём решил больше не ходить в этот магазин') +['Эта шоколадка за 400р. ничего из себя не представляла.', 'Артём решил больше не ходить в этот магазин'] +``` + +# Metrics + +The tokenizer has been tested on OpenCorpora and SynTagRus. There are two important metrics. + +Precision. First one is we took single sentences from the datasets and measured how many times tokenizer didn't split them. + +Recall. Second metric is we took two consecutive sentences from the datasets and joined each pair with a space characted. We measured how many times tokenizer correctly splitted a long sentence into two. + +<table> + <tr> + <th rowspan=2>tokenizer</th> + <th colspan=3>OpenCorpora</th> + <th colspan=3>SynTagRus</th> + </tr> + <tr> + <th>Precision</th> + <th>Recall</th> + <th>Execution Time (sec)</th> + <th>Precision</th> + <th>Recall</th> + <th>Execution Time (sec)</th> + </tr> + <tbody> + <tr> + <td>nltk.sent_tokenize</td> + <td>94.30</td> + <td>86.06</td> + <td>8.67</td> + <td>98.15</td> + <td>94.95</td> + <td>5.07</td> + </tr> + <tr> + <td>nltk.sent_tokenize(x, language='russian')</td> + <td>95.53</td> + <td>88.37</td> + <td>8.54</td> + <td>98.44</td> + <td>95.45</td> + <td>5.68</td> + </tr> + <tr> + <td>bureaucratic-labs.segmentator.split</td> + <td>97.16</td> + <td>88.62</td> + <td>359</td> + <td>96.79</td> + <td>92.55</td> + <td>210</td> + </tr> + <tr> + <td>ru_sent_tokenize</td> + <td>98.73</td> + <td>93.45</td> + <td>4.92</td> + <td>99.81</td> + <td>98.59</td> + <td>2.87</td> + </tr> + </tbody> +</table> + +[Notebook](https://github.com/deepmipt/ru_sentence_tokenizer/blob/master/metrics/calculate.ipynb) shows how the table above was calculated + + + +%prep +%autosetup -n rusenttokenize-0.0.5 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-rusenttokenize -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Fri May 05 2023 Python_Bot <Python_Bot@openeuler.org> - 0.0.5-1 +- Package Spec generated @@ -0,0 +1 @@ +9058f7d375e4c18278c3733e8dd10100 rusenttokenize-0.0.5.tar.gz |