path: root/python-clean-text.spec

%global _empty_manifest_terminate_build 0
Name:		python-clean-text
Version:	0.6.0
Release:	1
Summary:	Functions to preprocess and normalize text.
License:	Apache-2.0
URL:		https://pypi.org/project/clean-text/
Source0:	https://mirrors.nju.edu.cn/pypi/web/packages/c3/5c/3151736165b123611351c103908f24841d88df0dfe455ece15b2657adeae/clean-text-0.6.0.tar.gz
BuildArch:	noarch

Requires:	python3-emoji
Requires:	python3-ftfy
Requires:	python3-pandas
Requires:	python3-scikit-learn
Requires:	python3-unidecode

%description
# `clean-text` [![Build Status](https://img.shields.io/github/workflow/status/jfilter/clean-text/Test)](https://github.com/jfilter/clean-text/actions/workflows/test.yml) [![PyPI](https://img.shields.io/pypi/v/clean-text.svg)](https://pypi.org/project/clean-text/) [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/clean-text.svg)](https://pypi.org/project/clean-text/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/clean-text)](https://pypistats.org/packages/clean-text)

User-generated content on the Web and in social media is often dirty. Preprocess your scraped data with `clean-text` to create a normalized text representation. For instance, turn this corrupted input:

```txt
A bunch of \\u2018new\\u2019 references, including [Moana](https://en.wikipedia.org/wiki/Moana_%282016_film%29).


»Yóù àré     rïght <3!«
```

into this clean output:

```txt
A bunch of 'new' references, including [moana](<URL>).

"you are right <3!"
```

`clean-text` uses [ftfy](https://github.com/LuminosoInsight/python-ftfy), [unidecode](https://github.com/takluyver/Unidecode) and numerous hand-crafted rules, i.e., RegEx.

## Installation

To install the GPL-licensed package [unidecode](https://github.com/takluyver/Unidecode) alongside:

```bash
pip install clean-text[gpl]
```

You may want to abstain from GPL:

```bash
pip install clean-text
```

NB: This package is named `clean-text` and not `cleantext`.

If [unidecode](https://github.com/takluyver/Unidecode) is not available, `clean-text` will resort to Python's [unicodedata.normalize](https://docs.python.org/3.7/library/unicodedata.html#unicodedata.normalize) for [transliteration](https://en.wikipedia.org/wiki/Transliteration).
Transliteration to closest ASCII symbols involes manually mappings, i.e., `ê` to `e`.
`unidecode`'s mapping is superiour but unicodedata's are sufficent.
However, you may want to disable this feature altogether depending on your data and use case.

To make it clear: There are **inconsistencies** between processing text with or without `unidecode`.

## Usage

```python
from cleantext import clean

clean("some input",
    fix_unicode=True,               # fix various unicode errors
    to_ascii=True,                  # transliterate to closest ASCII representation
    lower=True,                     # lowercase text
    no_line_breaks=False,           # fully strip line breaks as opposed to only normalizing them
    no_urls=False,                  # replace all URLs with a special token
    no_emails=False,                # replace all email addresses with a special token
    no_phone_numbers=False,         # replace all phone numbers with a special token
    no_numbers=False,               # replace all numbers with a special token
    no_digits=False,                # replace all digits with a special token
    no_currency_symbols=False,      # replace all currency symbols with a special token
    no_punct=False,                 # remove punctuations
    replace_with_punct="",          # instead of removing punctuations you may replace them
    replace_with_url="<URL>",
    replace_with_email="<EMAIL>",
    replace_with_phone_number="<PHONE>",
    replace_with_number="<NUMBER>",
    replace_with_digit="0",
    replace_with_currency_symbol="<CUR>",
    lang="en"                       # set to 'de' for German special handling
)
```

Carefully choose the arguments that fit your task. The default parameters are listed above.

You may also only use specific functions for cleaning. For this, take a look at the [source code](https://github.com/jfilter/clean-text/blob/main/cleantext/clean.py).

### Supported languages

So far, only English and German are fully supported.
It should work for the majority of western languages.
If you need some special handling for your language, feel free to contribute. 🙃

### Using `clean-text` with `scikit-learn`

There is also **scikit-learn** compatible API to use in your pipelines.
All of the parameters above work here as well.

```bash
pip install clean-text[gpl,sklearn]
pip install clean-text[sklearn]
```

```python
from cleantext.sklearn import CleanTransformer

cleaner = CleanTransformer(no_punct=False, lower=False)

cleaner.transform(['Happily clean your text!', 'Another Input'])
```

## Development

[Use poetry.](https://python-poetry.org/)

## Contributing

If you have a **question**, found a **bug** or want to propose a new **feature**, have a look at the [issues page](https://github.com/jfilter/clean-text/issues).

**Pull requests** are especially welcomed when they fix bugs or improve the code quality.

If you don't like the output of `clean-text`, consider adding a [test](https://github.com/jfilter/clean-text/tree/main/tests) with your specific input and desired output.

## Related Work

### Generic text cleaning packages

-   https://github.com/pudo/normality
-   https://github.com/davidmogar/cucco
-   https://github.com/lyeoni/prenlp
-   https://github.com/s/preprocessor
-   https://github.com/artefactory/NLPretext
-   https://github.com/cbaziotis/ekphrasis

### Full-blown NLP libraries with some text cleaning

-   https://github.com/chartbeat-labs/textacy
-   https://github.com/jbesomi/texthero

### Remove or replace strings

-   https://github.com/vi3k6i5/flashtext
-   https://github.com/ddelange/retrie

### Detect dates

-   https://github.com/scrapinghub/dateparser

### Clean massive Common Crawl data

-   https://github.com/facebookresearch/cc_net

## Acknowledgements

Built upon the work by [Burton DeWilde](https://github.com/bdewilde) for [Textacy](https://github.com/chartbeat-labs/textacy).

## License

Apache


%package -n python3-clean-text
Summary:	Functions to preprocess and normalize text.
Provides:	python-clean-text
BuildRequires:	python3-devel
BuildRequires:	python3-setuptools
BuildRequires:	python3-pip
%description -n python3-clean-text
# `clean-text` [![Build Status](https://img.shields.io/github/workflow/status/jfilter/clean-text/Test)](https://github.com/jfilter/clean-text/actions/workflows/test.yml) [![PyPI](https://img.shields.io/pypi/v/clean-text.svg)](https://pypi.org/project/clean-text/) [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/clean-text.svg)](https://pypi.org/project/clean-text/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/clean-text)](https://pypistats.org/packages/clean-text)

User-generated content on the Web and in social media is often dirty. Preprocess your scraped data with `clean-text` to create a normalized text representation. For instance, turn this corrupted input:

```txt
A bunch of \\u2018new\\u2019 references, including [Moana](https://en.wikipedia.org/wiki/Moana_%282016_film%29).


»Yóù àré     rïght &lt;3!«
```

into this clean output:

```txt
A bunch of 'new' references, including [moana](<URL>).

"you are right <3!"
```

`clean-text` uses [ftfy](https://github.com/LuminosoInsight/python-ftfy), [unidecode](https://github.com/takluyver/Unidecode) and numerous hand-crafted rules, i.e., RegEx.

## Installation

To install the GPL-licensed package [unidecode](https://github.com/takluyver/Unidecode) alongside:

```bash
pip install clean-text[gpl]
```

You may want to abstain from GPL:

```bash
pip install clean-text
```

NB: This package is named `clean-text` and not `cleantext`.

If [unidecode](https://github.com/takluyver/Unidecode) is not available, `clean-text` will resort to Python's [unicodedata.normalize](https://docs.python.org/3.7/library/unicodedata.html#unicodedata.normalize) for [transliteration](https://en.wikipedia.org/wiki/Transliteration).
Transliteration to closest ASCII symbols involes manually mappings, i.e., `ê` to `e`.
`unidecode`'s mapping is superiour but unicodedata's are sufficent.
However, you may want to disable this feature altogether depending on your data and use case.

To make it clear: There are **inconsistencies** between processing text with or without `unidecode`.

## Usage

```python
from cleantext import clean

clean("some input",
    fix_unicode=True,               # fix various unicode errors
    to_ascii=True,                  # transliterate to closest ASCII representation
    lower=True,                     # lowercase text
    no_line_breaks=False,           # fully strip line breaks as opposed to only normalizing them
    no_urls=False,                  # replace all URLs with a special token
    no_emails=False,                # replace all email addresses with a special token
    no_phone_numbers=False,         # replace all phone numbers with a special token
    no_numbers=False,               # replace all numbers with a special token
    no_digits=False,                # replace all digits with a special token
    no_currency_symbols=False,      # replace all currency symbols with a special token
    no_punct=False,                 # remove punctuations
    replace_with_punct="",          # instead of removing punctuations you may replace them
    replace_with_url="<URL>",
    replace_with_email="<EMAIL>",
    replace_with_phone_number="<PHONE>",
    replace_with_number="<NUMBER>",
    replace_with_digit="0",
    replace_with_currency_symbol="<CUR>",
    lang="en"                       # set to 'de' for German special handling
)
```

Carefully choose the arguments that fit your task. The default parameters are listed above.

You may also only use specific functions for cleaning. For this, take a look at the [source code](https://github.com/jfilter/clean-text/blob/main/cleantext/clean.py).

### Supported languages

So far, only English and German are fully supported.
It should work for the majority of western languages.
If you need some special handling for your language, feel free to contribute. 🙃

### Using `clean-text` with `scikit-learn`

There is also **scikit-learn** compatible API to use in your pipelines.
All of the parameters above work here as well.

```bash
pip install clean-text[gpl,sklearn]
pip install clean-text[sklearn]
```

```python
from cleantext.sklearn import CleanTransformer

cleaner = CleanTransformer(no_punct=False, lower=False)

cleaner.transform(['Happily clean your text!', 'Another Input'])
```

## Development

[Use poetry.](https://python-poetry.org/)

## Contributing

If you have a **question**, found a **bug** or want to propose a new **feature**, have a look at the [issues page](https://github.com/jfilter/clean-text/issues).

**Pull requests** are especially welcomed when they fix bugs or improve the code quality.

If you don't like the output of `clean-text`, consider adding a [test](https://github.com/jfilter/clean-text/tree/main/tests) with your specific input and desired output.

## Related Work

### Generic text cleaning packages

-   https://github.com/pudo/normality
-   https://github.com/davidmogar/cucco
-   https://github.com/lyeoni/prenlp
-   https://github.com/s/preprocessor
-   https://github.com/artefactory/NLPretext
-   https://github.com/cbaziotis/ekphrasis

### Full-blown NLP libraries with some text cleaning

-   https://github.com/chartbeat-labs/textacy
-   https://github.com/jbesomi/texthero

### Remove or replace strings

-   https://github.com/vi3k6i5/flashtext
-   https://github.com/ddelange/retrie

### Detect dates

-   https://github.com/scrapinghub/dateparser

### Clean massive Common Crawl data

-   https://github.com/facebookresearch/cc_net

## Acknowledgements

Built upon the work by [Burton DeWilde](https://github.com/bdewilde) for [Textacy](https://github.com/chartbeat-labs/textacy).

## License

Apache


%package help
Summary:	Development documents and examples for clean-text
Provides:	python3-clean-text-doc
%description help
# `clean-text` [![Build Status](https://img.shields.io/github/workflow/status/jfilter/clean-text/Test)](https://github.com/jfilter/clean-text/actions/workflows/test.yml) [![PyPI](https://img.shields.io/pypi/v/clean-text.svg)](https://pypi.org/project/clean-text/) [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/clean-text.svg)](https://pypi.org/project/clean-text/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/clean-text)](https://pypistats.org/packages/clean-text)

User-generated content on the Web and in social media is often dirty. Preprocess your scraped data with `clean-text` to create a normalized text representation. For instance, turn this corrupted input:

```txt
A bunch of \\u2018new\\u2019 references, including [Moana](https://en.wikipedia.org/wiki/Moana_%282016_film%29).


»Yóù àré     rïght &lt;3!«
```

into this clean output:

```txt
A bunch of 'new' references, including [moana](<URL>).

"you are right <3!"
```

`clean-text` uses [ftfy](https://github.com/LuminosoInsight/python-ftfy), [unidecode](https://github.com/takluyver/Unidecode) and numerous hand-crafted rules, i.e., RegEx.

## Installation

To install the GPL-licensed package [unidecode](https://github.com/takluyver/Unidecode) alongside:

```bash
pip install clean-text[gpl]
```

You may want to abstain from GPL:

```bash
pip install clean-text
```

NB: This package is named `clean-text` and not `cleantext`.

If [unidecode](https://github.com/takluyver/Unidecode) is not available, `clean-text` will resort to Python's [unicodedata.normalize](https://docs.python.org/3.7/library/unicodedata.html#unicodedata.normalize) for [transliteration](https://en.wikipedia.org/wiki/Transliteration).
Transliteration to closest ASCII symbols involes manually mappings, i.e., `ê` to `e`.
`unidecode`'s mapping is superiour but unicodedata's are sufficent.
However, you may want to disable this feature altogether depending on your data and use case.

To make it clear: There are **inconsistencies** between processing text with or without `unidecode`.

## Usage

```python
from cleantext import clean

clean("some input",
    fix_unicode=True,               # fix various unicode errors
    to_ascii=True,                  # transliterate to closest ASCII representation
    lower=True,                     # lowercase text
    no_line_breaks=False,           # fully strip line breaks as opposed to only normalizing them
    no_urls=False,                  # replace all URLs with a special token
    no_emails=False,                # replace all email addresses with a special token
    no_phone_numbers=False,         # replace all phone numbers with a special token
    no_numbers=False,               # replace all numbers with a special token
    no_digits=False,                # replace all digits with a special token
    no_currency_symbols=False,      # replace all currency symbols with a special token
    no_punct=False,                 # remove punctuations
    replace_with_punct="",          # instead of removing punctuations you may replace them
    replace_with_url="<URL>",
    replace_with_email="<EMAIL>",
    replace_with_phone_number="<PHONE>",
    replace_with_number="<NUMBER>",
    replace_with_digit="0",
    replace_with_currency_symbol="<CUR>",
    lang="en"                       # set to 'de' for German special handling
)
```

Carefully choose the arguments that fit your task. The default parameters are listed above.

You may also only use specific functions for cleaning. For this, take a look at the [source code](https://github.com/jfilter/clean-text/blob/main/cleantext/clean.py).

### Supported languages

So far, only English and German are fully supported.
It should work for the majority of western languages.
If you need some special handling for your language, feel free to contribute. 🙃

### Using `clean-text` with `scikit-learn`

There is also **scikit-learn** compatible API to use in your pipelines.
All of the parameters above work here as well.

```bash
pip install clean-text[gpl,sklearn]
pip install clean-text[sklearn]
```

```python
from cleantext.sklearn import CleanTransformer

cleaner = CleanTransformer(no_punct=False, lower=False)

cleaner.transform(['Happily clean your text!', 'Another Input'])
```

## Development

[Use poetry.](https://python-poetry.org/)

## Contributing

If you have a **question**, found a **bug** or want to propose a new **feature**, have a look at the [issues page](https://github.com/jfilter/clean-text/issues).

**Pull requests** are especially welcomed when they fix bugs or improve the code quality.

If you don't like the output of `clean-text`, consider adding a [test](https://github.com/jfilter/clean-text/tree/main/tests) with your specific input and desired output.

## Related Work

### Generic text cleaning packages

-   https://github.com/pudo/normality
-   https://github.com/davidmogar/cucco
-   https://github.com/lyeoni/prenlp
-   https://github.com/s/preprocessor
-   https://github.com/artefactory/NLPretext
-   https://github.com/cbaziotis/ekphrasis

### Full-blown NLP libraries with some text cleaning

-   https://github.com/chartbeat-labs/textacy
-   https://github.com/jbesomi/texthero

### Remove or replace strings

-   https://github.com/vi3k6i5/flashtext
-   https://github.com/ddelange/retrie

### Detect dates

-   https://github.com/scrapinghub/dateparser

### Clean massive Common Crawl data

-   https://github.com/facebookresearch/cc_net

## Acknowledgements

Built upon the work by [Burton DeWilde](https://github.com/bdewilde) for [Textacy](https://github.com/chartbeat-labs/textacy).

## License

Apache


%prep
%autosetup -n clean-text-0.6.0

%build
%py3_build

%install
%py3_install
install -d -m755 %{buildroot}/%{_pkgdocdir}
if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
pushd %{buildroot}
if [ -d usr/lib ]; then
	find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/lib64 ]; then
	find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/bin ]; then
	find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/sbin ]; then
	find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
fi
touch doclist.lst
if [ -d usr/share/man ]; then
	find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
fi
popd
mv %{buildroot}/filelist.lst .
mv %{buildroot}/doclist.lst .

%files -n python3-clean-text -f filelist.lst
%dir %{python3_sitelib}/*

%files help -f doclist.lst
%{_docdir}/*

%changelog
* Sun Apr 23 2023 Python_Bot <Python_Bot@openeuler.org> - 0.6.0-1
- Package Spec generated