%global _empty_manifest_terminate_build 0 Name: python-minet Version: 0.67.1 Release: 1 Summary: A webmining CLI tool & library for python. License: MIT URL: http://github.com/medialab/minet Source0: https://mirrors.nju.edu.cn/pypi/web/packages/a2/80/7b6c09dad64f580c64378f90286e06df1027d849f3377f86c8aef235741f/minet-0.67.1.tar.gz BuildArch: noarch Requires: python3-beautifulsoup4 Requires: python3-browser-cookie3 Requires: python3-casanova Requires: python3-charset-normalizer Requires: python3-colorama Requires: python3-dateparser Requires: python3-ebbe Requires: python3-json5 Requires: python3-keyring Requires: python3-lxml Requires: python3-ndjson Requires: python3-persist-queue Requires: python3-pyyaml Requires: python3-quenouille Requires: python3-soupsieve Requires: python3-tenacity Requires: python3-termcolor Requires: python3-tqdm Requires: python3-trafilatura Requires: python3-twitwi Requires: python3-ural Requires: python3-urllib3 %description [![Build Status](https://github.com/medialab/minet/workflows/Tests/badge.svg)](https://github.com/medialab/minet/actions) [![DOI](https://zenodo.org/badge/169059797.svg)](https://zenodo.org/badge/latestdoi/169059797) [![download number](https://static.pepy.tech/badge/minet)](https://pepy.tech/project/minet) ![Minet](img/minet.png) **minet** is a webmining command line tool & library for python (>= 3.7) that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, CrowdTangle, YouTube, Twitter, Media Cloud etc. It adopts a very simple approach to various webmining problems by letting you perform a variety of actions from the comfort of the command line. No database needed: raw CSV files should be sufficient to do most of the work. In addition, **minet** also exposes its high-level programmatic interface as a python library so you can tweak its behavior at will. **Shortcuts**: [Command line documentation](./docs/cli.md), [Python library documentation](./docs/lib.md). ## Summary * [What it does](#what-it-does) * [Documented use cases](#documented-use-cases) * [Features (from a technical standpoint)](#features-from-a-technical-standpoint) * [Installation](#installation) * [Upgrading](#upgrading) * [Uninstallation](#uninstallation) * [Documentation](#documentation) * [Contributing](#contributing) * [How to cite](#how-to-cite) ## What it does Minet can single-handedly: * Extract URLs from a text file (or a table) * Parse URLs (get useful information, with Facebook- and Youtube-specific stuff) * Join two CSV files by matching the columns containing URLs * From a list of URLs, resolve their redirections * ...and check their HTTP status * ...and download the HTML * ...and extract hyperlinks * ...and extract the text content and other metadata (title...) * ...and scrape structured data (using a declarative language to define your heuristics) * Crawl (using a declarative language to define a browsing behavior, and what to harvest) * Mine or search: * *[Buzzsumo](https://buzzsumo.com/)* (requires API acess) * *[Crowdtangle](https://www.crowdtangle.com/)* (requires API access) * *[Mediacloud](https://mediacloud.org/)* (requires free API access) * *[Twitter](https://twitter.com)* (requires free API access) * *[Youtube](https://www.youtube.com/)* (requires free API access) * Scrape (without requiring special access, often just a user account): * *[Facebook](https://www.facebook.com/)* * *[Instagram](https://www.instagram.com/)* * *[Telegram](https://telegram.org/)* * *[TikTok](https://www.tiktok.com)* * *[Twitter](https://twitter.com)* * *[Google Drive](https://drive.google.com)* (spreadsheets etc.) * Grab & dump cookies from your browser * Dump *[Hyphe](https://hyphe.medialab.sciences-po.fr/)* data ## Documented use cases * [Fetching a large amount of urls](./cookbook/fetch.md) * [Joining 2 CSV files by urls](./cookbook/url_join.md) * [Using minet from a Jupyter notebook](./cookbook/notebooks/Minet%20in%20a%20Jupyter%20notebook.ipynb) (*very useful to experiment with the tool or teach students*) * [Downloading images associated with a given hashtag on Twitter](./cookbook/twitter_images.md) * [Scraping DSL Tutorial](./cookbook/scraping_dsl.md) ## Features (from a technical standpoint) * Multithreaded, memory-efficient fetching from the web. * Multithreaded, scalable crawling using a comfy DSL. * Multiprocessed raw text content extraction from HTML pages. * Multiprocessed scraping from HTML pages using a comfy DSL. * URL-related heuristics utilities such as extraction, normalization and matching. * Data collection from various APIs such as [CrowdTangle](https://www.crowdtangle.com/). ## Installation **minet** can be installed as a standalone CLI tool (currently only on mac >= 10.14, ubuntu & similar) by running the following command in your terminal: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash ``` Don't trust us enough to pipe the result of a HTTP request into `bash`? We wouldn't either, so feel free to read the installation script [here](./scripts/install.sh) and run it on your end if you prefer. On ubuntu & similar you might need to install `curl` and `unzip` before running the installation script if you don't already have it: ```shell sudo apt-get install curl unzip ``` Else, **minet** can be installed directly as a python CLI tool and library using pip: ```shell pip install minet ``` If you need more help to install and use **minet** from scratch, you can check those [installation documents](./docs/install.md). Finally if you want to install the standalone binaries by yourself (even for windows) you can find them in each release [here](https://github.com/medialab/minet/releases). ## Upgrading To upgrade the standalone version, simply run the install script once again: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash ``` To upgrade the python version you can use pip thusly: ```shell pip install -U minet ``` ## Uninstallation To uninstall the standalone version: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/uninstall.sh | bash ``` To uninstall the python version: ```shell pip uninstall minet ``` ## Documentation * [minet as a command line tool](./docs/cli.md) * [minet as a python library](./docs/lib.md) ## Contributing To contribute to **minet** you can check out [this](./CONTRIBUTING.md) documentation. ## How to cite **minet** is published on [Zenodo](https://zenodo.org/) as [![DOI](https://zenodo.org/badge/169059797.svg)](https://zenodo.org/badge/latestdoi/169059797) You can cite it thusly: > Guillaume Plique, Pauline Breteau, Jules Farjas, Héloïse Théro, Jean Descamps, Amélie Pellé, & Laura Miguel. (2019, October 14). Minet, a webmining CLI tool & library for python. Zenodo. http://doi.org/10.5281/zenodo.4564399 %package -n python3-minet Summary: A webmining CLI tool & library for python. Provides: python-minet BuildRequires: python3-devel BuildRequires: python3-setuptools BuildRequires: python3-pip %description -n python3-minet [![Build Status](https://github.com/medialab/minet/workflows/Tests/badge.svg)](https://github.com/medialab/minet/actions) [![DOI](https://zenodo.org/badge/169059797.svg)](https://zenodo.org/badge/latestdoi/169059797) [![download number](https://static.pepy.tech/badge/minet)](https://pepy.tech/project/minet) ![Minet](img/minet.png) **minet** is a webmining command line tool & library for python (>= 3.7) that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, CrowdTangle, YouTube, Twitter, Media Cloud etc. It adopts a very simple approach to various webmining problems by letting you perform a variety of actions from the comfort of the command line. No database needed: raw CSV files should be sufficient to do most of the work. In addition, **minet** also exposes its high-level programmatic interface as a python library so you can tweak its behavior at will. **Shortcuts**: [Command line documentation](./docs/cli.md), [Python library documentation](./docs/lib.md). ## Summary * [What it does](#what-it-does) * [Documented use cases](#documented-use-cases) * [Features (from a technical standpoint)](#features-from-a-technical-standpoint) * [Installation](#installation) * [Upgrading](#upgrading) * [Uninstallation](#uninstallation) * [Documentation](#documentation) * [Contributing](#contributing) * [How to cite](#how-to-cite) ## What it does Minet can single-handedly: * Extract URLs from a text file (or a table) * Parse URLs (get useful information, with Facebook- and Youtube-specific stuff) * Join two CSV files by matching the columns containing URLs * From a list of URLs, resolve their redirections * ...and check their HTTP status * ...and download the HTML * ...and extract hyperlinks * ...and extract the text content and other metadata (title...) * ...and scrape structured data (using a declarative language to define your heuristics) * Crawl (using a declarative language to define a browsing behavior, and what to harvest) * Mine or search: * *[Buzzsumo](https://buzzsumo.com/)* (requires API acess) * *[Crowdtangle](https://www.crowdtangle.com/)* (requires API access) * *[Mediacloud](https://mediacloud.org/)* (requires free API access) * *[Twitter](https://twitter.com)* (requires free API access) * *[Youtube](https://www.youtube.com/)* (requires free API access) * Scrape (without requiring special access, often just a user account): * *[Facebook](https://www.facebook.com/)* * *[Instagram](https://www.instagram.com/)* * *[Telegram](https://telegram.org/)* * *[TikTok](https://www.tiktok.com)* * *[Twitter](https://twitter.com)* * *[Google Drive](https://drive.google.com)* (spreadsheets etc.) * Grab & dump cookies from your browser * Dump *[Hyphe](https://hyphe.medialab.sciences-po.fr/)* data ## Documented use cases * [Fetching a large amount of urls](./cookbook/fetch.md) * [Joining 2 CSV files by urls](./cookbook/url_join.md) * [Using minet from a Jupyter notebook](./cookbook/notebooks/Minet%20in%20a%20Jupyter%20notebook.ipynb) (*very useful to experiment with the tool or teach students*) * [Downloading images associated with a given hashtag on Twitter](./cookbook/twitter_images.md) * [Scraping DSL Tutorial](./cookbook/scraping_dsl.md) ## Features (from a technical standpoint) * Multithreaded, memory-efficient fetching from the web. * Multithreaded, scalable crawling using a comfy DSL. * Multiprocessed raw text content extraction from HTML pages. * Multiprocessed scraping from HTML pages using a comfy DSL. * URL-related heuristics utilities such as extraction, normalization and matching. * Data collection from various APIs such as [CrowdTangle](https://www.crowdtangle.com/). ## Installation **minet** can be installed as a standalone CLI tool (currently only on mac >= 10.14, ubuntu & similar) by running the following command in your terminal: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash ``` Don't trust us enough to pipe the result of a HTTP request into `bash`? We wouldn't either, so feel free to read the installation script [here](./scripts/install.sh) and run it on your end if you prefer. On ubuntu & similar you might need to install `curl` and `unzip` before running the installation script if you don't already have it: ```shell sudo apt-get install curl unzip ``` Else, **minet** can be installed directly as a python CLI tool and library using pip: ```shell pip install minet ``` If you need more help to install and use **minet** from scratch, you can check those [installation documents](./docs/install.md). Finally if you want to install the standalone binaries by yourself (even for windows) you can find them in each release [here](https://github.com/medialab/minet/releases). ## Upgrading To upgrade the standalone version, simply run the install script once again: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash ``` To upgrade the python version you can use pip thusly: ```shell pip install -U minet ``` ## Uninstallation To uninstall the standalone version: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/uninstall.sh | bash ``` To uninstall the python version: ```shell pip uninstall minet ``` ## Documentation * [minet as a command line tool](./docs/cli.md) * [minet as a python library](./docs/lib.md) ## Contributing To contribute to **minet** you can check out [this](./CONTRIBUTING.md) documentation. ## How to cite **minet** is published on [Zenodo](https://zenodo.org/) as [![DOI](https://zenodo.org/badge/169059797.svg)](https://zenodo.org/badge/latestdoi/169059797) You can cite it thusly: > Guillaume Plique, Pauline Breteau, Jules Farjas, Héloïse Théro, Jean Descamps, Amélie Pellé, & Laura Miguel. (2019, October 14). Minet, a webmining CLI tool & library for python. Zenodo. http://doi.org/10.5281/zenodo.4564399 %package help Summary: Development documents and examples for minet Provides: python3-minet-doc %description help [![Build Status](https://github.com/medialab/minet/workflows/Tests/badge.svg)](https://github.com/medialab/minet/actions) [![DOI](https://zenodo.org/badge/169059797.svg)](https://zenodo.org/badge/latestdoi/169059797) [![download number](https://static.pepy.tech/badge/minet)](https://pepy.tech/project/minet) ![Minet](img/minet.png) **minet** is a webmining command line tool & library for python (>= 3.7) that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, CrowdTangle, YouTube, Twitter, Media Cloud etc. It adopts a very simple approach to various webmining problems by letting you perform a variety of actions from the comfort of the command line. No database needed: raw CSV files should be sufficient to do most of the work. In addition, **minet** also exposes its high-level programmatic interface as a python library so you can tweak its behavior at will. **Shortcuts**: [Command line documentation](./docs/cli.md), [Python library documentation](./docs/lib.md). ## Summary * [What it does](#what-it-does) * [Documented use cases](#documented-use-cases) * [Features (from a technical standpoint)](#features-from-a-technical-standpoint) * [Installation](#installation) * [Upgrading](#upgrading) * [Uninstallation](#uninstallation) * [Documentation](#documentation) * [Contributing](#contributing) * [How to cite](#how-to-cite) ## What it does Minet can single-handedly: * Extract URLs from a text file (or a table) * Parse URLs (get useful information, with Facebook- and Youtube-specific stuff) * Join two CSV files by matching the columns containing URLs * From a list of URLs, resolve their redirections * ...and check their HTTP status * ...and download the HTML * ...and extract hyperlinks * ...and extract the text content and other metadata (title...) * ...and scrape structured data (using a declarative language to define your heuristics) * Crawl (using a declarative language to define a browsing behavior, and what to harvest) * Mine or search: * *[Buzzsumo](https://buzzsumo.com/)* (requires API acess) * *[Crowdtangle](https://www.crowdtangle.com/)* (requires API access) * *[Mediacloud](https://mediacloud.org/)* (requires free API access) * *[Twitter](https://twitter.com)* (requires free API access) * *[Youtube](https://www.youtube.com/)* (requires free API access) * Scrape (without requiring special access, often just a user account): * *[Facebook](https://www.facebook.com/)* * *[Instagram](https://www.instagram.com/)* * *[Telegram](https://telegram.org/)* * *[TikTok](https://www.tiktok.com)* * *[Twitter](https://twitter.com)* * *[Google Drive](https://drive.google.com)* (spreadsheets etc.) * Grab & dump cookies from your browser * Dump *[Hyphe](https://hyphe.medialab.sciences-po.fr/)* data ## Documented use cases * [Fetching a large amount of urls](./cookbook/fetch.md) * [Joining 2 CSV files by urls](./cookbook/url_join.md) * [Using minet from a Jupyter notebook](./cookbook/notebooks/Minet%20in%20a%20Jupyter%20notebook.ipynb) (*very useful to experiment with the tool or teach students*) * [Downloading images associated with a given hashtag on Twitter](./cookbook/twitter_images.md) * [Scraping DSL Tutorial](./cookbook/scraping_dsl.md) ## Features (from a technical standpoint) * Multithreaded, memory-efficient fetching from the web. * Multithreaded, scalable crawling using a comfy DSL. * Multiprocessed raw text content extraction from HTML pages. * Multiprocessed scraping from HTML pages using a comfy DSL. * URL-related heuristics utilities such as extraction, normalization and matching. * Data collection from various APIs such as [CrowdTangle](https://www.crowdtangle.com/). ## Installation **minet** can be installed as a standalone CLI tool (currently only on mac >= 10.14, ubuntu & similar) by running the following command in your terminal: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash ``` Don't trust us enough to pipe the result of a HTTP request into `bash`? We wouldn't either, so feel free to read the installation script [here](./scripts/install.sh) and run it on your end if you prefer. On ubuntu & similar you might need to install `curl` and `unzip` before running the installation script if you don't already have it: ```shell sudo apt-get install curl unzip ``` Else, **minet** can be installed directly as a python CLI tool and library using pip: ```shell pip install minet ``` If you need more help to install and use **minet** from scratch, you can check those [installation documents](./docs/install.md). Finally if you want to install the standalone binaries by yourself (even for windows) you can find them in each release [here](https://github.com/medialab/minet/releases). ## Upgrading To upgrade the standalone version, simply run the install script once again: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash ``` To upgrade the python version you can use pip thusly: ```shell pip install -U minet ``` ## Uninstallation To uninstall the standalone version: ```shell curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/uninstall.sh | bash ``` To uninstall the python version: ```shell pip uninstall minet ``` ## Documentation * [minet as a command line tool](./docs/cli.md) * [minet as a python library](./docs/lib.md) ## Contributing To contribute to **minet** you can check out [this](./CONTRIBUTING.md) documentation. ## How to cite **minet** is published on [Zenodo](https://zenodo.org/) as [![DOI](https://zenodo.org/badge/169059797.svg)](https://zenodo.org/badge/latestdoi/169059797) You can cite it thusly: > Guillaume Plique, Pauline Breteau, Jules Farjas, Héloïse Théro, Jean Descamps, Amélie Pellé, & Laura Miguel. (2019, October 14). Minet, a webmining CLI tool & library for python. Zenodo. http://doi.org/10.5281/zenodo.4564399 %prep %autosetup -n minet-0.67.1 %build %py3_build %install %py3_install install -d -m755 %{buildroot}/%{_pkgdocdir} if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi pushd %{buildroot} if [ -d usr/lib ]; then find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst fi if [ -d usr/lib64 ]; then find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst fi if [ -d usr/bin ]; then find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst fi if [ -d usr/sbin ]; then find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst fi touch doclist.lst if [ -d usr/share/man ]; then find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst fi popd mv %{buildroot}/filelist.lst . mv %{buildroot}/doclist.lst . %files -n python3-minet -f filelist.lst %dir %{python3_sitelib}/* %files help -f doclist.lst %{_docdir}/* %changelog * Wed Apr 12 2023 Python_Bot - 0.67.1-1 - Package Spec generated