%global _empty_manifest_terminate_build 0 Name: python-cdp-scrapers Version: 0.7.0 Release: 1 Summary: Scratchpad for scraper development and general utilities. License: MIT License URL: https://github.com/CouncilDataProject/cdp-scrapers Source0: https://mirrors.aliyun.com/pypi/web/packages/2f/08/99d19efba78a0b2978efa637f2f1280a8f7f09b68a53e5c267dd1c1dcea6/cdp-scrapers-0.7.0.tar.gz BuildArch: noarch Requires: python3-beautifulsoup4 Requires: python3-cdp-backend Requires: python3-defusedxml Requires: python3-pytz Requires: python3-requests Requires: python3-clean-text Requires: python3-civic-scraper Requires: python3-yt-dlp Requires: python3-selenium Requires: python3-webdriver-manager Requires: python3-selenium Requires: python3-webdriver-manager Requires: python3-ipython Requires: python3-m2r2 Requires: python3-Sphinx Requires: python3-furo Requires: python3-numpydoc Requires: python3-sphinx-copybutton Requires: python3-docutils Requires: python3-black Requires: python3-check-manifest Requires: python3-ruff Requires: python3-mypy Requires: python3-pre-commit Requires: python3-coverage Requires: python3-pytest Requires: python3-pytest-cov Requires: python3-pytest-raises Requires: python3-pytest-rerunfailures %description ## Installation **Stable Release:** `pip install cdp-scrapers`
**Development Head:** `pip install git+https://github.com/CouncilDataProject/cdp-scrapers.git` ## Council Data Project Council Data Project is an open-source project dedicated to providing journalists, activists, researchers, and all members of each community we serve with the tools they need to stay informed and hold their Council Members accountable. For more information about Council Data Project, please visit [our website](https://councildataproject.org/). ## About `cdp-scrapers` is a collection of utilities and in-progress or actively maintained CDP instance event scrapers. The purpose of this library is to help new CDP instance maintainers have a quick plethora of examples for getting started on developing their event scraper functions. ## Quick Start ### Legistar General Legistar utility functions. ```python from cdp_scrapers.legistar_utils import get_legistar_events_for_timespan from cdp_scrapers.instances import get_seattle_events from datetime import datetime # Get all events (and minutes item and voting details) # for a provided timespan for a legistar client # Returns List[Dict] seattle_legistar_events = get_legistar_events_for_timespan( client="seattle", timezone="America/Los_Angeles", start=datetime(2021, 7, 12), end=datetime(2021, 7, 14), ) # Or parse and convert to CDP EventIngestionModel seattle_cdp_parsed_events = get_seattle_events( from_dt=datetime(2021, 7, 12), to_dt=datetime(2021, 7, 14), ) ``` ### Scrapers #### Event Scraper Structure Our current event scraper structure is as follows. The main function `get_events` gets all the required data and it calls the `get_content_uris` function to return the required video data. If your city uses Legistar and the Legistar data is publicly available. - You may be able to reuse our scraper with minimal modifications, such as providing the correct Legistar client ID for your municipality. - If the Legistar data returned only does not include the EventVideoPath field for the `Session.video_uri` data, you will only need to implement `get_content_uris`. If your city does not use Legistar. - You will need to build your own event scraper. Example of a completed scraper: [cdp_scrapers.instances.seattle.SeattleScraper](https://councildataproject.org/cdp-scrapers/_modules/cdp_scrapers/instances/seattle.html#SeattleScraper) For more details about creating a custom scraper for your municipality's Legistar data, please visit [here](https://councildataproject.org/cdp-scrapers/legistar_scraper.html). If you would like to deploy a CDP instance or would like to use this library as a method for retrieving formatted legislative data, please feel free to contribute a new custom municipality scraper! #### Creating a Custom Scraper If it isn't possible to use our generalized Legistar tooling to write your scraper, you will need to create your own event scraper to proceed with the deployment. 1. Please see our documentation on the [minimum data required for CDP event ingestion](https://councildataproject.org/cdp-backend/ingestion_models.html) to understand what data your scraper should return. 2. From there, begin with our [empty custom scraper function template](https://councildataproject.org/cdp-scrapers/_modules/cdp_scrapers/instances/empty.html#get_events) and fill in your scraper. 3. After your scraper is completed, you can create a pull request to add your scraper into the [cdp-scrapers repo](https://github.com/CouncilDataProject/cdp-scrapers) so it can be added into the final repo for your CDP instance. 4. Our automated action will run your scraper to verify it returns the correct data. If it is successful, you may proceed to the next deployment step. If not, we will automatically share the error message so you can fix the issue and the scraper can be tested again afterwards. ## Installation **Stable Release:** `pip install cdp-scrapers`
**Development Head:** `pip install git+https://github.com/CouncilDataProject/cdp-scrapers.git` ## Documentation For full package documentation please visit [councildataproject.org/cdp-scrapers](https://councildataproject.org/cdp-scrapers). ## Development Refer to [CONTRIBUTING.md](CONTRIBUTING.md) for information related to developing the code. **MPLv2 License** %package -n python3-cdp-scrapers Summary: Scratchpad for scraper development and general utilities. Provides: python-cdp-scrapers BuildRequires: python3-devel BuildRequires: python3-setuptools BuildRequires: python3-pip %description -n python3-cdp-scrapers ## Installation **Stable Release:** `pip install cdp-scrapers`
**Development Head:** `pip install git+https://github.com/CouncilDataProject/cdp-scrapers.git` ## Council Data Project Council Data Project is an open-source project dedicated to providing journalists, activists, researchers, and all members of each community we serve with the tools they need to stay informed and hold their Council Members accountable. For more information about Council Data Project, please visit [our website](https://councildataproject.org/). ## About `cdp-scrapers` is a collection of utilities and in-progress or actively maintained CDP instance event scrapers. The purpose of this library is to help new CDP instance maintainers have a quick plethora of examples for getting started on developing their event scraper functions. ## Quick Start ### Legistar General Legistar utility functions. ```python from cdp_scrapers.legistar_utils import get_legistar_events_for_timespan from cdp_scrapers.instances import get_seattle_events from datetime import datetime # Get all events (and minutes item and voting details) # for a provided timespan for a legistar client # Returns List[Dict] seattle_legistar_events = get_legistar_events_for_timespan( client="seattle", timezone="America/Los_Angeles", start=datetime(2021, 7, 12), end=datetime(2021, 7, 14), ) # Or parse and convert to CDP EventIngestionModel seattle_cdp_parsed_events = get_seattle_events( from_dt=datetime(2021, 7, 12), to_dt=datetime(2021, 7, 14), ) ``` ### Scrapers #### Event Scraper Structure Our current event scraper structure is as follows. The main function `get_events` gets all the required data and it calls the `get_content_uris` function to return the required video data. If your city uses Legistar and the Legistar data is publicly available. - You may be able to reuse our scraper with minimal modifications, such as providing the correct Legistar client ID for your municipality. - If the Legistar data returned only does not include the EventVideoPath field for the `Session.video_uri` data, you will only need to implement `get_content_uris`. If your city does not use Legistar. - You will need to build your own event scraper. Example of a completed scraper: [cdp_scrapers.instances.seattle.SeattleScraper](https://councildataproject.org/cdp-scrapers/_modules/cdp_scrapers/instances/seattle.html#SeattleScraper) For more details about creating a custom scraper for your municipality's Legistar data, please visit [here](https://councildataproject.org/cdp-scrapers/legistar_scraper.html). If you would like to deploy a CDP instance or would like to use this library as a method for retrieving formatted legislative data, please feel free to contribute a new custom municipality scraper! #### Creating a Custom Scraper If it isn't possible to use our generalized Legistar tooling to write your scraper, you will need to create your own event scraper to proceed with the deployment. 1. Please see our documentation on the [minimum data required for CDP event ingestion](https://councildataproject.org/cdp-backend/ingestion_models.html) to understand what data your scraper should return. 2. From there, begin with our [empty custom scraper function template](https://councildataproject.org/cdp-scrapers/_modules/cdp_scrapers/instances/empty.html#get_events) and fill in your scraper. 3. After your scraper is completed, you can create a pull request to add your scraper into the [cdp-scrapers repo](https://github.com/CouncilDataProject/cdp-scrapers) so it can be added into the final repo for your CDP instance. 4. Our automated action will run your scraper to verify it returns the correct data. If it is successful, you may proceed to the next deployment step. If not, we will automatically share the error message so you can fix the issue and the scraper can be tested again afterwards. ## Installation **Stable Release:** `pip install cdp-scrapers`
**Development Head:** `pip install git+https://github.com/CouncilDataProject/cdp-scrapers.git` ## Documentation For full package documentation please visit [councildataproject.org/cdp-scrapers](https://councildataproject.org/cdp-scrapers). ## Development Refer to [CONTRIBUTING.md](CONTRIBUTING.md) for information related to developing the code. **MPLv2 License** %package help Summary: Development documents and examples for cdp-scrapers Provides: python3-cdp-scrapers-doc %description help ## Installation **Stable Release:** `pip install cdp-scrapers`
**Development Head:** `pip install git+https://github.com/CouncilDataProject/cdp-scrapers.git` ## Council Data Project Council Data Project is an open-source project dedicated to providing journalists, activists, researchers, and all members of each community we serve with the tools they need to stay informed and hold their Council Members accountable. For more information about Council Data Project, please visit [our website](https://councildataproject.org/). ## About `cdp-scrapers` is a collection of utilities and in-progress or actively maintained CDP instance event scrapers. The purpose of this library is to help new CDP instance maintainers have a quick plethora of examples for getting started on developing their event scraper functions. ## Quick Start ### Legistar General Legistar utility functions. ```python from cdp_scrapers.legistar_utils import get_legistar_events_for_timespan from cdp_scrapers.instances import get_seattle_events from datetime import datetime # Get all events (and minutes item and voting details) # for a provided timespan for a legistar client # Returns List[Dict] seattle_legistar_events = get_legistar_events_for_timespan( client="seattle", timezone="America/Los_Angeles", start=datetime(2021, 7, 12), end=datetime(2021, 7, 14), ) # Or parse and convert to CDP EventIngestionModel seattle_cdp_parsed_events = get_seattle_events( from_dt=datetime(2021, 7, 12), to_dt=datetime(2021, 7, 14), ) ``` ### Scrapers #### Event Scraper Structure Our current event scraper structure is as follows. The main function `get_events` gets all the required data and it calls the `get_content_uris` function to return the required video data. If your city uses Legistar and the Legistar data is publicly available. - You may be able to reuse our scraper with minimal modifications, such as providing the correct Legistar client ID for your municipality. - If the Legistar data returned only does not include the EventVideoPath field for the `Session.video_uri` data, you will only need to implement `get_content_uris`. If your city does not use Legistar. - You will need to build your own event scraper. Example of a completed scraper: [cdp_scrapers.instances.seattle.SeattleScraper](https://councildataproject.org/cdp-scrapers/_modules/cdp_scrapers/instances/seattle.html#SeattleScraper) For more details about creating a custom scraper for your municipality's Legistar data, please visit [here](https://councildataproject.org/cdp-scrapers/legistar_scraper.html). If you would like to deploy a CDP instance or would like to use this library as a method for retrieving formatted legislative data, please feel free to contribute a new custom municipality scraper! #### Creating a Custom Scraper If it isn't possible to use our generalized Legistar tooling to write your scraper, you will need to create your own event scraper to proceed with the deployment. 1. Please see our documentation on the [minimum data required for CDP event ingestion](https://councildataproject.org/cdp-backend/ingestion_models.html) to understand what data your scraper should return. 2. From there, begin with our [empty custom scraper function template](https://councildataproject.org/cdp-scrapers/_modules/cdp_scrapers/instances/empty.html#get_events) and fill in your scraper. 3. After your scraper is completed, you can create a pull request to add your scraper into the [cdp-scrapers repo](https://github.com/CouncilDataProject/cdp-scrapers) so it can be added into the final repo for your CDP instance. 4. Our automated action will run your scraper to verify it returns the correct data. If it is successful, you may proceed to the next deployment step. If not, we will automatically share the error message so you can fix the issue and the scraper can be tested again afterwards. ## Installation **Stable Release:** `pip install cdp-scrapers`
**Development Head:** `pip install git+https://github.com/CouncilDataProject/cdp-scrapers.git` ## Documentation For full package documentation please visit [councildataproject.org/cdp-scrapers](https://councildataproject.org/cdp-scrapers). ## Development Refer to [CONTRIBUTING.md](CONTRIBUTING.md) for information related to developing the code. **MPLv2 License** %prep %autosetup -n cdp-scrapers-0.7.0 %build %py3_build %install %py3_install install -d -m755 %{buildroot}/%{_pkgdocdir} if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi pushd %{buildroot} if [ -d usr/lib ]; then find usr/lib -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi if [ -d usr/lib64 ]; then find usr/lib64 -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi if [ -d usr/bin ]; then find usr/bin -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi if [ -d usr/sbin ]; then find usr/sbin -type f -printf "\"/%h/%f\"\n" >> filelist.lst fi touch doclist.lst if [ -d usr/share/man ]; then find usr/share/man -type f -printf "\"/%h/%f.gz\"\n" >> doclist.lst fi popd mv %{buildroot}/filelist.lst . mv %{buildroot}/doclist.lst . %files -n python3-cdp-scrapers -f filelist.lst %dir %{python3_sitelib}/* %files help -f doclist.lst %{_docdir}/* %changelog * Fri Jun 09 2023 Python_Bot - 0.7.0-1 - Package Spec generated