%global _empty_manifest_terminate_build 0 Name: python-OpenFisca-Survey-Manager Version: 0.47.2 Release: 1 Summary: A tool for managing survey/administrative data. License: http://www.fsf.org/licensing/licenses/agpl-3.0.html URL: https://github.com/openfisca/openfisca-survey-manager Source0: https://mirrors.nju.edu.cn/pypi/web/packages/02/4b/8d0a7596808aebdc08824e8467413f811c4951d83ff0dd9562f105e2496e/OpenFisca-Survey-Manager-0.47.2.tar.gz BuildArch: noarch Requires: python3-chardet Requires: python3-configparser Requires: python3-humanize Requires: python3-openfisca-core Requires: python3-pandas Requires: python3-pyxdg Requires: python3-PyYAML Requires: python3-tables Requires: python3-tabulate Requires: python3-weightedcalcs Requires: python3-wquantiles Requires: python3-autopep8 Requires: python3-flake8 Requires: python3-pycodestyle Requires: python3-pytest Requires: python3-scipy Requires: python3-autopep8 Requires: python3-coveralls Requires: python3-flake8 Requires: python3-flake8-bugbear Requires: python3-flake8-docstrings Requires: python3-flake8-print Requires: python3-flake8-rst-docstrings Requires: python3-openfisca-country-template Requires: python3-pytest Requires: python3-pytest-cov Requires: python3-scipy Requires: python3-tables Requires: python3-feather Requires: python3-rpy2 Requires: python3-pyreadstat Requires: python3-SAS7BDAT %description # OpenFisca Survey Manager [![Newsletter](https://img.shields.io/badge/newsletter-subscribe!-informational.svg?style=flat)](mailto:contact%40openfisca.org?subject=Subscribe%20to%20your%20newsletter%20%7C%20S'inscrire%20%C3%A0%20votre%20newsletter&body=%5BEnglish%20version%20below%5D%0A%0ABonjour%2C%0A%0AVotre%C2%A0pr%C3%A9sence%C2%A0ici%C2%A0nous%C2%A0ravit%C2%A0!%20%F0%9F%98%83%0A%0AEnvoyez-nous%20cet%20email%20pour%20que%20l'on%20puisse%20vous%20inscrire%20%C3%A0%20la%20newsletter.%20%0A%0AAh%C2%A0!%20Et%20si%20vous%20pouviez%20remplir%20ce%20petit%20questionnaire%2C%20%C3%A7a%20serait%20encore%20mieux%C2%A0!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2F45M0VR1TYKD1RGzX2%0A%0AAmiti%C3%A9%2C%0AL%E2%80%99%C3%A9quipe%20OpenFisca%0A%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%20ENGLISH%20VERSION%20%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%0AHi%2C%20%0A%0AWe're%20glad%20to%20see%20you%20here!%20%F0%9F%98%83%0A%0APlease%20send%20us%20this%20email%2C%20so%20we%20can%20subscribe%20you%20to%20the%20newsletter.%0A%0AAlso%2C%20if%20you%20can%20fill%20out%20this%20short%20survey%2C%20even%20better!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2FsOg8K1abhhm441LG2%0A%0ACheers%2C%0AThe%20OpenFisca%20Team) [![Twitter](https://img.shields.io/badge/twitter-follow%20us!-9cf.svg?style=flat)](https://twitter.com/intent/follow?screen_name=openfisca) [![Slack](https://img.shields.io/badge/slack-join%20us!-blueviolet.svg?style=flat)](mailto:contact%40openfisca.org?subject=Join%20you%20on%20Slack%20%7C%20Nous%20rejoindre%20sur%20Slack&body=%5BEnglish%20version%20below%5D%0A%0ABonjour%2C%0A%0AVotre%C2%A0pr%C3%A9sence%C2%A0ici%C2%A0nous%C2%A0ravit%C2%A0!%20%F0%9F%98%83%0A%0ARacontez-nous%20un%20peu%20de%20vous%2C%20et%20du%20pourquoi%20de%20votre%20int%C3%A9r%C3%AAt%20de%20rejoindre%20la%20communaut%C3%A9%20OpenFisca%20sur%20Slack.%0A%0AAh%C2%A0!%20Et%20si%20vous%20pouviez%20remplir%20ce%20petit%20questionnaire%2C%20%C3%A7a%20serait%20encore%20mieux%C2%A0!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2F45M0VR1TYKD1RGzX2%0A%0AN%E2%80%99oubliez%20pas%20de%20nous%20envoyer%20cet%20email%C2%A0!%20Sinon%2C%20on%20ne%20pourra%20pas%20vous%20contacter%20ni%20vous%20inviter%20sur%20Slack.%0A%0AAmiti%C3%A9%2C%0AL%E2%80%99%C3%A9quipe%20OpenFisca%0A%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%20ENGLISH%20VERSION%20%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%0AHi%2C%20%0A%0AWe're%20glad%20to%20see%20you%20here!%20%F0%9F%98%83%0A%0APlease%20tell%20us%20a%20bit%20about%20you%20and%20why%20you%20want%20to%20join%20the%20OpenFisca%20community%20on%20Slack.%0A%0AAlso%2C%20if%20you%20can%20fill%20out%20this%20short%20survey%2C%20even%20better!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2FsOg8K1abhhm441LG2.%0A%0ADon't%20forget%20to%20send%20us%20this%20email!%20Otherwise%20we%20won't%20be%20able%20to%20contact%20you%20back%2C%20nor%20invite%20you%20on%20Slack.%0A%0ACheers%2C%0AThe%20OpenFisca%20Team) [![CircleCI](https://img.shields.io/circleci/project/github/openfisca/openfisca-survey-manager/master.svg?style=flat)](https://circleci.com/gh/openfisca/openfisca-survey-manager) [![Coveralls](https://img.shields.io/coveralls/github/openfisca/openfisca-survey-manager/master.svg?style=flat)](https://coveralls.io/github/openfisca/openfisca-survey-manager?branch=master) [![Python](https://img.shields.io/pypi/pyversions/openfisca-survey-manager.svg)](https://pypi.python.org/pypi/openfisca-survey-manager) [![PyPi](https://img.shields.io/pypi/v/openfisca-survey-manager.svg?style=flat)](https://pypi.python.org/pypi/openfisca-survey-manager) ## [EN] Introduction [OpenFisca](https://openfisca.org) is a versatile microsimulation free software. You can check the [online documentation](https://openfisca.org/doc/) for more details. This repository contains the Survey-Manager module, to work with OpenFisca and survey data. It provides two main features: * A Python API to access data in [Hierarchical Data Format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) (HDF). * A script that transforms SAS, Stata, SPSS, and CSV data files to HDF data files, along with some metadata so they can be used by the Python API. > For France survey data, you might find useful information on the next steps in [openfisca-france-data](https://github.com/openfisca/openfisca-france-data) repository. ## [FR] Introduction [OpenFisca](https://openfisca.org) est un logiciel libre de micro-simulation. Pour plus d'information, vous pouvez consulter la [documentation officielle](https://openfisca.org/doc/). Ce dépôt contient le module Survey-Manager. Il facilite l'usage d'OpenFisca avec des données d'enquête. Il fournit deux fonctionnalités principales: * Une API Python permettant l'accès à des données au format [Hierarchical Data Format](https://fr.wikipedia.org/wiki/Hierarchical_Data_Format) (HDF). * Un script qui tranforme les fichiers de données aux formats SAS, Stata, SPSS, and CSV data files en fichiers de données au format HDF, avec quelques metadonnées leur permettant d'être utilisés par l'API Python. > Si vous disposez de données d'enquête sur la France, le dépôt [openfisca-france-data](https://github.com/openfisca/openfisca-france-data) pourrait être utile à vos prochaines étapes de traitement. ## Environment OpenFisca-Survey-Manager runs on Python 3.7. More recent versions should work, but are not tested. > Backward compatibility with Python 2.7 has been dropped since January 1st, 2019. ## Usage ### Installation #### Install with PIP If you're developing your own script or looking to run `OpenFisca-Survey-Manager` without editing it, you don't need to get its source code. It just needs to be known by your environment. To do so, first, install the package with `pip`: ```shell pip install --upgrade pip pip install openfisca-survey-manager ``` This should not display any error and end with: `Successfully installed [... openfisca-survey-manager-xx.xx.xx ...]` It comes with `build-collection` command that we will use in the next steps. > If you want to improve this module, please see the `Development` section below. #### Install with Conda Create an anvironment and install openfisca-survey-manager ``` conda create -n survey python=3.8 conda activate survey conda install -c conda-forge -c openfisca openfisca-survey-manager ``` You are ready to go ! To exit your environment: ``` conda deactivate ``` ### Getting the configuration directory path To be able to use OpenFisca-Survey-Manager, you have to create two configuration files: * `raw_data.ini`, * and `config.ini`. To know where to copy them to, use the following command: ```shell build-collection --help ``` You should get the following result. ```shell usage: build-collection [-h] -c COLLECTION [-d] [-m] [-p PATH] [-s SURVEY] [-v] optional arguments: -h, --help show this help message and exit -c COLLECTION, --collection COLLECTION name of collection to build or update -d, --replace-data erase existing survey data HDF5 file (instead of failing when HDF5 file already exists) -m, --replace-metadata erase existing collection metadata JSON file (instead of just adding new surveys) -p PATH, --path PATH path to the config files directory (default = /your/path/.config/openfisca-survey-manager) -s SURVEY, --survey SURVEY name of survey to build or update (default = all) -v, --verbose increase output verbosity ``` Take note of the default configuration directory path in `-p PATH, --path PATH` option's description. This is the directory where you will set your `raw_data.ini` and `config.ini` files. In this example, it is `/Users/you/.config/openfisca-survey-manager`. > If you want to use a different path, you can pass the `--path /another/path` option to `build-collection`. This feature is still experimental though. ### Editing the config files Configuration files are INI files (text files). The `raw_data.ini` lists your input surveys while `config.ini` specifies the paths to SurveyManager outputs. > `raw_data.ini` and `config.ini` must not be committed (they are already ignored by [`.gitignore`](.gitignore)). #### raw_data.ini, for inputs configuration To initialise your `raw_data.ini` file, you can follow these steps: 1. Copy the template file [raw_data_template.ini](openfisca_survey_manager/config_files_templates/raw_data_template.ini) to the configuration directory path you identified in the previous step and rename it to `raw_data.ini`. Ex: `/your/path/.config/openfisca-survey-manager/raw_data.ini` 2. Edit the latter by adding a section title for your survey. For example, if you name your survey `housing_survey`, you should get a line with: ```ini [housing_survey] ``` 3. Add a reference to the location of your raw data **directory** (SAS, stata DTA files, SPSS, CSV files). For paths in Windows, use `/` instead of `\` to separate folders. You do not need to put quotes, even when the path name contains spaces. Your file should look like this: ```ini [housing_survey] 2014 = /path/to/your/raw/data/HOUSING_2014 ``` You can also set multiple surveys as follows: ```ini [revenue_survey] 2014 = /path/to/your/raw/data/REVENUE_2014 2015 = /path/to/your/raw/data/REVENUE_2015 2016 = /path/to/your/raw/data/REVENUE_2016 [housing_survey] 2014 = /path/to/your/raw/data/HOUSING_2014 ``` #### config.ini, for outputs configuration To initilalise your `config.ini` file: 1. Copy its template file [config_template.ini](openfisca_survey_manager/config_files_templates/config_template.ini) to your configuration directory and rename it to `config.ini`. Ex: `/your/path/.config/openfisca-survey-manager/config.ini`. 2. Define a `collections_directory` path where the SurveyManager will generate your survey inputs and outputs JSON description. Ex: `/.../openfisca-survey-manager/transformed_housing_survey` For a `housing_survey`, you will get a `/.../openfisca-survey-manager/transformed_housing_survey/housing_survey.json` file. 3. Define an `output_directory` where the generated HDF file will be registered. This directory could be a sub-directory of your `collections_directory`. 4. Define a `tmp_directory` that will store temporay calculation results. Its content will be deleted at the end of the calculation. This directory could be a sub-directory of your `collections_directory`. Your `config.ini` file should look similar to this: ```ini [collections] collections_directory = /path/to/your/collections/directory [data] output_directory = /path/to/your/data/output/directory tmp_directory = /path/to/your/data/tmp/directory ``` > Make sure those directories exist, otherwise the script will fail. ### Building the HDF5 files This step will read your configuration files and you survey data and generate a HDF5 file (`.h5`) for your survey. To build the HDF5 files, we'll use the [`build-collection`](openfisca_survey_manager/scripts/build_collection.py) script. Here is an example for one survey with one serie: our `housing_survey` that knows only 2014 serie. We call our survey as a collection (with `-c` option) and build the HDF5 file with this command: ```shell build-collection -c housing_survey -d -m -v ``` `-d -m` options put you on the safe side as they remove previous outputs if they exist. It will generate: * A `housing_survey.json` listing a `housing_survey_2014` survey with both: * your input `tables` and your input file paths in an `informations` key, * the transformed survey path in a `hdf5_file_path` key. * Your transformed survey in a `housing_survey_2014.h5` file. #### build-collection, what else? As `build-collection --help` shows, other options exist. Here are other usage examples. If you have multiple series of one survey like the `revenue_survey`, you can run the specific `2015` serie with: ```shell build-collection -c revenue_survey -s 2015 -d -m -v ``` Or if you want to specify a different configuration directory path: ```shell build-collection -p /another/path -c housing_survey -s 2014 -d -m -v ``` > The `--path /another/path` option is still experimental though. It should work. If it doesn't, please do not hesitate to [open an issue](https://github.com/openfisca/openfisca-survey-manager/issues/new). ## Development If you want to contribute to OpenFisca-Survey-Manager, please be welcomed! To install it locally in development mode: ```shell git clone https://github.com/openfisca/openfisca-survey-manager.git cd openfisca-survey-manager make install ``` ## Testing To run the entire test suite: ```sh make test ``` To run the entire test suite with the same config as in CI: ```sh CI=True make test ``` ## Style This repository adheres to a certain coding style, and we invite you to follow it for your contributions to be integrated promptly. To run the style checker: ```sh make check-style ``` To automatically style-format your code changes: ```sh make format-style ``` To automatically style-format your code changes each time you commit: ```sh touch .git/hooks/pre-commit chmod +x .git/hooks/pre-commit tee -a .git/hooks/pre-commit << END #!/bin/sh # # Automatically format your code before committing. exec make format-style END ``` %package -n python3-OpenFisca-Survey-Manager Summary: A tool for managing survey/administrative data. Provides: python-OpenFisca-Survey-Manager BuildRequires: python3-devel BuildRequires: python3-setuptools BuildRequires: python3-pip %description -n python3-OpenFisca-Survey-Manager # OpenFisca Survey Manager [![Newsletter](https://img.shields.io/badge/newsletter-subscribe!-informational.svg?style=flat)](mailto:contact%40openfisca.org?subject=Subscribe%20to%20your%20newsletter%20%7C%20S'inscrire%20%C3%A0%20votre%20newsletter&body=%5BEnglish%20version%20below%5D%0A%0ABonjour%2C%0A%0AVotre%C2%A0pr%C3%A9sence%C2%A0ici%C2%A0nous%C2%A0ravit%C2%A0!%20%F0%9F%98%83%0A%0AEnvoyez-nous%20cet%20email%20pour%20que%20l'on%20puisse%20vous%20inscrire%20%C3%A0%20la%20newsletter.%20%0A%0AAh%C2%A0!%20Et%20si%20vous%20pouviez%20remplir%20ce%20petit%20questionnaire%2C%20%C3%A7a%20serait%20encore%20mieux%C2%A0!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2F45M0VR1TYKD1RGzX2%0A%0AAmiti%C3%A9%2C%0AL%E2%80%99%C3%A9quipe%20OpenFisca%0A%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%20ENGLISH%20VERSION%20%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%0AHi%2C%20%0A%0AWe're%20glad%20to%20see%20you%20here!%20%F0%9F%98%83%0A%0APlease%20send%20us%20this%20email%2C%20so%20we%20can%20subscribe%20you%20to%20the%20newsletter.%0A%0AAlso%2C%20if%20you%20can%20fill%20out%20this%20short%20survey%2C%20even%20better!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2FsOg8K1abhhm441LG2%0A%0ACheers%2C%0AThe%20OpenFisca%20Team) [![Twitter](https://img.shields.io/badge/twitter-follow%20us!-9cf.svg?style=flat)](https://twitter.com/intent/follow?screen_name=openfisca) [![Slack](https://img.shields.io/badge/slack-join%20us!-blueviolet.svg?style=flat)](mailto:contact%40openfisca.org?subject=Join%20you%20on%20Slack%20%7C%20Nous%20rejoindre%20sur%20Slack&body=%5BEnglish%20version%20below%5D%0A%0ABonjour%2C%0A%0AVotre%C2%A0pr%C3%A9sence%C2%A0ici%C2%A0nous%C2%A0ravit%C2%A0!%20%F0%9F%98%83%0A%0ARacontez-nous%20un%20peu%20de%20vous%2C%20et%20du%20pourquoi%20de%20votre%20int%C3%A9r%C3%AAt%20de%20rejoindre%20la%20communaut%C3%A9%20OpenFisca%20sur%20Slack.%0A%0AAh%C2%A0!%20Et%20si%20vous%20pouviez%20remplir%20ce%20petit%20questionnaire%2C%20%C3%A7a%20serait%20encore%20mieux%C2%A0!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2F45M0VR1TYKD1RGzX2%0A%0AN%E2%80%99oubliez%20pas%20de%20nous%20envoyer%20cet%20email%C2%A0!%20Sinon%2C%20on%20ne%20pourra%20pas%20vous%20contacter%20ni%20vous%20inviter%20sur%20Slack.%0A%0AAmiti%C3%A9%2C%0AL%E2%80%99%C3%A9quipe%20OpenFisca%0A%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%20ENGLISH%20VERSION%20%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%0AHi%2C%20%0A%0AWe're%20glad%20to%20see%20you%20here!%20%F0%9F%98%83%0A%0APlease%20tell%20us%20a%20bit%20about%20you%20and%20why%20you%20want%20to%20join%20the%20OpenFisca%20community%20on%20Slack.%0A%0AAlso%2C%20if%20you%20can%20fill%20out%20this%20short%20survey%2C%20even%20better!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2FsOg8K1abhhm441LG2.%0A%0ADon't%20forget%20to%20send%20us%20this%20email!%20Otherwise%20we%20won't%20be%20able%20to%20contact%20you%20back%2C%20nor%20invite%20you%20on%20Slack.%0A%0ACheers%2C%0AThe%20OpenFisca%20Team) [![CircleCI](https://img.shields.io/circleci/project/github/openfisca/openfisca-survey-manager/master.svg?style=flat)](https://circleci.com/gh/openfisca/openfisca-survey-manager) [![Coveralls](https://img.shields.io/coveralls/github/openfisca/openfisca-survey-manager/master.svg?style=flat)](https://coveralls.io/github/openfisca/openfisca-survey-manager?branch=master) [![Python](https://img.shields.io/pypi/pyversions/openfisca-survey-manager.svg)](https://pypi.python.org/pypi/openfisca-survey-manager) [![PyPi](https://img.shields.io/pypi/v/openfisca-survey-manager.svg?style=flat)](https://pypi.python.org/pypi/openfisca-survey-manager) ## [EN] Introduction [OpenFisca](https://openfisca.org) is a versatile microsimulation free software. You can check the [online documentation](https://openfisca.org/doc/) for more details. This repository contains the Survey-Manager module, to work with OpenFisca and survey data. It provides two main features: * A Python API to access data in [Hierarchical Data Format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) (HDF). * A script that transforms SAS, Stata, SPSS, and CSV data files to HDF data files, along with some metadata so they can be used by the Python API. > For France survey data, you might find useful information on the next steps in [openfisca-france-data](https://github.com/openfisca/openfisca-france-data) repository. ## [FR] Introduction [OpenFisca](https://openfisca.org) est un logiciel libre de micro-simulation. Pour plus d'information, vous pouvez consulter la [documentation officielle](https://openfisca.org/doc/). Ce dépôt contient le module Survey-Manager. Il facilite l'usage d'OpenFisca avec des données d'enquête. Il fournit deux fonctionnalités principales: * Une API Python permettant l'accès à des données au format [Hierarchical Data Format](https://fr.wikipedia.org/wiki/Hierarchical_Data_Format) (HDF). * Un script qui tranforme les fichiers de données aux formats SAS, Stata, SPSS, and CSV data files en fichiers de données au format HDF, avec quelques metadonnées leur permettant d'être utilisés par l'API Python. > Si vous disposez de données d'enquête sur la France, le dépôt [openfisca-france-data](https://github.com/openfisca/openfisca-france-data) pourrait être utile à vos prochaines étapes de traitement. ## Environment OpenFisca-Survey-Manager runs on Python 3.7. More recent versions should work, but are not tested. > Backward compatibility with Python 2.7 has been dropped since January 1st, 2019. ## Usage ### Installation #### Install with PIP If you're developing your own script or looking to run `OpenFisca-Survey-Manager` without editing it, you don't need to get its source code. It just needs to be known by your environment. To do so, first, install the package with `pip`: ```shell pip install --upgrade pip pip install openfisca-survey-manager ``` This should not display any error and end with: `Successfully installed [... openfisca-survey-manager-xx.xx.xx ...]` It comes with `build-collection` command that we will use in the next steps. > If you want to improve this module, please see the `Development` section below. #### Install with Conda Create an anvironment and install openfisca-survey-manager ``` conda create -n survey python=3.8 conda activate survey conda install -c conda-forge -c openfisca openfisca-survey-manager ``` You are ready to go ! To exit your environment: ``` conda deactivate ``` ### Getting the configuration directory path To be able to use OpenFisca-Survey-Manager, you have to create two configuration files: * `raw_data.ini`, * and `config.ini`. To know where to copy them to, use the following command: ```shell build-collection --help ``` You should get the following result. ```shell usage: build-collection [-h] -c COLLECTION [-d] [-m] [-p PATH] [-s SURVEY] [-v] optional arguments: -h, --help show this help message and exit -c COLLECTION, --collection COLLECTION name of collection to build or update -d, --replace-data erase existing survey data HDF5 file (instead of failing when HDF5 file already exists) -m, --replace-metadata erase existing collection metadata JSON file (instead of just adding new surveys) -p PATH, --path PATH path to the config files directory (default = /your/path/.config/openfisca-survey-manager) -s SURVEY, --survey SURVEY name of survey to build or update (default = all) -v, --verbose increase output verbosity ``` Take note of the default configuration directory path in `-p PATH, --path PATH` option's description. This is the directory where you will set your `raw_data.ini` and `config.ini` files. In this example, it is `/Users/you/.config/openfisca-survey-manager`. > If you want to use a different path, you can pass the `--path /another/path` option to `build-collection`. This feature is still experimental though. ### Editing the config files Configuration files are INI files (text files). The `raw_data.ini` lists your input surveys while `config.ini` specifies the paths to SurveyManager outputs. > `raw_data.ini` and `config.ini` must not be committed (they are already ignored by [`.gitignore`](.gitignore)). #### raw_data.ini, for inputs configuration To initialise your `raw_data.ini` file, you can follow these steps: 1. Copy the template file [raw_data_template.ini](openfisca_survey_manager/config_files_templates/raw_data_template.ini) to the configuration directory path you identified in the previous step and rename it to `raw_data.ini`. Ex: `/your/path/.config/openfisca-survey-manager/raw_data.ini` 2. Edit the latter by adding a section title for your survey. For example, if you name your survey `housing_survey`, you should get a line with: ```ini [housing_survey] ``` 3. Add a reference to the location of your raw data **directory** (SAS, stata DTA files, SPSS, CSV files). For paths in Windows, use `/` instead of `\` to separate folders. You do not need to put quotes, even when the path name contains spaces. Your file should look like this: ```ini [housing_survey] 2014 = /path/to/your/raw/data/HOUSING_2014 ``` You can also set multiple surveys as follows: ```ini [revenue_survey] 2014 = /path/to/your/raw/data/REVENUE_2014 2015 = /path/to/your/raw/data/REVENUE_2015 2016 = /path/to/your/raw/data/REVENUE_2016 [housing_survey] 2014 = /path/to/your/raw/data/HOUSING_2014 ``` #### config.ini, for outputs configuration To initilalise your `config.ini` file: 1. Copy its template file [config_template.ini](openfisca_survey_manager/config_files_templates/config_template.ini) to your configuration directory and rename it to `config.ini`. Ex: `/your/path/.config/openfisca-survey-manager/config.ini`. 2. Define a `collections_directory` path where the SurveyManager will generate your survey inputs and outputs JSON description. Ex: `/.../openfisca-survey-manager/transformed_housing_survey` For a `housing_survey`, you will get a `/.../openfisca-survey-manager/transformed_housing_survey/housing_survey.json` file. 3. Define an `output_directory` where the generated HDF file will be registered. This directory could be a sub-directory of your `collections_directory`. 4. Define a `tmp_directory` that will store temporay calculation results. Its content will be deleted at the end of the calculation. This directory could be a sub-directory of your `collections_directory`. Your `config.ini` file should look similar to this: ```ini [collections] collections_directory = /path/to/your/collections/directory [data] output_directory = /path/to/your/data/output/directory tmp_directory = /path/to/your/data/tmp/directory ``` > Make sure those directories exist, otherwise the script will fail. ### Building the HDF5 files This step will read your configuration files and you survey data and generate a HDF5 file (`.h5`) for your survey. To build the HDF5 files, we'll use the [`build-collection`](openfisca_survey_manager/scripts/build_collection.py) script. Here is an example for one survey with one serie: our `housing_survey` that knows only 2014 serie. We call our survey as a collection (with `-c` option) and build the HDF5 file with this command: ```shell build-collection -c housing_survey -d -m -v ``` `-d -m` options put you on the safe side as they remove previous outputs if they exist. It will generate: * A `housing_survey.json` listing a `housing_survey_2014` survey with both: * your input `tables` and your input file paths in an `informations` key, * the transformed survey path in a `hdf5_file_path` key. * Your transformed survey in a `housing_survey_2014.h5` file. #### build-collection, what else? As `build-collection --help` shows, other options exist. Here are other usage examples. If you have multiple series of one survey like the `revenue_survey`, you can run the specific `2015` serie with: ```shell build-collection -c revenue_survey -s 2015 -d -m -v ``` Or if you want to specify a different configuration directory path: ```shell build-collection -p /another/path -c housing_survey -s 2014 -d -m -v ``` > The `--path /another/path` option is still experimental though. It should work. If it doesn't, please do not hesitate to [open an issue](https://github.com/openfisca/openfisca-survey-manager/issues/new). ## Development If you want to contribute to OpenFisca-Survey-Manager, please be welcomed! To install it locally in development mode: ```shell git clone https://github.com/openfisca/openfisca-survey-manager.git cd openfisca-survey-manager make install ``` ## Testing To run the entire test suite: ```sh make test ``` To run the entire test suite with the same config as in CI: ```sh CI=True make test ``` ## Style This repository adheres to a certain coding style, and we invite you to follow it for your contributions to be integrated promptly. To run the style checker: ```sh make check-style ``` To automatically style-format your code changes: ```sh make format-style ``` To automatically style-format your code changes each time you commit: ```sh touch .git/hooks/pre-commit chmod +x .git/hooks/pre-commit tee -a .git/hooks/pre-commit << END #!/bin/sh # # Automatically format your code before committing. exec make format-style END ``` %package help Summary: Development documents and examples for OpenFisca-Survey-Manager Provides: python3-OpenFisca-Survey-Manager-doc %description help # OpenFisca Survey Manager [![Newsletter](https://img.shields.io/badge/newsletter-subscribe!-informational.svg?style=flat)](mailto:contact%40openfisca.org?subject=Subscribe%20to%20your%20newsletter%20%7C%20S'inscrire%20%C3%A0%20votre%20newsletter&body=%5BEnglish%20version%20below%5D%0A%0ABonjour%2C%0A%0AVotre%C2%A0pr%C3%A9sence%C2%A0ici%C2%A0nous%C2%A0ravit%C2%A0!%20%F0%9F%98%83%0A%0AEnvoyez-nous%20cet%20email%20pour%20que%20l'on%20puisse%20vous%20inscrire%20%C3%A0%20la%20newsletter.%20%0A%0AAh%C2%A0!%20Et%20si%20vous%20pouviez%20remplir%20ce%20petit%20questionnaire%2C%20%C3%A7a%20serait%20encore%20mieux%C2%A0!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2F45M0VR1TYKD1RGzX2%0A%0AAmiti%C3%A9%2C%0AL%E2%80%99%C3%A9quipe%20OpenFisca%0A%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%20ENGLISH%20VERSION%20%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%0AHi%2C%20%0A%0AWe're%20glad%20to%20see%20you%20here!%20%F0%9F%98%83%0A%0APlease%20send%20us%20this%20email%2C%20so%20we%20can%20subscribe%20you%20to%20the%20newsletter.%0A%0AAlso%2C%20if%20you%20can%20fill%20out%20this%20short%20survey%2C%20even%20better!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2FsOg8K1abhhm441LG2%0A%0ACheers%2C%0AThe%20OpenFisca%20Team) [![Twitter](https://img.shields.io/badge/twitter-follow%20us!-9cf.svg?style=flat)](https://twitter.com/intent/follow?screen_name=openfisca) [![Slack](https://img.shields.io/badge/slack-join%20us!-blueviolet.svg?style=flat)](mailto:contact%40openfisca.org?subject=Join%20you%20on%20Slack%20%7C%20Nous%20rejoindre%20sur%20Slack&body=%5BEnglish%20version%20below%5D%0A%0ABonjour%2C%0A%0AVotre%C2%A0pr%C3%A9sence%C2%A0ici%C2%A0nous%C2%A0ravit%C2%A0!%20%F0%9F%98%83%0A%0ARacontez-nous%20un%20peu%20de%20vous%2C%20et%20du%20pourquoi%20de%20votre%20int%C3%A9r%C3%AAt%20de%20rejoindre%20la%20communaut%C3%A9%20OpenFisca%20sur%20Slack.%0A%0AAh%C2%A0!%20Et%20si%20vous%20pouviez%20remplir%20ce%20petit%20questionnaire%2C%20%C3%A7a%20serait%20encore%20mieux%C2%A0!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2F45M0VR1TYKD1RGzX2%0A%0AN%E2%80%99oubliez%20pas%20de%20nous%20envoyer%20cet%20email%C2%A0!%20Sinon%2C%20on%20ne%20pourra%20pas%20vous%20contacter%20ni%20vous%20inviter%20sur%20Slack.%0A%0AAmiti%C3%A9%2C%0AL%E2%80%99%C3%A9quipe%20OpenFisca%0A%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%20ENGLISH%20VERSION%20%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%0AHi%2C%20%0A%0AWe're%20glad%20to%20see%20you%20here!%20%F0%9F%98%83%0A%0APlease%20tell%20us%20a%20bit%20about%20you%20and%20why%20you%20want%20to%20join%20the%20OpenFisca%20community%20on%20Slack.%0A%0AAlso%2C%20if%20you%20can%20fill%20out%20this%20short%20survey%2C%20even%20better!%0Ahttps%3A%2F%2Fgoo.gl%2Fforms%2FsOg8K1abhhm441LG2.%0A%0ADon't%20forget%20to%20send%20us%20this%20email!%20Otherwise%20we%20won't%20be%20able%20to%20contact%20you%20back%2C%20nor%20invite%20you%20on%20Slack.%0A%0ACheers%2C%0AThe%20OpenFisca%20Team) [![CircleCI](https://img.shields.io/circleci/project/github/openfisca/openfisca-survey-manager/master.svg?style=flat)](https://circleci.com/gh/openfisca/openfisca-survey-manager) [![Coveralls](https://img.shields.io/coveralls/github/openfisca/openfisca-survey-manager/master.svg?style=flat)](https://coveralls.io/github/openfisca/openfisca-survey-manager?branch=master) [![Python](https://img.shields.io/pypi/pyversions/openfisca-survey-manager.svg)](https://pypi.python.org/pypi/openfisca-survey-manager) [![PyPi](https://img.shields.io/pypi/v/openfisca-survey-manager.svg?style=flat)](https://pypi.python.org/pypi/openfisca-survey-manager) ## [EN] Introduction [OpenFisca](https://openfisca.org) is a versatile microsimulation free software. You can check the [online documentation](https://openfisca.org/doc/) for more details. This repository contains the Survey-Manager module, to work with OpenFisca and survey data. It provides two main features: * A Python API to access data in [Hierarchical Data Format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) (HDF). * A script that transforms SAS, Stata, SPSS, and CSV data files to HDF data files, along with some metadata so they can be used by the Python API. > For France survey data, you might find useful information on the next steps in [openfisca-france-data](https://github.com/openfisca/openfisca-france-data) repository. ## [FR] Introduction [OpenFisca](https://openfisca.org) est un logiciel libre de micro-simulation. Pour plus d'information, vous pouvez consulter la [documentation officielle](https://openfisca.org/doc/). Ce dépôt contient le module Survey-Manager. Il facilite l'usage d'OpenFisca avec des données d'enquête. Il fournit deux fonctionnalités principales: * Une API Python permettant l'accès à des données au format [Hierarchical Data Format](https://fr.wikipedia.org/wiki/Hierarchical_Data_Format) (HDF). * Un script qui tranforme les fichiers de données aux formats SAS, Stata, SPSS, and CSV data files en fichiers de données au format HDF, avec quelques metadonnées leur permettant d'être utilisés par l'API Python. > Si vous disposez de données d'enquête sur la France, le dépôt [openfisca-france-data](https://github.com/openfisca/openfisca-france-data) pourrait être utile à vos prochaines étapes de traitement. ## Environment OpenFisca-Survey-Manager runs on Python 3.7. More recent versions should work, but are not tested. > Backward compatibility with Python 2.7 has been dropped since January 1st, 2019. ## Usage ### Installation #### Install with PIP If you're developing your own script or looking to run `OpenFisca-Survey-Manager` without editing it, you don't need to get its source code. It just needs to be known by your environment. To do so, first, install the package with `pip`: ```shell pip install --upgrade pip pip install openfisca-survey-manager ``` This should not display any error and end with: `Successfully installed [... openfisca-survey-manager-xx.xx.xx ...]` It comes with `build-collection` command that we will use in the next steps. > If you want to improve this module, please see the `Development` section below. #### Install with Conda Create an anvironment and install openfisca-survey-manager ``` conda create -n survey python=3.8 conda activate survey conda install -c conda-forge -c openfisca openfisca-survey-manager ``` You are ready to go ! To exit your environment: ``` conda deactivate ``` ### Getting the configuration directory path To be able to use OpenFisca-Survey-Manager, you have to create two configuration files: * `raw_data.ini`, * and `config.ini`. To know where to copy them to, use the following command: ```shell build-collection --help ``` You should get the following result. ```shell usage: build-collection [-h] -c COLLECTION [-d] [-m] [-p PATH] [-s SURVEY] [-v] optional arguments: -h, --help show this help message and exit -c COLLECTION, --collection COLLECTION name of collection to build or update -d, --replace-data erase existing survey data HDF5 file (instead of failing when HDF5 file already exists) -m, --replace-metadata erase existing collection metadata JSON file (instead of just adding new surveys) -p PATH, --path PATH path to the config files directory (default = /your/path/.config/openfisca-survey-manager) -s SURVEY, --survey SURVEY name of survey to build or update (default = all) -v, --verbose increase output verbosity ``` Take note of the default configuration directory path in `-p PATH, --path PATH` option's description. This is the directory where you will set your `raw_data.ini` and `config.ini` files. In this example, it is `/Users/you/.config/openfisca-survey-manager`. > If you want to use a different path, you can pass the `--path /another/path` option to `build-collection`. This feature is still experimental though. ### Editing the config files Configuration files are INI files (text files). The `raw_data.ini` lists your input surveys while `config.ini` specifies the paths to SurveyManager outputs. > `raw_data.ini` and `config.ini` must not be committed (they are already ignored by [`.gitignore`](.gitignore)). #### raw_data.ini, for inputs configuration To initialise your `raw_data.ini` file, you can follow these steps: 1. Copy the template file [raw_data_template.ini](openfisca_survey_manager/config_files_templates/raw_data_template.ini) to the configuration directory path you identified in the previous step and rename it to `raw_data.ini`. Ex: `/your/path/.config/openfisca-survey-manager/raw_data.ini` 2. Edit the latter by adding a section title for your survey. For example, if you name your survey `housing_survey`, you should get a line with: ```ini [housing_survey] ``` 3. Add a reference to the location of your raw data **directory** (SAS, stata DTA files, SPSS, CSV files). For paths in Windows, use `/` instead of `\` to separate folders. You do not need to put quotes, even when the path name contains spaces. Your file should look like this: ```ini [housing_survey] 2014 = /path/to/your/raw/data/HOUSING_2014 ``` You can also set multiple surveys as follows: ```ini [revenue_survey] 2014 = /path/to/your/raw/data/REVENUE_2014 2015 = /path/to/your/raw/data/REVENUE_2015 2016 = /path/to/your/raw/data/REVENUE_2016 [housing_survey] 2014 = /path/to/your/raw/data/HOUSING_2014 ``` #### config.ini, for outputs configuration To initilalise your `config.ini` file: 1. Copy its template file [config_template.ini](openfisca_survey_manager/config_files_templates/config_template.ini) to your configuration directory and rename it to `config.ini`. Ex: `/your/path/.config/openfisca-survey-manager/config.ini`. 2. Define a `collections_directory` path where the SurveyManager will generate your survey inputs and outputs JSON description. Ex: `/.../openfisca-survey-manager/transformed_housing_survey` For a `housing_survey`, you will get a `/.../openfisca-survey-manager/transformed_housing_survey/housing_survey.json` file. 3. Define an `output_directory` where the generated HDF file will be registered. This directory could be a sub-directory of your `collections_directory`. 4. Define a `tmp_directory` that will store temporay calculation results. Its content will be deleted at the end of the calculation. This directory could be a sub-directory of your `collections_directory`. Your `config.ini` file should look similar to this: ```ini [collections] collections_directory = /path/to/your/collections/directory [data] output_directory = /path/to/your/data/output/directory tmp_directory = /path/to/your/data/tmp/directory ``` > Make sure those directories exist, otherwise the script will fail. ### Building the HDF5 files This step will read your configuration files and you survey data and generate a HDF5 file (`.h5`) for your survey. To build the HDF5 files, we'll use the [`build-collection`](openfisca_survey_manager/scripts/build_collection.py) script. Here is an example for one survey with one serie: our `housing_survey` that knows only 2014 serie. We call our survey as a collection (with `-c` option) and build the HDF5 file with this command: ```shell build-collection -c housing_survey -d -m -v ``` `-d -m` options put you on the safe side as they remove previous outputs if they exist. It will generate: * A `housing_survey.json` listing a `housing_survey_2014` survey with both: * your input `tables` and your input file paths in an `informations` key, * the transformed survey path in a `hdf5_file_path` key. * Your transformed survey in a `housing_survey_2014.h5` file. #### build-collection, what else? As `build-collection --help` shows, other options exist. Here are other usage examples. If you have multiple series of one survey like the `revenue_survey`, you can run the specific `2015` serie with: ```shell build-collection -c revenue_survey -s 2015 -d -m -v ``` Or if you want to specify a different configuration directory path: ```shell build-collection -p /another/path -c housing_survey -s 2014 -d -m -v ``` > The `--path /another/path` option is still experimental though. It should work. If it doesn't, please do not hesitate to [open an issue](https://github.com/openfisca/openfisca-survey-manager/issues/new). ## Development If you want to contribute to OpenFisca-Survey-Manager, please be welcomed! To install it locally in development mode: ```shell git clone https://github.com/openfisca/openfisca-survey-manager.git cd openfisca-survey-manager make install ``` ## Testing To run the entire test suite: ```sh make test ``` To run the entire test suite with the same config as in CI: ```sh CI=True make test ``` ## Style This repository adheres to a certain coding style, and we invite you to follow it for your contributions to be integrated promptly. To run the style checker: ```sh make check-style ``` To automatically style-format your code changes: ```sh make format-style ``` To automatically style-format your code changes each time you commit: ```sh touch .git/hooks/pre-commit chmod +x .git/hooks/pre-commit tee -a .git/hooks/pre-commit << END #!/bin/sh # # Automatically format your code before committing. exec make format-style END ``` %prep %autosetup -n OpenFisca-Survey-Manager-0.47.2 %build %py3_build %install %py3_install install -d -m755 %{buildroot}/%{_pkgdocdir} if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi pushd %{buildroot} if [ -d usr/lib ]; then find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst fi if [ -d usr/lib64 ]; then find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst fi if [ -d usr/bin ]; then find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst fi if [ -d usr/sbin ]; then find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst fi touch doclist.lst if [ -d usr/share/man ]; then find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst fi popd mv %{buildroot}/filelist.lst . mv %{buildroot}/doclist.lst . %files -n python3-OpenFisca-Survey-Manager -f filelist.lst %dir %{python3_sitelib}/* %files help -f doclist.lst %{_docdir}/* %changelog * Fri May 05 2023 Python_Bot - 0.47.2-1 - Package Spec generated