path: root/python-flatterer.spec

%global _empty_manifest_terminate_build 0
Name:		python-flatterer
Version:	0.18.2
Release:	1
Summary:	Opinionated JSON to CSV converter
License:	MIT
URL:		https://pypi.org/project/flatterer/
Source0:	https://mirrors.nju.edu.cn/pypi/web/packages/d4/42/c7a2bc47fd7641ee7151c269d7758627791357a9fca046429bbe05bc4080/flatterer-0.18.2.tar.gz

Requires:	python3-orjson
Requires:	python3-ijson
Requires:	python3-click
Requires:	python3-pandas

%description
<img width="500" src="./docs/_static/flatterer-with-text.svg">

[Full Documentation](http://flatterer.opendata.coop/)

## Introduction

An opinionated JSON to CSV/XLSX/SQLITE/PARQUET converter which tries to make a useful relational output for data analysis.

[Web playgroud of CSV/XLSX conversions](https://lite.flatterer.dev/)

## Rationale

When receiving a JSON file where the structure is deeply nested or not well specified, it is hard to determine what the data contains. Also, even after knowing the JSON structure, it requires a lot of time to work out how to flatten the JSON into a relational structure to do data analysis on and to be part of a data pipeline. 

Flatterer aims to be the first tool to go to when faced with the above problem.  It may not be the tool that you end up using to flatten the JSON in your data pipeline, as hand written flattening may be required, but it could be.  It has many benefits over most hand written approaches:

* It is fast, written in rust but with python bindings for ease of use.  It can be 10x faster than hand written python flattening.
* Memory efficient.  Uses a custom streaming JSON parser to mean that long list of objects nested with the JSON will be streamed, so not much data needs to be loaded into memory at once.
* Fast memory efficient output to CSV/XLSX/SQLITE/PARQUET
* Uses best practice that has been learnt from flattening JSON countless times, such as generating keys to link one-to-many tables to their parents.

### Install

```bash
pip install flatterer
```

Flatterer requires Python 3.6 or greater. It is written as a python extension in Rust but has binaries (wheels) for linux (x64), macos (x64 and universal) and windows (x64, x86).  On other platforms a rust toolchain will need to be installed.

### Example JSON

Say you have a JSON data like this named `games.json`:

```json
[
  {
    "id": 1,
    "title": "A Game",
    "releaseDate": "2015-01-01",
    "platforms": [
      {"name":"Xbox"},
      {"name":"Playstation"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  },
  {
    "id": 2,
    "title": "B Game",
    "releaseDate": "2016-01-01",
    "platforms": [
      {"name":"PC"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  }
]
```


### Running Flatterer

Run the above file with flatterer.

```bash
flatterer games.json games_dir
```

### Output Files


By running the above you will get the following files:

```bash
tree games_dir

games_dir/
├── csv
│   ├── games.csv
│   └── platforms.csv
├── datapackage.json
├── fields.csv
└── ...
```

#### Main Table

`games.csv` contains:

|_link|_link_games|id |rating_code|rating_name|releaseDate|title |
|-----|---------- |---|-----------|-----------|-----------|------|
|1    |1          |1  |E          |Everyone   |2015-01-01 |A Game|
|2    |2          |2  |E          |Everyone   |2016-01-01 |B Game|


Special column `_link` is generated. `_link` is the primary key there unique per game. 

Also the `rating` sub-object is promoted to this table it has a one-to-one relationship with `games`. 
Sub-object properties are separated by '_'.  

#### One To Many Table

`platforms` is an array so is a one-to-many with games therefore needs its own table: 
`platforms.csv` contains:

|_link|_link_games|name|
|-----|----------|----|
|1.platforms.0|1 |Xbox|
|1.platforms.1|1 |Playstation|
|2.platforms.0|2 |PC  |

#### Link Fields

`_link` is the primary key for the `platforms` table too.  Every table except `games` table, contains a `_link_games` field to easily join to the main `games` table.

If there was a sub-array of `platforms` then that would have `_link`,  `_link_games` and  `_link_platforms` fields. 

To generalize this the `_link__<table_name>` fields joins to the `_link` field of `<table_name>` i.e the `_link__<table_name>` are the foreign keys refrencing `<table_name>._link`.

#### Fields CSV

`fields.csv` contains some metadata about the output tables:

|table_name |field_name|field_type|count|field_title|
|-----------|----------|----------|-----|----------|
|platforms  |_link     |text      |3    |_link     |
|platforms  |_link_games|text     |3    |_link_games|
|platforms  |name      |text      |3    |name      |
|games      |_link     |text     |2    | _link     |
|games      |id        |number   |2    | id        |
|games      |rating_code|text    |2    | rating_code|
|games      |rating_name|text    |2    | rating_name|
|games      |releaseDate|date    |2    | releaseDate|
|games      |title     |text     |2    | title     |

The `field_type` column contains a type guess useful for inserting into a database. The `field_title` is the column heading in the CSV file or XLSX tab, which is initally the same as the field_name.
After editing this file then you can rerun the transform:


```bash
flatterer games.json new_games_dir -f myfields.csv --only-fields
```

This can be useful for renameing columns, rearranging the field order or if you want to remove some fields the `--only-fields` flag will only include the fields in the edited file.

`datapackage.json` contains metadata in the [Tabular Datapackge Spec](https://specs.frictionlessdata.io/tabular-data-package/#language)



%package -n python3-flatterer
Summary:	Opinionated JSON to CSV converter
Provides:	python-flatterer
BuildRequires:	python3-devel
BuildRequires:	python3-setuptools
BuildRequires:	python3-pip
BuildRequires:	python3-cffi
BuildRequires:	gcc
BuildRequires:	gdb
%description -n python3-flatterer
<img width="500" src="./docs/_static/flatterer-with-text.svg">

[Full Documentation](http://flatterer.opendata.coop/)

## Introduction

An opinionated JSON to CSV/XLSX/SQLITE/PARQUET converter which tries to make a useful relational output for data analysis.

[Web playgroud of CSV/XLSX conversions](https://lite.flatterer.dev/)

## Rationale

When receiving a JSON file where the structure is deeply nested or not well specified, it is hard to determine what the data contains. Also, even after knowing the JSON structure, it requires a lot of time to work out how to flatten the JSON into a relational structure to do data analysis on and to be part of a data pipeline. 

Flatterer aims to be the first tool to go to when faced with the above problem.  It may not be the tool that you end up using to flatten the JSON in your data pipeline, as hand written flattening may be required, but it could be.  It has many benefits over most hand written approaches:

* It is fast, written in rust but with python bindings for ease of use.  It can be 10x faster than hand written python flattening.
* Memory efficient.  Uses a custom streaming JSON parser to mean that long list of objects nested with the JSON will be streamed, so not much data needs to be loaded into memory at once.
* Fast memory efficient output to CSV/XLSX/SQLITE/PARQUET
* Uses best practice that has been learnt from flattening JSON countless times, such as generating keys to link one-to-many tables to their parents.

### Install

```bash
pip install flatterer
```

Flatterer requires Python 3.6 or greater. It is written as a python extension in Rust but has binaries (wheels) for linux (x64), macos (x64 and universal) and windows (x64, x86).  On other platforms a rust toolchain will need to be installed.

### Example JSON

Say you have a JSON data like this named `games.json`:

```json
[
  {
    "id": 1,
    "title": "A Game",
    "releaseDate": "2015-01-01",
    "platforms": [
      {"name":"Xbox"},
      {"name":"Playstation"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  },
  {
    "id": 2,
    "title": "B Game",
    "releaseDate": "2016-01-01",
    "platforms": [
      {"name":"PC"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  }
]
```


### Running Flatterer

Run the above file with flatterer.

```bash
flatterer games.json games_dir
```

### Output Files


By running the above you will get the following files:

```bash
tree games_dir

games_dir/
├── csv
│   ├── games.csv
│   └── platforms.csv
├── datapackage.json
├── fields.csv
└── ...
```

#### Main Table

`games.csv` contains:

|_link|_link_games|id |rating_code|rating_name|releaseDate|title |
|-----|---------- |---|-----------|-----------|-----------|------|
|1    |1          |1  |E          |Everyone   |2015-01-01 |A Game|
|2    |2          |2  |E          |Everyone   |2016-01-01 |B Game|


Special column `_link` is generated. `_link` is the primary key there unique per game. 

Also the `rating` sub-object is promoted to this table it has a one-to-one relationship with `games`. 
Sub-object properties are separated by '_'.  

#### One To Many Table

`platforms` is an array so is a one-to-many with games therefore needs its own table: 
`platforms.csv` contains:

|_link|_link_games|name|
|-----|----------|----|
|1.platforms.0|1 |Xbox|
|1.platforms.1|1 |Playstation|
|2.platforms.0|2 |PC  |

#### Link Fields

`_link` is the primary key for the `platforms` table too.  Every table except `games` table, contains a `_link_games` field to easily join to the main `games` table.

If there was a sub-array of `platforms` then that would have `_link`,  `_link_games` and  `_link_platforms` fields. 

To generalize this the `_link__<table_name>` fields joins to the `_link` field of `<table_name>` i.e the `_link__<table_name>` are the foreign keys refrencing `<table_name>._link`.

#### Fields CSV

`fields.csv` contains some metadata about the output tables:

|table_name |field_name|field_type|count|field_title|
|-----------|----------|----------|-----|----------|
|platforms  |_link     |text      |3    |_link     |
|platforms  |_link_games|text     |3    |_link_games|
|platforms  |name      |text      |3    |name      |
|games      |_link     |text     |2    | _link     |
|games      |id        |number   |2    | id        |
|games      |rating_code|text    |2    | rating_code|
|games      |rating_name|text    |2    | rating_name|
|games      |releaseDate|date    |2    | releaseDate|
|games      |title     |text     |2    | title     |

The `field_type` column contains a type guess useful for inserting into a database. The `field_title` is the column heading in the CSV file or XLSX tab, which is initally the same as the field_name.
After editing this file then you can rerun the transform:


```bash
flatterer games.json new_games_dir -f myfields.csv --only-fields
```

This can be useful for renameing columns, rearranging the field order or if you want to remove some fields the `--only-fields` flag will only include the fields in the edited file.

`datapackage.json` contains metadata in the [Tabular Datapackge Spec](https://specs.frictionlessdata.io/tabular-data-package/#language)



%package help
Summary:	Development documents and examples for flatterer
Provides:	python3-flatterer-doc
%description help
<img width="500" src="./docs/_static/flatterer-with-text.svg">

[Full Documentation](http://flatterer.opendata.coop/)

## Introduction

An opinionated JSON to CSV/XLSX/SQLITE/PARQUET converter which tries to make a useful relational output for data analysis.

[Web playgroud of CSV/XLSX conversions](https://lite.flatterer.dev/)

## Rationale

When receiving a JSON file where the structure is deeply nested or not well specified, it is hard to determine what the data contains. Also, even after knowing the JSON structure, it requires a lot of time to work out how to flatten the JSON into a relational structure to do data analysis on and to be part of a data pipeline. 

Flatterer aims to be the first tool to go to when faced with the above problem.  It may not be the tool that you end up using to flatten the JSON in your data pipeline, as hand written flattening may be required, but it could be.  It has many benefits over most hand written approaches:

* It is fast, written in rust but with python bindings for ease of use.  It can be 10x faster than hand written python flattening.
* Memory efficient.  Uses a custom streaming JSON parser to mean that long list of objects nested with the JSON will be streamed, so not much data needs to be loaded into memory at once.
* Fast memory efficient output to CSV/XLSX/SQLITE/PARQUET
* Uses best practice that has been learnt from flattening JSON countless times, such as generating keys to link one-to-many tables to their parents.

### Install

```bash
pip install flatterer
```

Flatterer requires Python 3.6 or greater. It is written as a python extension in Rust but has binaries (wheels) for linux (x64), macos (x64 and universal) and windows (x64, x86).  On other platforms a rust toolchain will need to be installed.

### Example JSON

Say you have a JSON data like this named `games.json`:

```json
[
  {
    "id": 1,
    "title": "A Game",
    "releaseDate": "2015-01-01",
    "platforms": [
      {"name":"Xbox"},
      {"name":"Playstation"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  },
  {
    "id": 2,
    "title": "B Game",
    "releaseDate": "2016-01-01",
    "platforms": [
      {"name":"PC"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  }
]
```


### Running Flatterer

Run the above file with flatterer.

```bash
flatterer games.json games_dir
```

### Output Files


By running the above you will get the following files:

```bash
tree games_dir

games_dir/
├── csv
│   ├── games.csv
│   └── platforms.csv
├── datapackage.json
├── fields.csv
└── ...
```

#### Main Table

`games.csv` contains:

|_link|_link_games|id |rating_code|rating_name|releaseDate|title |
|-----|---------- |---|-----------|-----------|-----------|------|
|1    |1          |1  |E          |Everyone   |2015-01-01 |A Game|
|2    |2          |2  |E          |Everyone   |2016-01-01 |B Game|


Special column `_link` is generated. `_link` is the primary key there unique per game. 

Also the `rating` sub-object is promoted to this table it has a one-to-one relationship with `games`. 
Sub-object properties are separated by '_'.  

#### One To Many Table

`platforms` is an array so is a one-to-many with games therefore needs its own table: 
`platforms.csv` contains:

|_link|_link_games|name|
|-----|----------|----|
|1.platforms.0|1 |Xbox|
|1.platforms.1|1 |Playstation|
|2.platforms.0|2 |PC  |

#### Link Fields

`_link` is the primary key for the `platforms` table too.  Every table except `games` table, contains a `_link_games` field to easily join to the main `games` table.

If there was a sub-array of `platforms` then that would have `_link`,  `_link_games` and  `_link_platforms` fields. 

To generalize this the `_link__<table_name>` fields joins to the `_link` field of `<table_name>` i.e the `_link__<table_name>` are the foreign keys refrencing `<table_name>._link`.

#### Fields CSV

`fields.csv` contains some metadata about the output tables:

|table_name |field_name|field_type|count|field_title|
|-----------|----------|----------|-----|----------|
|platforms  |_link     |text      |3    |_link     |
|platforms  |_link_games|text     |3    |_link_games|
|platforms  |name      |text      |3    |name      |
|games      |_link     |text     |2    | _link     |
|games      |id        |number   |2    | id        |
|games      |rating_code|text    |2    | rating_code|
|games      |rating_name|text    |2    | rating_name|
|games      |releaseDate|date    |2    | releaseDate|
|games      |title     |text     |2    | title     |

The `field_type` column contains a type guess useful for inserting into a database. The `field_title` is the column heading in the CSV file or XLSX tab, which is initally the same as the field_name.
After editing this file then you can rerun the transform:


```bash
flatterer games.json new_games_dir -f myfields.csv --only-fields
```

This can be useful for renameing columns, rearranging the field order or if you want to remove some fields the `--only-fields` flag will only include the fields in the edited file.

`datapackage.json` contains metadata in the [Tabular Datapackge Spec](https://specs.frictionlessdata.io/tabular-data-package/#language)



%prep
%autosetup -n flatterer-0.18.2

%build
%py3_build

%install
%py3_install
install -d -m755 %{buildroot}/%{_pkgdocdir}
if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
pushd %{buildroot}
if [ -d usr/lib ]; then
	find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/lib64 ]; then
	find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/bin ]; then
	find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/sbin ]; then
	find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
fi
touch doclist.lst
if [ -d usr/share/man ]; then
	find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
fi
popd
mv %{buildroot}/filelist.lst .
mv %{buildroot}/doclist.lst .

%files -n python3-flatterer -f filelist.lst
%dir %{python3_sitearch}/*

%files help -f doclist.lst
%{_docdir}/*

%changelog
* Tue Apr 25 2023 Python_Bot <Python_Bot@openeuler.org> - 0.18.2-1
- Package Spec generated