diff options
| author | CoprDistGit <infra@openeuler.org> | 2023-04-11 00:43:14 +0000 |
|---|---|---|
| committer | CoprDistGit <infra@openeuler.org> | 2023-04-11 00:43:14 +0000 |
| commit | 03c919ee699628e94b4fbd1671aead988a22d277 (patch) | |
| tree | 559b5918c15a59ccf00ec81f7db4ce1c97072f07 | |
| parent | c9953cb83aa5785094cbfe76b80bbb02768f1dfc (diff) | |
automatic import of python-nietopeneuler20.03
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | python-niet.spec | 1974 | ||||
| -rw-r--r-- | sources | 1 |
3 files changed, 1976 insertions, 0 deletions
@@ -0,0 +1 @@ +/niet-2.5.0.tar.gz diff --git a/python-niet.spec b/python-niet.spec new file mode 100644 index 0000000..10c96f7 --- /dev/null +++ b/python-niet.spec @@ -0,0 +1,1974 @@ +%global _empty_manifest_terminate_build 0 +Name: python-niet +Version: 2.5.0 +Release: 1 +Summary: Shell config file parser (json, yaml) +License: MIT +URL: https://github.com/openuado/niet/ +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/2f/74/138a39e06d992f1f80935995be5a21e3ec37e72513a34851308ed2b22cdd/niet-2.5.0.tar.gz +BuildArch: noarch + + +%description +> How to easily parse and retrieve data from YAML file in our shell? +The previous question, few years ago, led us to the development of niet. +Indeed, at that time, we needed a way to store and retrieve data for our own +needs. We created niet to read those data. The goal was to develop a tools +that will allow us to standardize how we parse YAML locally or in our CI +pipelines. We wanted something reusable and easily distribuable. Niet was born. +Niet is like [xmllint](http://xmlsoft.org/xmllint.html) or +[jq](https://stedolan.github.io/jq/) but for YAML and JSON data - +you can use it to slice and filter and map and transform structured data. +You can easily retrieve data by using simple expressions or using +xpath advanced features to access non-trivial data. +You can easily convert YAML format into JSON format and vice versa. +Niet is writen in Python so you can install it from a package manager (from +PyPi) or directly by cloning this repository - no specific system rights are +needed to install it. +## Features +- Extract elements by using xpath syntax +- Extract values from json format +- Extract values from yaml format +- Automaticaly detect format (json/yaml) +- Read data from a web resource +- Read data from file or pass data from stdin +- Format output values +- Format output to be reused by shell `eval` +- Convert YAML to JSON +- Convert JSON to YAML +## Install or Update niet +```sh +$ pip install -U niet +``` +## Requirements +- Python 3.6 or higher +## Supported versions +Since niet 2.0 the support of python 2.7 have been dropped so if +if you only have python 2.7 at hands then you can use previous version (lower +to 2.0) but you should consider first that the no support will be given on +these versions (no bugfix, no new feature, etc). If you report an issue or +or propose a new feature then they will be addressed only for current or +higher version. +## Usage +### Help and options +```shell +$ niet --help +usage: niet [-h] [-f {json,yaml,eval,newline,ifs,squote,dquote,comma}] [-s] [-v] + object [file] +Read data from YAML or JSON file +positional arguments: + object Path to object separated by dot (.). Use '.' to get + whole file. eg: a.b.c + file Optional JSON or YAML filename. If not provided niet + read from stdin +optional arguments: + -h, --help show this help message and exit + -f {json,yaml,eval,newline,ifs,squote,dquote,comma}, --format {json,yaml,eval,newline,ifs,squote,dquote,comma} + output format + -i, --in-place Perform modification in place. Will so alter read file + -o OUTPUT_FILE, --output OUTPUT_FILE + Print output in a file instead of stdout (surcharged + by infile parameter if set) + -s, --silent silent mode, doesn't display message when element was + not found + -v, --version print the Niet version number and exit (also + --version) + --debug Activate the debug mode (based on pdb) +output formats: + json Return object in JSON + yaml Return object in YAML + eval Return result in a string evaluable by a shell eval command as an input + newline Return all elements of a list in a new line + ifs Return all elements of a list separated by IFS env var + squote Add single quotes to result + dquote Add double quotes to result + comma Return all elements separated by commas +``` +### With Json from stdin +```shell +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}' | niet fizz.buzz +1 +2 +Fizz +4 +Buzz +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}' | niet fizz.buzz -f squote +'1' '2''Fizz' '4' 'Buzz' +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "fizz", "4", "buzz"]}}' | niet . -f yaml +fizz: + buzz: + - '1' + - '2' + - fizz + - '4' + - buzz +foo: bar +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet "fizz.buzz[2]" +two +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote "fizz.buzz[0:2]" +"zero" "one" +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote "fizz.buzz[:3]" +"zero" "one" "two" +``` +### With YAML file +Consider the yaml file with the following content: +```yaml +# /path/to/your/file.yaml +project: + meta: + name: my-project + foo: bar + list: + - item1 + - item2 + - item3 + test-dash: value +``` +You can [download the previous example](https://gist.githubusercontent.com/4383/53e1599663b369f499aa28e27009f2cd/raw/389b82c19499b8cb84a464784e9c79aa25d3a9d3/file.yaml) locally for testing purpose or use the command line for this: +```shell +wget https://gist.githubusercontent.com/4383/53e1599663b369f499aa28e27009f2cd/raw/389b82c19499b8cb84a464784e9c79aa25d3a9d3/file.yaml +``` +You can retrieve data from this file by using niet like this: +```sh +$ niet ".project.meta.name" /path/to/your/file.yaml +my-project +$ niet ".project.foo" /path/to/your/file.yaml +bar +$ niet ".project.list" /path/to/your/file.yaml +item1 item2 item3 +$ # assign return value to shell variable +$ NAME=$(niet ".project.meta.name" /path/to/your/file.yaml) +$ echo $NAME +my-project +$ niet project.'"test-dash"' /path/to/your/file.json +value +``` +### With JSON file +Consider the json file with the following content: +```json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +You can [download the previous example](https://gist.githubusercontent.com/4383/1bab8973474625de738f5f6471894322/raw/0048cd2310df2d98bf4f230ffe20da8fa615cef3/file.json) locally for testing purpose or use the command line for this: +```shell +wget https://gist.githubusercontent.com/4383/1bab8973474625de738f5f6471894322/raw/0048cd2310df2d98bf4f230ffe20da8fa615cef3/file.json +``` +You can retrieve data from this file by using niet like this: +```sh +$ niet "project.meta.name" /path/to/your/file.json +my-project +$ niet "project.foo" /path/to/your/file.json +bar +$ niet "project.list" /path/to/your/file.json +item1 item2 item3 +$ # assign return value to shell variable +$ NAME=$(niet "project.meta.name" /path/to/your/file.json) +$ echo $NAME +my-project +$ niet project.'"test-dash"' /path/to/your/file.json +value +``` +### Object Identifiers +An identifier is the most basic expression and can be used to extract a single +element from a JSON/YAML document. The return value for an identifier is +the value associated with the identifier. If the identifier does not +exist in the JSON/YAML document, than niet display a specific message and +return the error code `1`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo +Element not found: fizz.gogo +$ echo $? +1 +``` +See the [related section](#deal-with-errors) for more info on how to manage +errors with `niet`. +Niet is based on `jmespath` to find results so for complex research you can +refer to the [jmespath specifications](http://jmespath.org/specification.html#identifiers) +to use identifiers properly. +If you try to search for an identifier who use some dash you need to surround +your research expression with simple and double quotes, examples: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote '"foo-biz"' +bar +$ echo '{"key-test": "value"}' | niet '"key-test"' +value +``` +However, `niet` will detect related issues and surround automatically your +identifier if `jmespath` fail to handle it. +Hence, the following examples will return similar results than the previous +examples: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote foo-biz +bar +$ echo '{"key-test": "value"}' | niet key-test +value +``` +If your object is not at the root of your path, an example is available in +`tests/sample/sample.json`, then you need to only surround the researched +identifier like this `project.'"test-dash"'` +```json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +Example: +```sh +niet project.'"test-dash"' tests/sample/sample.json +``` +Further examples with [`jmespath` identifiers](http://jmespath.org/specification.html#examples). +### Output +#### Stdout +By default, niet print the output on stdout. +#### Save output to a file +It if possible to pass a filename using -o or --output argument to writes +directly in a file. This file will be created if not exists or will be +replaced if already exists. +#### In-file modification +It is possible to modify directly a file using -i or --in-place argument. This will replace +the input file by the output of niet command. This can be used to extract some data of a file or +reindent a file. +### Output formats +You can change the output format using the -f or --format optional +argument. +By default, niet detect the input format and display complex objects +in the same format. If the object is a list or a value, newline output +format will be used. +Output formats are: + - ifs + - squote + - dquote + - newline + - yaml + - json +#### ifs +Ifs output format print all values of a list or a single value in one line. +All values are separated by the content of IFS environment variable if defined, +space otherwise. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ IFS="|" niet .project.list /path/to/your/file.yaml -f ifs +item1|item2|item3 +$ IFS=" " niet .project.list /path/to/your/file.yaml -f ifs +item1 item2 item3 +$ IFS="@" niet .project.list /path/to/your/file.yaml -f ifs +item1@item2@item3 +``` +This is usefull in a shell for loop, +but your content must, of course, don't contain IFS value: +```shell +OIFS="$IFS" +IFS="|" +for i in $(niet .project.list /path/to/your/file.yaml -f ifs); do + echo ${i} +done +IFS="${OIFS}" +``` +Previous example provide the following output: +```sh +item1 +item2 +item3 +``` +For single quoted see [squote](#squote) ouput or [dquote](#dquote) double quoted output with IFS +#### squote +Squotes output format print all values of a list or a single value in one line. +All values are quoted with single quotes and are separated by IFS value. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ # With the default IFS +$ niet .project.list /path/to/your/file.yaml -f squote +'item1' 'item2' 'item3' +$ # With a specified IFS +$ IFS="|" niet .project.list /path/to/your/file.yaml -f squote +'item1'|'item2'|'item3' +``` +#### dquote +Dquotes output format print all values of a list or a single value in one line. +All values are quoted with a double quotes and are separated by IFS value. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ # With the default IFS +$ niet .project.list /path/to/your/file.yaml -f dquote +'item1' 'item2' 'item3' +$ # With a specified IFS +$ IFS="|" niet .project.list /path/to/your/file.yaml -f dquote +"item1"|"item2"|"item3" +``` +#### newline +`newline` output format print one value of a list or a single value per line. +The `newline` format is mostly usefull with shell while read loops and +with script interactions. +Example: +```sh +while read value: do + echo $value +done < $(niet --format newline project.list your-file.json) +``` +#### comma +`comma` output format print results on the same line and separated by commas. +The `comma` format allow you to format your outputs to consume your results +with other commands lines interfaces. By example some argument parser +allow you to pass multi values for the same parameter (the +[beagle command](https://beagle-hound.readthedocs.io/en/latest/) per +example allow you to +[repeat the `--repo` option](https://beagle-hound.readthedocs.io/en/latest/cli/index.html#cmdoption-beagle-search-repo)). +Example of integration with beagle and shell: +```sh +$ OSLO_PROJECTS_URL=https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +$ beagle search \ + -f link \ + --repo $(niet "oslo.deliverables.*.repos[0]" ${OSLO_PROJECTS_URL} -f comma) 'venv' +``` +The previous command will return all the links of files +who contains `venv` on the openstack oslo's scope of projects (pbr, +taskflow, oslo.messaging, etc). +Else another with a more reduced scope on openstack oslo's projects: +```sh +$ niet "oslo.deliverables.*.repos[0][?contains(@, \`oslo\`) == \`true\`]" \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml \ + -f comma +openstack/oslo-cookiecutter,openstack/oslo-specs,openstack/oslo.cache, +openstack/oslo.concurrency,openstack/oslo.config,openstack/oslo.context, +openstack/oslo.db,openstack/oslo.i18n,openstack/oslo.limit,openstack/oslo.log, +openstack/oslo.messaging,openstack/oslo.middleware, +openstack/oslo.policy,openstack/oslo.privsep,openstack/oslo.reports, +openstack/oslo.rootwrap,openstack/oslo.serialization,openstack/oslo.service, +openstack/oslo.tools,openstack/oslo.upgradecheck,openstack/oslo.utils, +openstack/oslo.versionedobjects,openstack/oslo.vmware,openstack/oslotest +``` +In the previous example we retrieve only the projects repos who contains +`oslo` in their names, so other projects like `taskflow`, `pbr`, etc will +be ignored. +#### eval +Eval output format allow you to eval output string to initialize shell +variable generated from your JSON/YAML content. +You can intialize shell variables from your entire content, example: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval . + foo_biz="bar";fizz__buzz=( zero one two three ) +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval .) +$ echo ${foo_biz} +bar +$ echo ${fizz__buzz} +zero one two three +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval '"foo-biz"'); echo ${foo_biz} +bar +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval fizz.buzz +fizz_buzz=( zero one two three ); +``` +Parent elements are separated by `__` by example the `fizz.buzz` element +will be represented by a variable named `fizz__buzz`. You need to consider +that when you call your expected variables. +Also you can initialize some shell array from your content and loop over in +a shell maner: +```sh +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval fizz.buzz) +$ for el in ${fizz_buzz}; do echo $el; done +zero +one +two +three +``` +#### yaml +Yaml output format force output to be in YAML regardless the input file format. +#### json +Json output format force output to be in JSON regardless the input file format. +### Read data from a web resource +Niet allow you to read data (json/yaml) from a web resource accessible by using +the HTTP protocole (introduced in niet 2.1). +This can be done by passing an url to niet which refer to a raw content (json +or yaml). +Here is some examples with the [openstack governance's projects data](https://github.com/openstack/governance/blob/master/reference/projects.yaml): +```sh +$ # List all the oslo projects repos (https://wiki.openstack.org/wiki/Oslo) +$ niet "oslo.deliverables.*.repos[0]" \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +openstack/automaton +openstack/castellan +openstack/debtcollector +openstack/futurist +openstack/oslo.cache +openstack/oslo.concurrency +openstack/oslo.config +openstack/oslo.context +openstack/oslo.db +openstack/oslo.i18n +openstack/oslo.limit +openstack/oslo.log +openstack/oslo.messaging +openstack/oslo.middleware +openstack/oslo.policy +openstack/oslo.service +openstack/osprofiler +openstack/pbr +openstack/stevedore +openstack/taskflow +openstack/tooz +openstack/whereto +$ niet oslo.service \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +Common libraries +$ # Get the openstack oslo's mission +$ niet oslo.mission \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +To produce a set of python libraries containing code shared by OpenStack projects. +The APIs provided by these libraries should be high quality, stable, consistent, +documented and generally applicable. +$ eval $(niet oslo.service \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f eval) && \ + test "${oslo_service}" = "Common libraries" +$ # Get the name of the oslo PTL +$ eval $(niet oslo.ptl.name \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f eval) +$ echo "${oslo_ptl_name}" # now display your evaluated result +$ # Convert original distant yaml file into json +$ niet . https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f json +``` +For further examples of filters and selections please take a look to +[the jmespath's doc](https://jmespath.org/examples.html). +### Result not found +By default when no results was found niet display a specific message and return +the error code `1`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo +Element not found: fizz.gogo +$ echo $? +1 +``` +You can avoid this behavior by passing niet into a silent mode. +Silent mode allow you to hide the specific message error but continue to return +a status code equal to `1` when the key was not found. +You can use the silent mode by using the flag `-s/--silent`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo -s +$ echo $? +1 +``` +### Deal with errors +When your JSON file content are not valid niet display an error and exit +with return code `1` +You can easily protect your script like this: +```sh +PROJECT_NAME=$(niet project.meta.name your-file.yaml) +if [ "$?" = "1" ]; then + echo "Error occur ${PROJECT_NAME}" +else + echo "Project name: ${PROJECT_NAME}" +fi +``` +## Examples +You can try niet by using the samples provided with the project sources code. +> All the following examples use the sample file available in niet sources code +at the following location `tests/samples/sample.yaml`. +Sample example: +```yaml +# tests/samples/sample.yaml +project: + meta: + name: my-project + foo: bar + list: + - item1 + - item2 + - item3 +``` +### Extract a single value +Retrieve the project name: +```sh +$ niet project.meta.name tests/samples/sample.yaml +my-project +``` +### Complexe search +Consider the following content: +``` +$ cat /var/lib/libvirt/dnsmasq/virbr0.status +[ + { + "ip-address": "192.168.122.113", + "mac-address": "52:54:00:91:14:02", + "hostname": "rhel79", + "expiry-time": 1644251254 + }, + { + "ip-address": "192.168.122.162", + "mac-address": "52:54:00:23:37:ed", + "hostname": "satellite", + "expiry-time": 1644251837 + } +] +``` +Here we want to retrieve the value of the ip-address field when the hostname +is equal to `satellite`. The following command will allow you to get this +value: +```sh +$ sed 's/ip/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='satellite'].ip" +192.168.122.162 +``` +You should notice that first we replace `-` by `_` by using the sed +command. We do that because `jmespath`, the underlying library used by `niet` +, poorly handle key that contain `-`. We chosen to replace all - by _ to avoid +any issues elsewhere on the file +Here is an exemple of an automated ssh connection in a kvm virtualised lab +environment by looking for vmname in dhcp file with `niet` and performing the +ssh connection to the server even if its ip changed. +The ssh connection here can be performed with this command: +```sh +ssh -o ProxyCommand='nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='''%h'''].ip_address") %p' root@rhel79 +``` +Tips - to ease that use you can for example set this `.ssh/config` entry: +``` +host lab-* +user root +ProxyCommand /usr/bin/nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='$(echo %h | cut -d'-' -f2 )'].ip_address") %p +``` +And then perform a `ssh lab-rhel79` or a `ssh lab-satellite` to join all VMs +from your lab, by the hostname prefixed by `lab-`. +### Extract a list and parse it in shell +Deal with list of items +```sh +$ for el in $(niet project.list tests/samples/sample.yaml); do echo ${el}; done +item1 +item2 +item3 +``` +Also you can `eval` your `niet` output to setput some shell variables +that you can reuse in your shell scripts, the following example is similar to +the previous example but make use of the eval ouput format (`-f eval`): +```sh +$ eval $(niet -f eval project.list tests/samples/sample.yaml) +$ for el in ${project__list}; do echo $el; done +zero +one +two +three +``` +### Extract a complex object and parse it in shell +Extract the object as JSON to store it in shell variable : +```shell +$ project="$(niet -f json .project tests/samples/sample.yaml)" +``` +Then parse it after in bash in this example: +```shell +$ niet .meta.name <<< $project +my-project +``` +### Transform JSON to YAML +With niet you can easily convert your JSON to YAML +```shell +$ niet . tests/samples/sample.json -f yaml +project: + foo: bar + list: + - item1 + - item2 + - item3 + meta: + name: my-project +``` +### Transform YAML to JSON +With niet you can easily convert your YAML to JSON +```shell +$ niet . tests/samples/sample.yaml -f json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ] + } +} +``` +### Indent JSON file +This is an example of how to indent a JSON file : +```shell +$ niet . tests/samples/sample_not_indented.json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +## Tips +You can pass your search with or without quotes like this: +```sh +$ niet project.meta.name your-file.yaml +$ niet "project.meta.name" your-file.yaml +``` +You can execute `niet` step by step by using the debug mode. It will allow +you to inspect your execution during your debug sessions. +## Contribute +If you want to contribute to niet [please first read the contribution guidelines](CONTRIBUTING.md) +## Licence +This project is under the MIT License. +[See the license file for more details](LICENSE) +Henry Tang <henryykt@gmail.com> +Hervé Beraud <hberaud@redhat.com> +Hervé Beraud <herveberaud.pro@gmail.com> +Sebastien Boyron <sebastien@boyron.eu> +Sébastien Boyron <sboyron@localhost.localdomain> +dj4ngo <sebastien@boyron.eu> +rechthemoon <rechthemoon@gmail.com> + +%package -n python3-niet +Summary: Shell config file parser (json, yaml) +Provides: python-niet +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-niet +> How to easily parse and retrieve data from YAML file in our shell? +The previous question, few years ago, led us to the development of niet. +Indeed, at that time, we needed a way to store and retrieve data for our own +needs. We created niet to read those data. The goal was to develop a tools +that will allow us to standardize how we parse YAML locally or in our CI +pipelines. We wanted something reusable and easily distribuable. Niet was born. +Niet is like [xmllint](http://xmlsoft.org/xmllint.html) or +[jq](https://stedolan.github.io/jq/) but for YAML and JSON data - +you can use it to slice and filter and map and transform structured data. +You can easily retrieve data by using simple expressions or using +xpath advanced features to access non-trivial data. +You can easily convert YAML format into JSON format and vice versa. +Niet is writen in Python so you can install it from a package manager (from +PyPi) or directly by cloning this repository - no specific system rights are +needed to install it. +## Features +- Extract elements by using xpath syntax +- Extract values from json format +- Extract values from yaml format +- Automaticaly detect format (json/yaml) +- Read data from a web resource +- Read data from file or pass data from stdin +- Format output values +- Format output to be reused by shell `eval` +- Convert YAML to JSON +- Convert JSON to YAML +## Install or Update niet +```sh +$ pip install -U niet +``` +## Requirements +- Python 3.6 or higher +## Supported versions +Since niet 2.0 the support of python 2.7 have been dropped so if +if you only have python 2.7 at hands then you can use previous version (lower +to 2.0) but you should consider first that the no support will be given on +these versions (no bugfix, no new feature, etc). If you report an issue or +or propose a new feature then they will be addressed only for current or +higher version. +## Usage +### Help and options +```shell +$ niet --help +usage: niet [-h] [-f {json,yaml,eval,newline,ifs,squote,dquote,comma}] [-s] [-v] + object [file] +Read data from YAML or JSON file +positional arguments: + object Path to object separated by dot (.). Use '.' to get + whole file. eg: a.b.c + file Optional JSON or YAML filename. If not provided niet + read from stdin +optional arguments: + -h, --help show this help message and exit + -f {json,yaml,eval,newline,ifs,squote,dquote,comma}, --format {json,yaml,eval,newline,ifs,squote,dquote,comma} + output format + -i, --in-place Perform modification in place. Will so alter read file + -o OUTPUT_FILE, --output OUTPUT_FILE + Print output in a file instead of stdout (surcharged + by infile parameter if set) + -s, --silent silent mode, doesn't display message when element was + not found + -v, --version print the Niet version number and exit (also + --version) + --debug Activate the debug mode (based on pdb) +output formats: + json Return object in JSON + yaml Return object in YAML + eval Return result in a string evaluable by a shell eval command as an input + newline Return all elements of a list in a new line + ifs Return all elements of a list separated by IFS env var + squote Add single quotes to result + dquote Add double quotes to result + comma Return all elements separated by commas +``` +### With Json from stdin +```shell +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}' | niet fizz.buzz +1 +2 +Fizz +4 +Buzz +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}' | niet fizz.buzz -f squote +'1' '2''Fizz' '4' 'Buzz' +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "fizz", "4", "buzz"]}}' | niet . -f yaml +fizz: + buzz: + - '1' + - '2' + - fizz + - '4' + - buzz +foo: bar +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet "fizz.buzz[2]" +two +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote "fizz.buzz[0:2]" +"zero" "one" +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote "fizz.buzz[:3]" +"zero" "one" "two" +``` +### With YAML file +Consider the yaml file with the following content: +```yaml +# /path/to/your/file.yaml +project: + meta: + name: my-project + foo: bar + list: + - item1 + - item2 + - item3 + test-dash: value +``` +You can [download the previous example](https://gist.githubusercontent.com/4383/53e1599663b369f499aa28e27009f2cd/raw/389b82c19499b8cb84a464784e9c79aa25d3a9d3/file.yaml) locally for testing purpose or use the command line for this: +```shell +wget https://gist.githubusercontent.com/4383/53e1599663b369f499aa28e27009f2cd/raw/389b82c19499b8cb84a464784e9c79aa25d3a9d3/file.yaml +``` +You can retrieve data from this file by using niet like this: +```sh +$ niet ".project.meta.name" /path/to/your/file.yaml +my-project +$ niet ".project.foo" /path/to/your/file.yaml +bar +$ niet ".project.list" /path/to/your/file.yaml +item1 item2 item3 +$ # assign return value to shell variable +$ NAME=$(niet ".project.meta.name" /path/to/your/file.yaml) +$ echo $NAME +my-project +$ niet project.'"test-dash"' /path/to/your/file.json +value +``` +### With JSON file +Consider the json file with the following content: +```json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +You can [download the previous example](https://gist.githubusercontent.com/4383/1bab8973474625de738f5f6471894322/raw/0048cd2310df2d98bf4f230ffe20da8fa615cef3/file.json) locally for testing purpose or use the command line for this: +```shell +wget https://gist.githubusercontent.com/4383/1bab8973474625de738f5f6471894322/raw/0048cd2310df2d98bf4f230ffe20da8fa615cef3/file.json +``` +You can retrieve data from this file by using niet like this: +```sh +$ niet "project.meta.name" /path/to/your/file.json +my-project +$ niet "project.foo" /path/to/your/file.json +bar +$ niet "project.list" /path/to/your/file.json +item1 item2 item3 +$ # assign return value to shell variable +$ NAME=$(niet "project.meta.name" /path/to/your/file.json) +$ echo $NAME +my-project +$ niet project.'"test-dash"' /path/to/your/file.json +value +``` +### Object Identifiers +An identifier is the most basic expression and can be used to extract a single +element from a JSON/YAML document. The return value for an identifier is +the value associated with the identifier. If the identifier does not +exist in the JSON/YAML document, than niet display a specific message and +return the error code `1`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo +Element not found: fizz.gogo +$ echo $? +1 +``` +See the [related section](#deal-with-errors) for more info on how to manage +errors with `niet`. +Niet is based on `jmespath` to find results so for complex research you can +refer to the [jmespath specifications](http://jmespath.org/specification.html#identifiers) +to use identifiers properly. +If you try to search for an identifier who use some dash you need to surround +your research expression with simple and double quotes, examples: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote '"foo-biz"' +bar +$ echo '{"key-test": "value"}' | niet '"key-test"' +value +``` +However, `niet` will detect related issues and surround automatically your +identifier if `jmespath` fail to handle it. +Hence, the following examples will return similar results than the previous +examples: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote foo-biz +bar +$ echo '{"key-test": "value"}' | niet key-test +value +``` +If your object is not at the root of your path, an example is available in +`tests/sample/sample.json`, then you need to only surround the researched +identifier like this `project.'"test-dash"'` +```json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +Example: +```sh +niet project.'"test-dash"' tests/sample/sample.json +``` +Further examples with [`jmespath` identifiers](http://jmespath.org/specification.html#examples). +### Output +#### Stdout +By default, niet print the output on stdout. +#### Save output to a file +It if possible to pass a filename using -o or --output argument to writes +directly in a file. This file will be created if not exists or will be +replaced if already exists. +#### In-file modification +It is possible to modify directly a file using -i or --in-place argument. This will replace +the input file by the output of niet command. This can be used to extract some data of a file or +reindent a file. +### Output formats +You can change the output format using the -f or --format optional +argument. +By default, niet detect the input format and display complex objects +in the same format. If the object is a list or a value, newline output +format will be used. +Output formats are: + - ifs + - squote + - dquote + - newline + - yaml + - json +#### ifs +Ifs output format print all values of a list or a single value in one line. +All values are separated by the content of IFS environment variable if defined, +space otherwise. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ IFS="|" niet .project.list /path/to/your/file.yaml -f ifs +item1|item2|item3 +$ IFS=" " niet .project.list /path/to/your/file.yaml -f ifs +item1 item2 item3 +$ IFS="@" niet .project.list /path/to/your/file.yaml -f ifs +item1@item2@item3 +``` +This is usefull in a shell for loop, +but your content must, of course, don't contain IFS value: +```shell +OIFS="$IFS" +IFS="|" +for i in $(niet .project.list /path/to/your/file.yaml -f ifs); do + echo ${i} +done +IFS="${OIFS}" +``` +Previous example provide the following output: +```sh +item1 +item2 +item3 +``` +For single quoted see [squote](#squote) ouput or [dquote](#dquote) double quoted output with IFS +#### squote +Squotes output format print all values of a list or a single value in one line. +All values are quoted with single quotes and are separated by IFS value. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ # With the default IFS +$ niet .project.list /path/to/your/file.yaml -f squote +'item1' 'item2' 'item3' +$ # With a specified IFS +$ IFS="|" niet .project.list /path/to/your/file.yaml -f squote +'item1'|'item2'|'item3' +``` +#### dquote +Dquotes output format print all values of a list or a single value in one line. +All values are quoted with a double quotes and are separated by IFS value. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ # With the default IFS +$ niet .project.list /path/to/your/file.yaml -f dquote +'item1' 'item2' 'item3' +$ # With a specified IFS +$ IFS="|" niet .project.list /path/to/your/file.yaml -f dquote +"item1"|"item2"|"item3" +``` +#### newline +`newline` output format print one value of a list or a single value per line. +The `newline` format is mostly usefull with shell while read loops and +with script interactions. +Example: +```sh +while read value: do + echo $value +done < $(niet --format newline project.list your-file.json) +``` +#### comma +`comma` output format print results on the same line and separated by commas. +The `comma` format allow you to format your outputs to consume your results +with other commands lines interfaces. By example some argument parser +allow you to pass multi values for the same parameter (the +[beagle command](https://beagle-hound.readthedocs.io/en/latest/) per +example allow you to +[repeat the `--repo` option](https://beagle-hound.readthedocs.io/en/latest/cli/index.html#cmdoption-beagle-search-repo)). +Example of integration with beagle and shell: +```sh +$ OSLO_PROJECTS_URL=https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +$ beagle search \ + -f link \ + --repo $(niet "oslo.deliverables.*.repos[0]" ${OSLO_PROJECTS_URL} -f comma) 'venv' +``` +The previous command will return all the links of files +who contains `venv` on the openstack oslo's scope of projects (pbr, +taskflow, oslo.messaging, etc). +Else another with a more reduced scope on openstack oslo's projects: +```sh +$ niet "oslo.deliverables.*.repos[0][?contains(@, \`oslo\`) == \`true\`]" \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml \ + -f comma +openstack/oslo-cookiecutter,openstack/oslo-specs,openstack/oslo.cache, +openstack/oslo.concurrency,openstack/oslo.config,openstack/oslo.context, +openstack/oslo.db,openstack/oslo.i18n,openstack/oslo.limit,openstack/oslo.log, +openstack/oslo.messaging,openstack/oslo.middleware, +openstack/oslo.policy,openstack/oslo.privsep,openstack/oslo.reports, +openstack/oslo.rootwrap,openstack/oslo.serialization,openstack/oslo.service, +openstack/oslo.tools,openstack/oslo.upgradecheck,openstack/oslo.utils, +openstack/oslo.versionedobjects,openstack/oslo.vmware,openstack/oslotest +``` +In the previous example we retrieve only the projects repos who contains +`oslo` in their names, so other projects like `taskflow`, `pbr`, etc will +be ignored. +#### eval +Eval output format allow you to eval output string to initialize shell +variable generated from your JSON/YAML content. +You can intialize shell variables from your entire content, example: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval . + foo_biz="bar";fizz__buzz=( zero one two three ) +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval .) +$ echo ${foo_biz} +bar +$ echo ${fizz__buzz} +zero one two three +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval '"foo-biz"'); echo ${foo_biz} +bar +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval fizz.buzz +fizz_buzz=( zero one two three ); +``` +Parent elements are separated by `__` by example the `fizz.buzz` element +will be represented by a variable named `fizz__buzz`. You need to consider +that when you call your expected variables. +Also you can initialize some shell array from your content and loop over in +a shell maner: +```sh +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval fizz.buzz) +$ for el in ${fizz_buzz}; do echo $el; done +zero +one +two +three +``` +#### yaml +Yaml output format force output to be in YAML regardless the input file format. +#### json +Json output format force output to be in JSON regardless the input file format. +### Read data from a web resource +Niet allow you to read data (json/yaml) from a web resource accessible by using +the HTTP protocole (introduced in niet 2.1). +This can be done by passing an url to niet which refer to a raw content (json +or yaml). +Here is some examples with the [openstack governance's projects data](https://github.com/openstack/governance/blob/master/reference/projects.yaml): +```sh +$ # List all the oslo projects repos (https://wiki.openstack.org/wiki/Oslo) +$ niet "oslo.deliverables.*.repos[0]" \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +openstack/automaton +openstack/castellan +openstack/debtcollector +openstack/futurist +openstack/oslo.cache +openstack/oslo.concurrency +openstack/oslo.config +openstack/oslo.context +openstack/oslo.db +openstack/oslo.i18n +openstack/oslo.limit +openstack/oslo.log +openstack/oslo.messaging +openstack/oslo.middleware +openstack/oslo.policy +openstack/oslo.service +openstack/osprofiler +openstack/pbr +openstack/stevedore +openstack/taskflow +openstack/tooz +openstack/whereto +$ niet oslo.service \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +Common libraries +$ # Get the openstack oslo's mission +$ niet oslo.mission \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +To produce a set of python libraries containing code shared by OpenStack projects. +The APIs provided by these libraries should be high quality, stable, consistent, +documented and generally applicable. +$ eval $(niet oslo.service \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f eval) && \ + test "${oslo_service}" = "Common libraries" +$ # Get the name of the oslo PTL +$ eval $(niet oslo.ptl.name \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f eval) +$ echo "${oslo_ptl_name}" # now display your evaluated result +$ # Convert original distant yaml file into json +$ niet . https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f json +``` +For further examples of filters and selections please take a look to +[the jmespath's doc](https://jmespath.org/examples.html). +### Result not found +By default when no results was found niet display a specific message and return +the error code `1`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo +Element not found: fizz.gogo +$ echo $? +1 +``` +You can avoid this behavior by passing niet into a silent mode. +Silent mode allow you to hide the specific message error but continue to return +a status code equal to `1` when the key was not found. +You can use the silent mode by using the flag `-s/--silent`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo -s +$ echo $? +1 +``` +### Deal with errors +When your JSON file content are not valid niet display an error and exit +with return code `1` +You can easily protect your script like this: +```sh +PROJECT_NAME=$(niet project.meta.name your-file.yaml) +if [ "$?" = "1" ]; then + echo "Error occur ${PROJECT_NAME}" +else + echo "Project name: ${PROJECT_NAME}" +fi +``` +## Examples +You can try niet by using the samples provided with the project sources code. +> All the following examples use the sample file available in niet sources code +at the following location `tests/samples/sample.yaml`. +Sample example: +```yaml +# tests/samples/sample.yaml +project: + meta: + name: my-project + foo: bar + list: + - item1 + - item2 + - item3 +``` +### Extract a single value +Retrieve the project name: +```sh +$ niet project.meta.name tests/samples/sample.yaml +my-project +``` +### Complexe search +Consider the following content: +``` +$ cat /var/lib/libvirt/dnsmasq/virbr0.status +[ + { + "ip-address": "192.168.122.113", + "mac-address": "52:54:00:91:14:02", + "hostname": "rhel79", + "expiry-time": 1644251254 + }, + { + "ip-address": "192.168.122.162", + "mac-address": "52:54:00:23:37:ed", + "hostname": "satellite", + "expiry-time": 1644251837 + } +] +``` +Here we want to retrieve the value of the ip-address field when the hostname +is equal to `satellite`. The following command will allow you to get this +value: +```sh +$ sed 's/ip/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='satellite'].ip" +192.168.122.162 +``` +You should notice that first we replace `-` by `_` by using the sed +command. We do that because `jmespath`, the underlying library used by `niet` +, poorly handle key that contain `-`. We chosen to replace all - by _ to avoid +any issues elsewhere on the file +Here is an exemple of an automated ssh connection in a kvm virtualised lab +environment by looking for vmname in dhcp file with `niet` and performing the +ssh connection to the server even if its ip changed. +The ssh connection here can be performed with this command: +```sh +ssh -o ProxyCommand='nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='''%h'''].ip_address") %p' root@rhel79 +``` +Tips - to ease that use you can for example set this `.ssh/config` entry: +``` +host lab-* +user root +ProxyCommand /usr/bin/nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='$(echo %h | cut -d'-' -f2 )'].ip_address") %p +``` +And then perform a `ssh lab-rhel79` or a `ssh lab-satellite` to join all VMs +from your lab, by the hostname prefixed by `lab-`. +### Extract a list and parse it in shell +Deal with list of items +```sh +$ for el in $(niet project.list tests/samples/sample.yaml); do echo ${el}; done +item1 +item2 +item3 +``` +Also you can `eval` your `niet` output to setput some shell variables +that you can reuse in your shell scripts, the following example is similar to +the previous example but make use of the eval ouput format (`-f eval`): +```sh +$ eval $(niet -f eval project.list tests/samples/sample.yaml) +$ for el in ${project__list}; do echo $el; done +zero +one +two +three +``` +### Extract a complex object and parse it in shell +Extract the object as JSON to store it in shell variable : +```shell +$ project="$(niet -f json .project tests/samples/sample.yaml)" +``` +Then parse it after in bash in this example: +```shell +$ niet .meta.name <<< $project +my-project +``` +### Transform JSON to YAML +With niet you can easily convert your JSON to YAML +```shell +$ niet . tests/samples/sample.json -f yaml +project: + foo: bar + list: + - item1 + - item2 + - item3 + meta: + name: my-project +``` +### Transform YAML to JSON +With niet you can easily convert your YAML to JSON +```shell +$ niet . tests/samples/sample.yaml -f json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ] + } +} +``` +### Indent JSON file +This is an example of how to indent a JSON file : +```shell +$ niet . tests/samples/sample_not_indented.json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +## Tips +You can pass your search with or without quotes like this: +```sh +$ niet project.meta.name your-file.yaml +$ niet "project.meta.name" your-file.yaml +``` +You can execute `niet` step by step by using the debug mode. It will allow +you to inspect your execution during your debug sessions. +## Contribute +If you want to contribute to niet [please first read the contribution guidelines](CONTRIBUTING.md) +## Licence +This project is under the MIT License. +[See the license file for more details](LICENSE) +Henry Tang <henryykt@gmail.com> +Hervé Beraud <hberaud@redhat.com> +Hervé Beraud <herveberaud.pro@gmail.com> +Sebastien Boyron <sebastien@boyron.eu> +Sébastien Boyron <sboyron@localhost.localdomain> +dj4ngo <sebastien@boyron.eu> +rechthemoon <rechthemoon@gmail.com> + +%package help +Summary: Development documents and examples for niet +Provides: python3-niet-doc +%description help +> How to easily parse and retrieve data from YAML file in our shell? +The previous question, few years ago, led us to the development of niet. +Indeed, at that time, we needed a way to store and retrieve data for our own +needs. We created niet to read those data. The goal was to develop a tools +that will allow us to standardize how we parse YAML locally or in our CI +pipelines. We wanted something reusable and easily distribuable. Niet was born. +Niet is like [xmllint](http://xmlsoft.org/xmllint.html) or +[jq](https://stedolan.github.io/jq/) but for YAML and JSON data - +you can use it to slice and filter and map and transform structured data. +You can easily retrieve data by using simple expressions or using +xpath advanced features to access non-trivial data. +You can easily convert YAML format into JSON format and vice versa. +Niet is writen in Python so you can install it from a package manager (from +PyPi) or directly by cloning this repository - no specific system rights are +needed to install it. +## Features +- Extract elements by using xpath syntax +- Extract values from json format +- Extract values from yaml format +- Automaticaly detect format (json/yaml) +- Read data from a web resource +- Read data from file or pass data from stdin +- Format output values +- Format output to be reused by shell `eval` +- Convert YAML to JSON +- Convert JSON to YAML +## Install or Update niet +```sh +$ pip install -U niet +``` +## Requirements +- Python 3.6 or higher +## Supported versions +Since niet 2.0 the support of python 2.7 have been dropped so if +if you only have python 2.7 at hands then you can use previous version (lower +to 2.0) but you should consider first that the no support will be given on +these versions (no bugfix, no new feature, etc). If you report an issue or +or propose a new feature then they will be addressed only for current or +higher version. +## Usage +### Help and options +```shell +$ niet --help +usage: niet [-h] [-f {json,yaml,eval,newline,ifs,squote,dquote,comma}] [-s] [-v] + object [file] +Read data from YAML or JSON file +positional arguments: + object Path to object separated by dot (.). Use '.' to get + whole file. eg: a.b.c + file Optional JSON or YAML filename. If not provided niet + read from stdin +optional arguments: + -h, --help show this help message and exit + -f {json,yaml,eval,newline,ifs,squote,dquote,comma}, --format {json,yaml,eval,newline,ifs,squote,dquote,comma} + output format + -i, --in-place Perform modification in place. Will so alter read file + -o OUTPUT_FILE, --output OUTPUT_FILE + Print output in a file instead of stdout (surcharged + by infile parameter if set) + -s, --silent silent mode, doesn't display message when element was + not found + -v, --version print the Niet version number and exit (also + --version) + --debug Activate the debug mode (based on pdb) +output formats: + json Return object in JSON + yaml Return object in YAML + eval Return result in a string evaluable by a shell eval command as an input + newline Return all elements of a list in a new line + ifs Return all elements of a list separated by IFS env var + squote Add single quotes to result + dquote Add double quotes to result + comma Return all elements separated by commas +``` +### With Json from stdin +```shell +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}' | niet fizz.buzz +1 +2 +Fizz +4 +Buzz +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "Fizz", "4", "Buzz"]}}' | niet fizz.buzz -f squote +'1' '2''Fizz' '4' 'Buzz' +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "fizz", "4", "buzz"]}}' | niet . -f yaml +fizz: + buzz: + - '1' + - '2' + - fizz + - '4' + - buzz +foo: bar +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet "fizz.buzz[2]" +two +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote "fizz.buzz[0:2]" +"zero" "one" +$ echo '{"foo": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote "fizz.buzz[:3]" +"zero" "one" "two" +``` +### With YAML file +Consider the yaml file with the following content: +```yaml +# /path/to/your/file.yaml +project: + meta: + name: my-project + foo: bar + list: + - item1 + - item2 + - item3 + test-dash: value +``` +You can [download the previous example](https://gist.githubusercontent.com/4383/53e1599663b369f499aa28e27009f2cd/raw/389b82c19499b8cb84a464784e9c79aa25d3a9d3/file.yaml) locally for testing purpose or use the command line for this: +```shell +wget https://gist.githubusercontent.com/4383/53e1599663b369f499aa28e27009f2cd/raw/389b82c19499b8cb84a464784e9c79aa25d3a9d3/file.yaml +``` +You can retrieve data from this file by using niet like this: +```sh +$ niet ".project.meta.name" /path/to/your/file.yaml +my-project +$ niet ".project.foo" /path/to/your/file.yaml +bar +$ niet ".project.list" /path/to/your/file.yaml +item1 item2 item3 +$ # assign return value to shell variable +$ NAME=$(niet ".project.meta.name" /path/to/your/file.yaml) +$ echo $NAME +my-project +$ niet project.'"test-dash"' /path/to/your/file.json +value +``` +### With JSON file +Consider the json file with the following content: +```json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +You can [download the previous example](https://gist.githubusercontent.com/4383/1bab8973474625de738f5f6471894322/raw/0048cd2310df2d98bf4f230ffe20da8fa615cef3/file.json) locally for testing purpose or use the command line for this: +```shell +wget https://gist.githubusercontent.com/4383/1bab8973474625de738f5f6471894322/raw/0048cd2310df2d98bf4f230ffe20da8fa615cef3/file.json +``` +You can retrieve data from this file by using niet like this: +```sh +$ niet "project.meta.name" /path/to/your/file.json +my-project +$ niet "project.foo" /path/to/your/file.json +bar +$ niet "project.list" /path/to/your/file.json +item1 item2 item3 +$ # assign return value to shell variable +$ NAME=$(niet "project.meta.name" /path/to/your/file.json) +$ echo $NAME +my-project +$ niet project.'"test-dash"' /path/to/your/file.json +value +``` +### Object Identifiers +An identifier is the most basic expression and can be used to extract a single +element from a JSON/YAML document. The return value for an identifier is +the value associated with the identifier. If the identifier does not +exist in the JSON/YAML document, than niet display a specific message and +return the error code `1`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo +Element not found: fizz.gogo +$ echo $? +1 +``` +See the [related section](#deal-with-errors) for more info on how to manage +errors with `niet`. +Niet is based on `jmespath` to find results so for complex research you can +refer to the [jmespath specifications](http://jmespath.org/specification.html#identifiers) +to use identifiers properly. +If you try to search for an identifier who use some dash you need to surround +your research expression with simple and double quotes, examples: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote '"foo-biz"' +bar +$ echo '{"key-test": "value"}' | niet '"key-test"' +value +``` +However, `niet` will detect related issues and surround automatically your +identifier if `jmespath` fail to handle it. +Hence, the following examples will return similar results than the previous +examples: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f dquote foo-biz +bar +$ echo '{"key-test": "value"}' | niet key-test +value +``` +If your object is not at the root of your path, an example is available in +`tests/sample/sample.json`, then you need to only surround the researched +identifier like this `project.'"test-dash"'` +```json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +Example: +```sh +niet project.'"test-dash"' tests/sample/sample.json +``` +Further examples with [`jmespath` identifiers](http://jmespath.org/specification.html#examples). +### Output +#### Stdout +By default, niet print the output on stdout. +#### Save output to a file +It if possible to pass a filename using -o or --output argument to writes +directly in a file. This file will be created if not exists or will be +replaced if already exists. +#### In-file modification +It is possible to modify directly a file using -i or --in-place argument. This will replace +the input file by the output of niet command. This can be used to extract some data of a file or +reindent a file. +### Output formats +You can change the output format using the -f or --format optional +argument. +By default, niet detect the input format and display complex objects +in the same format. If the object is a list or a value, newline output +format will be used. +Output formats are: + - ifs + - squote + - dquote + - newline + - yaml + - json +#### ifs +Ifs output format print all values of a list or a single value in one line. +All values are separated by the content of IFS environment variable if defined, +space otherwise. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ IFS="|" niet .project.list /path/to/your/file.yaml -f ifs +item1|item2|item3 +$ IFS=" " niet .project.list /path/to/your/file.yaml -f ifs +item1 item2 item3 +$ IFS="@" niet .project.list /path/to/your/file.yaml -f ifs +item1@item2@item3 +``` +This is usefull in a shell for loop, +but your content must, of course, don't contain IFS value: +```shell +OIFS="$IFS" +IFS="|" +for i in $(niet .project.list /path/to/your/file.yaml -f ifs); do + echo ${i} +done +IFS="${OIFS}" +``` +Previous example provide the following output: +```sh +item1 +item2 +item3 +``` +For single quoted see [squote](#squote) ouput or [dquote](#dquote) double quoted output with IFS +#### squote +Squotes output format print all values of a list or a single value in one line. +All values are quoted with single quotes and are separated by IFS value. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ # With the default IFS +$ niet .project.list /path/to/your/file.yaml -f squote +'item1' 'item2' 'item3' +$ # With a specified IFS +$ IFS="|" niet .project.list /path/to/your/file.yaml -f squote +'item1'|'item2'|'item3' +``` +#### dquote +Dquotes output format print all values of a list or a single value in one line. +All values are quoted with a double quotes and are separated by IFS value. +Examples (consider the previous [YAML file example](#with-yaml-file)): +```shell +$ # With the default IFS +$ niet .project.list /path/to/your/file.yaml -f dquote +'item1' 'item2' 'item3' +$ # With a specified IFS +$ IFS="|" niet .project.list /path/to/your/file.yaml -f dquote +"item1"|"item2"|"item3" +``` +#### newline +`newline` output format print one value of a list or a single value per line. +The `newline` format is mostly usefull with shell while read loops and +with script interactions. +Example: +```sh +while read value: do + echo $value +done < $(niet --format newline project.list your-file.json) +``` +#### comma +`comma` output format print results on the same line and separated by commas. +The `comma` format allow you to format your outputs to consume your results +with other commands lines interfaces. By example some argument parser +allow you to pass multi values for the same parameter (the +[beagle command](https://beagle-hound.readthedocs.io/en/latest/) per +example allow you to +[repeat the `--repo` option](https://beagle-hound.readthedocs.io/en/latest/cli/index.html#cmdoption-beagle-search-repo)). +Example of integration with beagle and shell: +```sh +$ OSLO_PROJECTS_URL=https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +$ beagle search \ + -f link \ + --repo $(niet "oslo.deliverables.*.repos[0]" ${OSLO_PROJECTS_URL} -f comma) 'venv' +``` +The previous command will return all the links of files +who contains `venv` on the openstack oslo's scope of projects (pbr, +taskflow, oslo.messaging, etc). +Else another with a more reduced scope on openstack oslo's projects: +```sh +$ niet "oslo.deliverables.*.repos[0][?contains(@, \`oslo\`) == \`true\`]" \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml \ + -f comma +openstack/oslo-cookiecutter,openstack/oslo-specs,openstack/oslo.cache, +openstack/oslo.concurrency,openstack/oslo.config,openstack/oslo.context, +openstack/oslo.db,openstack/oslo.i18n,openstack/oslo.limit,openstack/oslo.log, +openstack/oslo.messaging,openstack/oslo.middleware, +openstack/oslo.policy,openstack/oslo.privsep,openstack/oslo.reports, +openstack/oslo.rootwrap,openstack/oslo.serialization,openstack/oslo.service, +openstack/oslo.tools,openstack/oslo.upgradecheck,openstack/oslo.utils, +openstack/oslo.versionedobjects,openstack/oslo.vmware,openstack/oslotest +``` +In the previous example we retrieve only the projects repos who contains +`oslo` in their names, so other projects like `taskflow`, `pbr`, etc will +be ignored. +#### eval +Eval output format allow you to eval output string to initialize shell +variable generated from your JSON/YAML content. +You can intialize shell variables from your entire content, example: +```sh +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval . + foo_biz="bar";fizz__buzz=( zero one two three ) +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval .) +$ echo ${foo_biz} +bar +$ echo ${fizz__buzz} +zero one two three +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval '"foo-biz"'); echo ${foo_biz} +bar +$ echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval fizz.buzz +fizz_buzz=( zero one two three ); +``` +Parent elements are separated by `__` by example the `fizz.buzz` element +will be represented by a variable named `fizz__buzz`. You need to consider +that when you call your expected variables. +Also you can initialize some shell array from your content and loop over in +a shell maner: +```sh +$ eval $(echo '{"foo-biz": "bar", "fizz": {"buzz": ["zero", "one", "two", "three"]}}' | niet -f eval fizz.buzz) +$ for el in ${fizz_buzz}; do echo $el; done +zero +one +two +three +``` +#### yaml +Yaml output format force output to be in YAML regardless the input file format. +#### json +Json output format force output to be in JSON regardless the input file format. +### Read data from a web resource +Niet allow you to read data (json/yaml) from a web resource accessible by using +the HTTP protocole (introduced in niet 2.1). +This can be done by passing an url to niet which refer to a raw content (json +or yaml). +Here is some examples with the [openstack governance's projects data](https://github.com/openstack/governance/blob/master/reference/projects.yaml): +```sh +$ # List all the oslo projects repos (https://wiki.openstack.org/wiki/Oslo) +$ niet "oslo.deliverables.*.repos[0]" \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +openstack/automaton +openstack/castellan +openstack/debtcollector +openstack/futurist +openstack/oslo.cache +openstack/oslo.concurrency +openstack/oslo.config +openstack/oslo.context +openstack/oslo.db +openstack/oslo.i18n +openstack/oslo.limit +openstack/oslo.log +openstack/oslo.messaging +openstack/oslo.middleware +openstack/oslo.policy +openstack/oslo.service +openstack/osprofiler +openstack/pbr +openstack/stevedore +openstack/taskflow +openstack/tooz +openstack/whereto +$ niet oslo.service \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +Common libraries +$ # Get the openstack oslo's mission +$ niet oslo.mission \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml +To produce a set of python libraries containing code shared by OpenStack projects. +The APIs provided by these libraries should be high quality, stable, consistent, +documented and generally applicable. +$ eval $(niet oslo.service \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f eval) && \ + test "${oslo_service}" = "Common libraries" +$ # Get the name of the oslo PTL +$ eval $(niet oslo.ptl.name \ + https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f eval) +$ echo "${oslo_ptl_name}" # now display your evaluated result +$ # Convert original distant yaml file into json +$ niet . https://raw.githubusercontent.com/openstack/governance/master/reference/projects.yaml -f json +``` +For further examples of filters and selections please take a look to +[the jmespath's doc](https://jmespath.org/examples.html). +### Result not found +By default when no results was found niet display a specific message and return +the error code `1`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo +Element not found: fizz.gogo +$ echo $? +1 +``` +You can avoid this behavior by passing niet into a silent mode. +Silent mode allow you to hide the specific message error but continue to return +a status code equal to `1` when the key was not found. +You can use the silent mode by using the flag `-s/--silent`, example: +```sh +$ echo '{"foo": "bar", "fizz": {"buzz": ["1", "2", "3"]}}' | niet fizz.gogo -s +$ echo $? +1 +``` +### Deal with errors +When your JSON file content are not valid niet display an error and exit +with return code `1` +You can easily protect your script like this: +```sh +PROJECT_NAME=$(niet project.meta.name your-file.yaml) +if [ "$?" = "1" ]; then + echo "Error occur ${PROJECT_NAME}" +else + echo "Project name: ${PROJECT_NAME}" +fi +``` +## Examples +You can try niet by using the samples provided with the project sources code. +> All the following examples use the sample file available in niet sources code +at the following location `tests/samples/sample.yaml`. +Sample example: +```yaml +# tests/samples/sample.yaml +project: + meta: + name: my-project + foo: bar + list: + - item1 + - item2 + - item3 +``` +### Extract a single value +Retrieve the project name: +```sh +$ niet project.meta.name tests/samples/sample.yaml +my-project +``` +### Complexe search +Consider the following content: +``` +$ cat /var/lib/libvirt/dnsmasq/virbr0.status +[ + { + "ip-address": "192.168.122.113", + "mac-address": "52:54:00:91:14:02", + "hostname": "rhel79", + "expiry-time": 1644251254 + }, + { + "ip-address": "192.168.122.162", + "mac-address": "52:54:00:23:37:ed", + "hostname": "satellite", + "expiry-time": 1644251837 + } +] +``` +Here we want to retrieve the value of the ip-address field when the hostname +is equal to `satellite`. The following command will allow you to get this +value: +```sh +$ sed 's/ip/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='satellite'].ip" +192.168.122.162 +``` +You should notice that first we replace `-` by `_` by using the sed +command. We do that because `jmespath`, the underlying library used by `niet` +, poorly handle key that contain `-`. We chosen to replace all - by _ to avoid +any issues elsewhere on the file +Here is an exemple of an automated ssh connection in a kvm virtualised lab +environment by looking for vmname in dhcp file with `niet` and performing the +ssh connection to the server even if its ip changed. +The ssh connection here can be performed with this command: +```sh +ssh -o ProxyCommand='nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='''%h'''].ip_address") %p' root@rhel79 +``` +Tips - to ease that use you can for example set this `.ssh/config` entry: +``` +host lab-* +user root +ProxyCommand /usr/bin/nc $(sed 's/-/_/g' /var/lib/libvirt/dnsmasq/virbr0.status | niet "[?hostname=='$(echo %h | cut -d'-' -f2 )'].ip_address") %p +``` +And then perform a `ssh lab-rhel79` or a `ssh lab-satellite` to join all VMs +from your lab, by the hostname prefixed by `lab-`. +### Extract a list and parse it in shell +Deal with list of items +```sh +$ for el in $(niet project.list tests/samples/sample.yaml); do echo ${el}; done +item1 +item2 +item3 +``` +Also you can `eval` your `niet` output to setput some shell variables +that you can reuse in your shell scripts, the following example is similar to +the previous example but make use of the eval ouput format (`-f eval`): +```sh +$ eval $(niet -f eval project.list tests/samples/sample.yaml) +$ for el in ${project__list}; do echo $el; done +zero +one +two +three +``` +### Extract a complex object and parse it in shell +Extract the object as JSON to store it in shell variable : +```shell +$ project="$(niet -f json .project tests/samples/sample.yaml)" +``` +Then parse it after in bash in this example: +```shell +$ niet .meta.name <<< $project +my-project +``` +### Transform JSON to YAML +With niet you can easily convert your JSON to YAML +```shell +$ niet . tests/samples/sample.json -f yaml +project: + foo: bar + list: + - item1 + - item2 + - item3 + meta: + name: my-project +``` +### Transform YAML to JSON +With niet you can easily convert your YAML to JSON +```shell +$ niet . tests/samples/sample.yaml -f json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ] + } +} +``` +### Indent JSON file +This is an example of how to indent a JSON file : +```shell +$ niet . tests/samples/sample_not_indented.json +{ + "project": { + "meta": { + "name": "my-project" + }, + "foo": "bar", + "list": [ + "item1", + "item2", + "item3" + ], + "test-dash": "value" + } +} +``` +## Tips +You can pass your search with or without quotes like this: +```sh +$ niet project.meta.name your-file.yaml +$ niet "project.meta.name" your-file.yaml +``` +You can execute `niet` step by step by using the debug mode. It will allow +you to inspect your execution during your debug sessions. +## Contribute +If you want to contribute to niet [please first read the contribution guidelines](CONTRIBUTING.md) +## Licence +This project is under the MIT License. +[See the license file for more details](LICENSE) +Henry Tang <henryykt@gmail.com> +Hervé Beraud <hberaud@redhat.com> +Hervé Beraud <herveberaud.pro@gmail.com> +Sebastien Boyron <sebastien@boyron.eu> +Sébastien Boyron <sboyron@localhost.localdomain> +dj4ngo <sebastien@boyron.eu> +rechthemoon <rechthemoon@gmail.com> + +%prep +%autosetup -n niet-2.5.0 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-niet -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Tue Apr 11 2023 Python_Bot <Python_Bot@openeuler.org> - 2.5.0-1 +- Package Spec generated @@ -0,0 +1 @@ +f1fd46a015cb3e038be9fb686196e7bb niet-2.5.0.tar.gz |
