diff options
author | CoprDistGit <infra@openeuler.org> | 2023-06-20 08:43:27 +0000 |
---|---|---|
committer | CoprDistGit <infra@openeuler.org> | 2023-06-20 08:43:27 +0000 |
commit | 7acbb0e00a9240eef1309d0bbf4ea91b59db59b5 (patch) | |
tree | a1a22d03369baff51c78d7a92506922c390f55c1 | |
parent | 15ddebcebc579dd6e0979ae262975d785486310e (diff) |
automatic import of python-pygeofilteropeneuler20.03
-rw-r--r-- | .gitignore | 1 | ||||
-rw-r--r-- | python-pygeofilter.spec | 1169 | ||||
-rw-r--r-- | sources | 1 |
3 files changed, 1171 insertions, 0 deletions
@@ -0,0 +1 @@ +/pygeofilter-0.2.1.tar.gz diff --git a/python-pygeofilter.spec b/python-pygeofilter.spec new file mode 100644 index 0000000..cddbdf9 --- /dev/null +++ b/python-pygeofilter.spec @@ -0,0 +1,1169 @@ +%global _empty_manifest_terminate_build 0 +Name: python-pygeofilter +Version: 0.2.1 +Release: 1 +Summary: pygeofilter is a pure Python parser implementation of OGC filtering standards +License: MIT +URL: https://github.com/geopython/pygeofilter +Source0: https://mirrors.aliyun.com/pypi/web/packages/72/43/016390fdbc996bf2dcc7428d25212a6e638e7da998a3da83bead1bb3bb67/pygeofilter-0.2.1.tar.gz +BuildArch: noarch + +Requires: python3-dateparser +Requires: python3-lark +Requires: python3-pygeoif +Requires: python3-dataclasses +Requires: python3-django +Requires: python3-elasticsearch +Requires: python3-elasticsearch-dsl +Requires: python3-shapely +Requires: python3-geoalchemy2 +Requires: python3-sqlalchemy +Requires: python3-pygml + +%description +# pygeofilter +pygeofilter is a pure Python parser implementation of OGC filtering standards + +[](https://badge.fury.io/py/pygeofilter) +[](https://github.com/geopython/pygeofilter/actions) +[](https://pygeofilter.readthedocs.io/en/latest/?badge=latest) + + +## Features + +* Parsing of several filter encoding standards + * [CQL as defined in CSW 2.0](https://portal.ogc.org/files/?artifact_id=20555) + * [CQL JSON as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)](https://portal.ogc.org/files/96288#cql-json-schema) + * [JSON Filter Expressions (JFE)](https://github.com/tschaub/ogcapi-features/tree/json-array-expression/extensions/cql/jfe) + * Soon: + * [CQL Text as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)](https://portal.ogc.org/files/96288#cql-bnf) + * [FES](http://docs.opengeospatial.org/is/09-026r2/09-026r2.html) +* Several backends included + * [Django](https://www.djangoproject.com/) + * [SQLAlchemy](https://www.sqlalchemy.org/) + * [(Geo)Pandas](https://pandas.pydata.org/) + * Native Python objects + + +## Installation + +The package can be installed via PIP: + +```bash +pip install pygeofilter +``` + +Some features require additional dependencies. This currently only affects the backends. To install these, the features have to be listed: + +```bash +# for the Django backend +pip install pygeofilter[backend-django] + +# for the sqlalchemy backend +pip install pygeofilter[backend-sqlalchemy] + +# for the native backend +pip install pygeofilter[backend-native] +``` + +## Usage + +pygeofilter can be used on various levels. It provides parsers for various filtering languages, such as ECQL or CQL-JSON. Each parser lives in its own sub-package: + +```python +>>> from pygeofilter.parsers.ecql import parse as parse_ecql +>>> filters = parse_ecql(filter_expression) +>>> from pygeofilter.parsers.cql_json import parse as parse_json +>>> filters = parse_json(filter_expression) +``` + +Each parser creates an abstract syntax tree (AST) representation of that filter expression and thus unifies all possible languages to a single common denominator. All possible nodes are defined as classes in the `pygeofilter.ast` module. + +### Inspection + +The easiest way to inspect the resulting AST is to use the `get_repr` function, which returns a +nice string representation of what was parsed: + +```python +>>> filters = pygeofilter.parsers.ecql.parse('id = 10') +>>> print(pygeofilter.get_repr(ast)) +ATTRIBUTE id = LITERAL 10.0 +>>> +>>> +>>> filter_expr = '(number BETWEEN 5 AND 10 AND string NOT LIKE \'%B\') OR INTERSECTS(geometry, LINESTRING(0 0, 1 1))' +>>> print(pygeofilter.ast.get_repr(pygeofilter.parse(filter_expr))) +( + ( + ATTRIBUTE number BETWEEN 5 AND 10 + ) AND ( + ATTRIBUTE string NOT LIKE '%B' + ) +) OR ( + INTERSECTS(ATTRIBUTE geometry, Geometry(geometry={'type': 'LineString', 'coordinates': ((0.0, 0.0), (1.0, 1.0))})) +) +``` + +### Evaluation + +A parsed AST can then be evaluated and transformed into filtering mechanisms in the required context. Usually this is a language such as SQL or an object-relational mapper (ORM) interfacing a data store of some kind. + +There are a number of pre-defined backends available, where parsed expressions can be applied. For the moment this includes: + +* Django +* sqlalchemy +* (Geo)Pandas +* Pure Python object filtering + +The usage of those are described in their own documentation. + +pygeofilter provides mechanisms to help building such an evaluator (the included backends use them as well). The `Evaluator` class allows to conveniently walk through an AST depth-first and build the filters for the API in question. Only handled node classes are evaluated, unsupported ones will raise an exception. + +Consider this example: + +```python + +from pygeofilter import ast +from pygeofilter.backends.evaluator import Evaluator, handle +from myapi import filters # <- this is where the filters are created. + # of course, this can also be done in the + # evaluator itself + +# Evaluators must derive from the base class `Evaluator` to work +class MyAPIEvaluator(Evaluator): + # you can use any constructor as you need + def __init__(self, field_mapping=None, mapping_choices=None): + self.field_mapping = field_mapping + self.mapping_choices = mapping_choices + + # specify the handled classes in the `handle` decorator to mark + # this function as the handler for that node class(es) + @handle(ast.Not) + def not_(self, node, sub): + return filters.negate(sub) + + # multiple classes can be declared for the same handler function + @handle(ast.And, ast.Or) + def combination(self, node, lhs, rhs): + return filters.combine((lhs, rhs), node.op.value) + + # handle all sub-classes, like ast.Equal, ast.NotEqual, + # ast.LessThan, ast.GreaterThan, ... + @handle(ast.Comparison, subclasses=True) + def comparison(self, node, lhs, rhs): + return filters.compare( + lhs, + rhs, + node.op.value, + self.mapping_choices + ) + + @handle(ast.Between) + def between(self, node, lhs, low, high): + return filters.between( + lhs, + low, + high, + node.not_ + ) + + @handle(ast.Like) + def like(self, node, lhs): + return filters.like( + lhs, + node.pattern, + node.nocase, + node.not_, + self.mapping_choices + ) + + @handle(ast.In) + def in_(self, node, lhs, *options): + return filters.contains( + lhs, + options, + node.not_, + self.mapping_choices + ) + + def adopt(self, node, *sub_args): + # a "catch-all" function for node classes that are not + # handled elsewhere. Use with caution and raise exceptions + # yourself when a node class is not supported. + ... + + # ...further ast handlings removed for brevity +``` + +## Testing + +For testing, several requirements must be satisfied. These can be installed, via pip: + +```bash +pip install -r requirements-dev.txt +pip install -r requirements-test.txt +``` + +The functionality can be tested using `pytest`. + +```bash +python -m pytest +``` + +### Docker + +To execute tests in Docker: + +``` +docker build -t pygeofilter/test -f Dockerfile-3.9 . +docker run --rm pygeofilter/test +``` + +## Backends + +The following backends are shipped with `pygeofilter`. Some require additional dependencies, refer to the [installation](#installation) section for further details. + +### Django + +For Django there is a default backend implementation, where all the filters are translated to the +Django ORM. In order to use this integration, we need two dictionaries, one mapping the available +fields to the Django model fields, and one to map the fields that use `choices`. Consider the +following example models: + +```python +from django.contrib.gis.db import models + + +optional = dict(null=True, blank=True) + +class Record(models.Model): + identifier = models.CharField(max_length=256, unique=True, null=False) + geometry = models.GeometryField() + + float_attribute = models.FloatField(**optional) + int_attribute = models.IntegerField(**optional) + str_attribute = models.CharField(max_length=256, **optional) + datetime_attribute = models.DateTimeField(**optional) + choice_attribute = models.PositiveSmallIntegerField(choices=[ + (1, 'ASCENDING'), + (2, 'DESCENDING'),], + **optional) + + +class RecordMeta(models.Model): + record = models.ForeignKey(Record, on_delete=models.CASCADE, related_name='record_metas') + + float_meta_attribute = models.FloatField(**optional) + int_meta_attribute = models.IntegerField(**optional) + str_meta_attribute = models.CharField(max_length=256, **optional) + datetime_meta_attribute = models.DateTimeField(**optional) + choice_meta_attribute = models.PositiveSmallIntegerField(choices=[ + (1, 'X'), + (2, 'Y'), + (3, 'Z')], + **optional) +``` + +Now we can specify the field mappings and mapping choices to be used when applying the filters: + +```python +FIELD_MAPPING = { + 'identifier': 'identifier', + 'geometry': 'geometry', + 'floatAttribute': 'float_attribute', + 'intAttribute': 'int_attribute', + 'strAttribute': 'str_attribute', + 'datetimeAttribute': 'datetime_attribute', + 'choiceAttribute': 'choice_attribute', + + # meta fields + 'floatMetaAttribute': 'record_metas__float_meta_attribute', + 'intMetaAttribute': 'record_metas__int_meta_attribute', + 'strMetaAttribute': 'record_metas__str_meta_attribute', + 'datetimeMetaAttribute': 'record_metas__datetime_meta_attribute', + 'choiceMetaAttribute': 'record_metas__choice_meta_attribute', +} + +MAPPING_CHOICES = { + 'choiceAttribute': dict(Record._meta.get_field('choice_attribute').choices), + 'choiceMetaAttribute': dict(RecordMeta._meta.get_field('choice_meta_attribute').choices), +} +``` + +Finally we are able to connect the CQL AST to the Django database models. We also provide factory +functions to parse the timestamps, durations, geometries and envelopes, so that they can be used +with the ORM layer: + +```python +from pygeofilter.backends.django import to_filter +from pygeofilter.parsers.ecql import parse + +cql_expr = 'strMetaAttribute LIKE \'%parent%\' AND datetimeAttribute BEFORE 2000-01-01T00:00:01Z' + +ast = parse(cql_expr) +filters = to_filter(ast, mapping, mapping_choices) + +qs = Record.objects.filter(**filters) +``` + +### SQL + +`pygeofilter` provides a rudimentary way to create an SQL `WHERE` clause from an AST. The following example shows this usage in conjunction with the OGR `ExecuteSQL` function: + +```python +from osgeo import ogr +from pygeofilter.backends.sql import to_sql_where +from pygeofilter.parsers.ecql import parse + + +FIELD_MAPPING = { + 'str_attr': 'str_attr', + 'maybe_str_attr': 'maybe_str_attr', + 'int_attr': 'int_attr', + 'float_attr': 'float_attr', + 'date_attr': 'date_attr', + 'datetime_attr': 'datetime_attr', + 'point_attr': 'GEOMETRY', +} + +FUNCTION_MAP = { + 'sin': 'sin' +} + +# parse the expression +ast = parse('int_attr > 6') + +# open an OGR DataSource +data = ogr.Open(...) + +# create the WHERE clause, field and function mappings must be provided +where = to_sql_where(ast, FIELD_MAPPING, FUNCTION_MAP) + +# filter the DataSource to get a result Layer +layer = data.ExecuteSQL(f""" + SELECT id, str_attr, maybe_str_attr, int_attr, float_attr, date_attr, datetime_attr, GEOMETRY + FROM layer + WHERE {where} +""", None, "SQLite") +``` + +Note that it is vital to specify the `SQLite` dialect as this is the one used internally. + +:warning: Input values are *not* sanitized/separated from the generated SQL text. This is due to the compatibility with the OGR API not allowing to separate the SQL from the arguments. + + +### Optimization + +This is a special kind of backend, as the result of the AST evaluation is actually a new AST. The purpose of this backend is to eliminate static branches of the AST, potentially reducing the cost of an actual evaluation for filtering values. + +What parts of an AST can be optimized: + +- Arithmetic operations of purely static operands +- All predicates (spatial, temporal, array, `like`, `between`, `in`) if all of the operands are already static +- Functions, when passed in a special lookup table and all arguments are static +- `And` and `Or` combinators can be eliminated if either branch can be predicted + +What cannot be optimized are branches that contain references to attributes or functions not passed in the dictionary. + +The following example shows how a static computation can be optimized to a static value, replacing the whole branch of the AST: + +```python +>>> import math +>>> from pygeofilter import ast +>>> from pygeofilter.parsers.ecql import parse +>>> from pygeofilter.backends.optimize import optimize +>>> +>>> root = parse('attr < sin(3.7) - 5') +>>> optimized_root = optimize(root, {'sin': math.sin}) +>>> print(ast.get_repr(root)) +ATTRIBUTE attr < ( + ( + sin (3.7) + ) - 5 +) +>>> print(ast.get_repr(optimized_root)) +ATTRIBUTE attr < -5.529836140908493 +``` + + +%package -n python3-pygeofilter +Summary: pygeofilter is a pure Python parser implementation of OGC filtering standards +Provides: python-pygeofilter +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-pygeofilter +# pygeofilter +pygeofilter is a pure Python parser implementation of OGC filtering standards + +[](https://badge.fury.io/py/pygeofilter) +[](https://github.com/geopython/pygeofilter/actions) +[](https://pygeofilter.readthedocs.io/en/latest/?badge=latest) + + +## Features + +* Parsing of several filter encoding standards + * [CQL as defined in CSW 2.0](https://portal.ogc.org/files/?artifact_id=20555) + * [CQL JSON as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)](https://portal.ogc.org/files/96288#cql-json-schema) + * [JSON Filter Expressions (JFE)](https://github.com/tschaub/ogcapi-features/tree/json-array-expression/extensions/cql/jfe) + * Soon: + * [CQL Text as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)](https://portal.ogc.org/files/96288#cql-bnf) + * [FES](http://docs.opengeospatial.org/is/09-026r2/09-026r2.html) +* Several backends included + * [Django](https://www.djangoproject.com/) + * [SQLAlchemy](https://www.sqlalchemy.org/) + * [(Geo)Pandas](https://pandas.pydata.org/) + * Native Python objects + + +## Installation + +The package can be installed via PIP: + +```bash +pip install pygeofilter +``` + +Some features require additional dependencies. This currently only affects the backends. To install these, the features have to be listed: + +```bash +# for the Django backend +pip install pygeofilter[backend-django] + +# for the sqlalchemy backend +pip install pygeofilter[backend-sqlalchemy] + +# for the native backend +pip install pygeofilter[backend-native] +``` + +## Usage + +pygeofilter can be used on various levels. It provides parsers for various filtering languages, such as ECQL or CQL-JSON. Each parser lives in its own sub-package: + +```python +>>> from pygeofilter.parsers.ecql import parse as parse_ecql +>>> filters = parse_ecql(filter_expression) +>>> from pygeofilter.parsers.cql_json import parse as parse_json +>>> filters = parse_json(filter_expression) +``` + +Each parser creates an abstract syntax tree (AST) representation of that filter expression and thus unifies all possible languages to a single common denominator. All possible nodes are defined as classes in the `pygeofilter.ast` module. + +### Inspection + +The easiest way to inspect the resulting AST is to use the `get_repr` function, which returns a +nice string representation of what was parsed: + +```python +>>> filters = pygeofilter.parsers.ecql.parse('id = 10') +>>> print(pygeofilter.get_repr(ast)) +ATTRIBUTE id = LITERAL 10.0 +>>> +>>> +>>> filter_expr = '(number BETWEEN 5 AND 10 AND string NOT LIKE \'%B\') OR INTERSECTS(geometry, LINESTRING(0 0, 1 1))' +>>> print(pygeofilter.ast.get_repr(pygeofilter.parse(filter_expr))) +( + ( + ATTRIBUTE number BETWEEN 5 AND 10 + ) AND ( + ATTRIBUTE string NOT LIKE '%B' + ) +) OR ( + INTERSECTS(ATTRIBUTE geometry, Geometry(geometry={'type': 'LineString', 'coordinates': ((0.0, 0.0), (1.0, 1.0))})) +) +``` + +### Evaluation + +A parsed AST can then be evaluated and transformed into filtering mechanisms in the required context. Usually this is a language such as SQL or an object-relational mapper (ORM) interfacing a data store of some kind. + +There are a number of pre-defined backends available, where parsed expressions can be applied. For the moment this includes: + +* Django +* sqlalchemy +* (Geo)Pandas +* Pure Python object filtering + +The usage of those are described in their own documentation. + +pygeofilter provides mechanisms to help building such an evaluator (the included backends use them as well). The `Evaluator` class allows to conveniently walk through an AST depth-first and build the filters for the API in question. Only handled node classes are evaluated, unsupported ones will raise an exception. + +Consider this example: + +```python + +from pygeofilter import ast +from pygeofilter.backends.evaluator import Evaluator, handle +from myapi import filters # <- this is where the filters are created. + # of course, this can also be done in the + # evaluator itself + +# Evaluators must derive from the base class `Evaluator` to work +class MyAPIEvaluator(Evaluator): + # you can use any constructor as you need + def __init__(self, field_mapping=None, mapping_choices=None): + self.field_mapping = field_mapping + self.mapping_choices = mapping_choices + + # specify the handled classes in the `handle` decorator to mark + # this function as the handler for that node class(es) + @handle(ast.Not) + def not_(self, node, sub): + return filters.negate(sub) + + # multiple classes can be declared for the same handler function + @handle(ast.And, ast.Or) + def combination(self, node, lhs, rhs): + return filters.combine((lhs, rhs), node.op.value) + + # handle all sub-classes, like ast.Equal, ast.NotEqual, + # ast.LessThan, ast.GreaterThan, ... + @handle(ast.Comparison, subclasses=True) + def comparison(self, node, lhs, rhs): + return filters.compare( + lhs, + rhs, + node.op.value, + self.mapping_choices + ) + + @handle(ast.Between) + def between(self, node, lhs, low, high): + return filters.between( + lhs, + low, + high, + node.not_ + ) + + @handle(ast.Like) + def like(self, node, lhs): + return filters.like( + lhs, + node.pattern, + node.nocase, + node.not_, + self.mapping_choices + ) + + @handle(ast.In) + def in_(self, node, lhs, *options): + return filters.contains( + lhs, + options, + node.not_, + self.mapping_choices + ) + + def adopt(self, node, *sub_args): + # a "catch-all" function for node classes that are not + # handled elsewhere. Use with caution and raise exceptions + # yourself when a node class is not supported. + ... + + # ...further ast handlings removed for brevity +``` + +## Testing + +For testing, several requirements must be satisfied. These can be installed, via pip: + +```bash +pip install -r requirements-dev.txt +pip install -r requirements-test.txt +``` + +The functionality can be tested using `pytest`. + +```bash +python -m pytest +``` + +### Docker + +To execute tests in Docker: + +``` +docker build -t pygeofilter/test -f Dockerfile-3.9 . +docker run --rm pygeofilter/test +``` + +## Backends + +The following backends are shipped with `pygeofilter`. Some require additional dependencies, refer to the [installation](#installation) section for further details. + +### Django + +For Django there is a default backend implementation, where all the filters are translated to the +Django ORM. In order to use this integration, we need two dictionaries, one mapping the available +fields to the Django model fields, and one to map the fields that use `choices`. Consider the +following example models: + +```python +from django.contrib.gis.db import models + + +optional = dict(null=True, blank=True) + +class Record(models.Model): + identifier = models.CharField(max_length=256, unique=True, null=False) + geometry = models.GeometryField() + + float_attribute = models.FloatField(**optional) + int_attribute = models.IntegerField(**optional) + str_attribute = models.CharField(max_length=256, **optional) + datetime_attribute = models.DateTimeField(**optional) + choice_attribute = models.PositiveSmallIntegerField(choices=[ + (1, 'ASCENDING'), + (2, 'DESCENDING'),], + **optional) + + +class RecordMeta(models.Model): + record = models.ForeignKey(Record, on_delete=models.CASCADE, related_name='record_metas') + + float_meta_attribute = models.FloatField(**optional) + int_meta_attribute = models.IntegerField(**optional) + str_meta_attribute = models.CharField(max_length=256, **optional) + datetime_meta_attribute = models.DateTimeField(**optional) + choice_meta_attribute = models.PositiveSmallIntegerField(choices=[ + (1, 'X'), + (2, 'Y'), + (3, 'Z')], + **optional) +``` + +Now we can specify the field mappings and mapping choices to be used when applying the filters: + +```python +FIELD_MAPPING = { + 'identifier': 'identifier', + 'geometry': 'geometry', + 'floatAttribute': 'float_attribute', + 'intAttribute': 'int_attribute', + 'strAttribute': 'str_attribute', + 'datetimeAttribute': 'datetime_attribute', + 'choiceAttribute': 'choice_attribute', + + # meta fields + 'floatMetaAttribute': 'record_metas__float_meta_attribute', + 'intMetaAttribute': 'record_metas__int_meta_attribute', + 'strMetaAttribute': 'record_metas__str_meta_attribute', + 'datetimeMetaAttribute': 'record_metas__datetime_meta_attribute', + 'choiceMetaAttribute': 'record_metas__choice_meta_attribute', +} + +MAPPING_CHOICES = { + 'choiceAttribute': dict(Record._meta.get_field('choice_attribute').choices), + 'choiceMetaAttribute': dict(RecordMeta._meta.get_field('choice_meta_attribute').choices), +} +``` + +Finally we are able to connect the CQL AST to the Django database models. We also provide factory +functions to parse the timestamps, durations, geometries and envelopes, so that they can be used +with the ORM layer: + +```python +from pygeofilter.backends.django import to_filter +from pygeofilter.parsers.ecql import parse + +cql_expr = 'strMetaAttribute LIKE \'%parent%\' AND datetimeAttribute BEFORE 2000-01-01T00:00:01Z' + +ast = parse(cql_expr) +filters = to_filter(ast, mapping, mapping_choices) + +qs = Record.objects.filter(**filters) +``` + +### SQL + +`pygeofilter` provides a rudimentary way to create an SQL `WHERE` clause from an AST. The following example shows this usage in conjunction with the OGR `ExecuteSQL` function: + +```python +from osgeo import ogr +from pygeofilter.backends.sql import to_sql_where +from pygeofilter.parsers.ecql import parse + + +FIELD_MAPPING = { + 'str_attr': 'str_attr', + 'maybe_str_attr': 'maybe_str_attr', + 'int_attr': 'int_attr', + 'float_attr': 'float_attr', + 'date_attr': 'date_attr', + 'datetime_attr': 'datetime_attr', + 'point_attr': 'GEOMETRY', +} + +FUNCTION_MAP = { + 'sin': 'sin' +} + +# parse the expression +ast = parse('int_attr > 6') + +# open an OGR DataSource +data = ogr.Open(...) + +# create the WHERE clause, field and function mappings must be provided +where = to_sql_where(ast, FIELD_MAPPING, FUNCTION_MAP) + +# filter the DataSource to get a result Layer +layer = data.ExecuteSQL(f""" + SELECT id, str_attr, maybe_str_attr, int_attr, float_attr, date_attr, datetime_attr, GEOMETRY + FROM layer + WHERE {where} +""", None, "SQLite") +``` + +Note that it is vital to specify the `SQLite` dialect as this is the one used internally. + +:warning: Input values are *not* sanitized/separated from the generated SQL text. This is due to the compatibility with the OGR API not allowing to separate the SQL from the arguments. + + +### Optimization + +This is a special kind of backend, as the result of the AST evaluation is actually a new AST. The purpose of this backend is to eliminate static branches of the AST, potentially reducing the cost of an actual evaluation for filtering values. + +What parts of an AST can be optimized: + +- Arithmetic operations of purely static operands +- All predicates (spatial, temporal, array, `like`, `between`, `in`) if all of the operands are already static +- Functions, when passed in a special lookup table and all arguments are static +- `And` and `Or` combinators can be eliminated if either branch can be predicted + +What cannot be optimized are branches that contain references to attributes or functions not passed in the dictionary. + +The following example shows how a static computation can be optimized to a static value, replacing the whole branch of the AST: + +```python +>>> import math +>>> from pygeofilter import ast +>>> from pygeofilter.parsers.ecql import parse +>>> from pygeofilter.backends.optimize import optimize +>>> +>>> root = parse('attr < sin(3.7) - 5') +>>> optimized_root = optimize(root, {'sin': math.sin}) +>>> print(ast.get_repr(root)) +ATTRIBUTE attr < ( + ( + sin (3.7) + ) - 5 +) +>>> print(ast.get_repr(optimized_root)) +ATTRIBUTE attr < -5.529836140908493 +``` + + +%package help +Summary: Development documents and examples for pygeofilter +Provides: python3-pygeofilter-doc +%description help +# pygeofilter +pygeofilter is a pure Python parser implementation of OGC filtering standards + +[](https://badge.fury.io/py/pygeofilter) +[](https://github.com/geopython/pygeofilter/actions) +[](https://pygeofilter.readthedocs.io/en/latest/?badge=latest) + + +## Features + +* Parsing of several filter encoding standards + * [CQL as defined in CSW 2.0](https://portal.ogc.org/files/?artifact_id=20555) + * [CQL JSON as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)](https://portal.ogc.org/files/96288#cql-json-schema) + * [JSON Filter Expressions (JFE)](https://github.com/tschaub/ogcapi-features/tree/json-array-expression/extensions/cql/jfe) + * Soon: + * [CQL Text as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)](https://portal.ogc.org/files/96288#cql-bnf) + * [FES](http://docs.opengeospatial.org/is/09-026r2/09-026r2.html) +* Several backends included + * [Django](https://www.djangoproject.com/) + * [SQLAlchemy](https://www.sqlalchemy.org/) + * [(Geo)Pandas](https://pandas.pydata.org/) + * Native Python objects + + +## Installation + +The package can be installed via PIP: + +```bash +pip install pygeofilter +``` + +Some features require additional dependencies. This currently only affects the backends. To install these, the features have to be listed: + +```bash +# for the Django backend +pip install pygeofilter[backend-django] + +# for the sqlalchemy backend +pip install pygeofilter[backend-sqlalchemy] + +# for the native backend +pip install pygeofilter[backend-native] +``` + +## Usage + +pygeofilter can be used on various levels. It provides parsers for various filtering languages, such as ECQL or CQL-JSON. Each parser lives in its own sub-package: + +```python +>>> from pygeofilter.parsers.ecql import parse as parse_ecql +>>> filters = parse_ecql(filter_expression) +>>> from pygeofilter.parsers.cql_json import parse as parse_json +>>> filters = parse_json(filter_expression) +``` + +Each parser creates an abstract syntax tree (AST) representation of that filter expression and thus unifies all possible languages to a single common denominator. All possible nodes are defined as classes in the `pygeofilter.ast` module. + +### Inspection + +The easiest way to inspect the resulting AST is to use the `get_repr` function, which returns a +nice string representation of what was parsed: + +```python +>>> filters = pygeofilter.parsers.ecql.parse('id = 10') +>>> print(pygeofilter.get_repr(ast)) +ATTRIBUTE id = LITERAL 10.0 +>>> +>>> +>>> filter_expr = '(number BETWEEN 5 AND 10 AND string NOT LIKE \'%B\') OR INTERSECTS(geometry, LINESTRING(0 0, 1 1))' +>>> print(pygeofilter.ast.get_repr(pygeofilter.parse(filter_expr))) +( + ( + ATTRIBUTE number BETWEEN 5 AND 10 + ) AND ( + ATTRIBUTE string NOT LIKE '%B' + ) +) OR ( + INTERSECTS(ATTRIBUTE geometry, Geometry(geometry={'type': 'LineString', 'coordinates': ((0.0, 0.0), (1.0, 1.0))})) +) +``` + +### Evaluation + +A parsed AST can then be evaluated and transformed into filtering mechanisms in the required context. Usually this is a language such as SQL or an object-relational mapper (ORM) interfacing a data store of some kind. + +There are a number of pre-defined backends available, where parsed expressions can be applied. For the moment this includes: + +* Django +* sqlalchemy +* (Geo)Pandas +* Pure Python object filtering + +The usage of those are described in their own documentation. + +pygeofilter provides mechanisms to help building such an evaluator (the included backends use them as well). The `Evaluator` class allows to conveniently walk through an AST depth-first and build the filters for the API in question. Only handled node classes are evaluated, unsupported ones will raise an exception. + +Consider this example: + +```python + +from pygeofilter import ast +from pygeofilter.backends.evaluator import Evaluator, handle +from myapi import filters # <- this is where the filters are created. + # of course, this can also be done in the + # evaluator itself + +# Evaluators must derive from the base class `Evaluator` to work +class MyAPIEvaluator(Evaluator): + # you can use any constructor as you need + def __init__(self, field_mapping=None, mapping_choices=None): + self.field_mapping = field_mapping + self.mapping_choices = mapping_choices + + # specify the handled classes in the `handle` decorator to mark + # this function as the handler for that node class(es) + @handle(ast.Not) + def not_(self, node, sub): + return filters.negate(sub) + + # multiple classes can be declared for the same handler function + @handle(ast.And, ast.Or) + def combination(self, node, lhs, rhs): + return filters.combine((lhs, rhs), node.op.value) + + # handle all sub-classes, like ast.Equal, ast.NotEqual, + # ast.LessThan, ast.GreaterThan, ... + @handle(ast.Comparison, subclasses=True) + def comparison(self, node, lhs, rhs): + return filters.compare( + lhs, + rhs, + node.op.value, + self.mapping_choices + ) + + @handle(ast.Between) + def between(self, node, lhs, low, high): + return filters.between( + lhs, + low, + high, + node.not_ + ) + + @handle(ast.Like) + def like(self, node, lhs): + return filters.like( + lhs, + node.pattern, + node.nocase, + node.not_, + self.mapping_choices + ) + + @handle(ast.In) + def in_(self, node, lhs, *options): + return filters.contains( + lhs, + options, + node.not_, + self.mapping_choices + ) + + def adopt(self, node, *sub_args): + # a "catch-all" function for node classes that are not + # handled elsewhere. Use with caution and raise exceptions + # yourself when a node class is not supported. + ... + + # ...further ast handlings removed for brevity +``` + +## Testing + +For testing, several requirements must be satisfied. These can be installed, via pip: + +```bash +pip install -r requirements-dev.txt +pip install -r requirements-test.txt +``` + +The functionality can be tested using `pytest`. + +```bash +python -m pytest +``` + +### Docker + +To execute tests in Docker: + +``` +docker build -t pygeofilter/test -f Dockerfile-3.9 . +docker run --rm pygeofilter/test +``` + +## Backends + +The following backends are shipped with `pygeofilter`. Some require additional dependencies, refer to the [installation](#installation) section for further details. + +### Django + +For Django there is a default backend implementation, where all the filters are translated to the +Django ORM. In order to use this integration, we need two dictionaries, one mapping the available +fields to the Django model fields, and one to map the fields that use `choices`. Consider the +following example models: + +```python +from django.contrib.gis.db import models + + +optional = dict(null=True, blank=True) + +class Record(models.Model): + identifier = models.CharField(max_length=256, unique=True, null=False) + geometry = models.GeometryField() + + float_attribute = models.FloatField(**optional) + int_attribute = models.IntegerField(**optional) + str_attribute = models.CharField(max_length=256, **optional) + datetime_attribute = models.DateTimeField(**optional) + choice_attribute = models.PositiveSmallIntegerField(choices=[ + (1, 'ASCENDING'), + (2, 'DESCENDING'),], + **optional) + + +class RecordMeta(models.Model): + record = models.ForeignKey(Record, on_delete=models.CASCADE, related_name='record_metas') + + float_meta_attribute = models.FloatField(**optional) + int_meta_attribute = models.IntegerField(**optional) + str_meta_attribute = models.CharField(max_length=256, **optional) + datetime_meta_attribute = models.DateTimeField(**optional) + choice_meta_attribute = models.PositiveSmallIntegerField(choices=[ + (1, 'X'), + (2, 'Y'), + (3, 'Z')], + **optional) +``` + +Now we can specify the field mappings and mapping choices to be used when applying the filters: + +```python +FIELD_MAPPING = { + 'identifier': 'identifier', + 'geometry': 'geometry', + 'floatAttribute': 'float_attribute', + 'intAttribute': 'int_attribute', + 'strAttribute': 'str_attribute', + 'datetimeAttribute': 'datetime_attribute', + 'choiceAttribute': 'choice_attribute', + + # meta fields + 'floatMetaAttribute': 'record_metas__float_meta_attribute', + 'intMetaAttribute': 'record_metas__int_meta_attribute', + 'strMetaAttribute': 'record_metas__str_meta_attribute', + 'datetimeMetaAttribute': 'record_metas__datetime_meta_attribute', + 'choiceMetaAttribute': 'record_metas__choice_meta_attribute', +} + +MAPPING_CHOICES = { + 'choiceAttribute': dict(Record._meta.get_field('choice_attribute').choices), + 'choiceMetaAttribute': dict(RecordMeta._meta.get_field('choice_meta_attribute').choices), +} +``` + +Finally we are able to connect the CQL AST to the Django database models. We also provide factory +functions to parse the timestamps, durations, geometries and envelopes, so that they can be used +with the ORM layer: + +```python +from pygeofilter.backends.django import to_filter +from pygeofilter.parsers.ecql import parse + +cql_expr = 'strMetaAttribute LIKE \'%parent%\' AND datetimeAttribute BEFORE 2000-01-01T00:00:01Z' + +ast = parse(cql_expr) +filters = to_filter(ast, mapping, mapping_choices) + +qs = Record.objects.filter(**filters) +``` + +### SQL + +`pygeofilter` provides a rudimentary way to create an SQL `WHERE` clause from an AST. The following example shows this usage in conjunction with the OGR `ExecuteSQL` function: + +```python +from osgeo import ogr +from pygeofilter.backends.sql import to_sql_where +from pygeofilter.parsers.ecql import parse + + +FIELD_MAPPING = { + 'str_attr': 'str_attr', + 'maybe_str_attr': 'maybe_str_attr', + 'int_attr': 'int_attr', + 'float_attr': 'float_attr', + 'date_attr': 'date_attr', + 'datetime_attr': 'datetime_attr', + 'point_attr': 'GEOMETRY', +} + +FUNCTION_MAP = { + 'sin': 'sin' +} + +# parse the expression +ast = parse('int_attr > 6') + +# open an OGR DataSource +data = ogr.Open(...) + +# create the WHERE clause, field and function mappings must be provided +where = to_sql_where(ast, FIELD_MAPPING, FUNCTION_MAP) + +# filter the DataSource to get a result Layer +layer = data.ExecuteSQL(f""" + SELECT id, str_attr, maybe_str_attr, int_attr, float_attr, date_attr, datetime_attr, GEOMETRY + FROM layer + WHERE {where} +""", None, "SQLite") +``` + +Note that it is vital to specify the `SQLite` dialect as this is the one used internally. + +:warning: Input values are *not* sanitized/separated from the generated SQL text. This is due to the compatibility with the OGR API not allowing to separate the SQL from the arguments. + + +### Optimization + +This is a special kind of backend, as the result of the AST evaluation is actually a new AST. The purpose of this backend is to eliminate static branches of the AST, potentially reducing the cost of an actual evaluation for filtering values. + +What parts of an AST can be optimized: + +- Arithmetic operations of purely static operands +- All predicates (spatial, temporal, array, `like`, `between`, `in`) if all of the operands are already static +- Functions, when passed in a special lookup table and all arguments are static +- `And` and `Or` combinators can be eliminated if either branch can be predicted + +What cannot be optimized are branches that contain references to attributes or functions not passed in the dictionary. + +The following example shows how a static computation can be optimized to a static value, replacing the whole branch of the AST: + +```python +>>> import math +>>> from pygeofilter import ast +>>> from pygeofilter.parsers.ecql import parse +>>> from pygeofilter.backends.optimize import optimize +>>> +>>> root = parse('attr < sin(3.7) - 5') +>>> optimized_root = optimize(root, {'sin': math.sin}) +>>> print(ast.get_repr(root)) +ATTRIBUTE attr < ( + ( + sin (3.7) + ) - 5 +) +>>> print(ast.get_repr(optimized_root)) +ATTRIBUTE attr < -5.529836140908493 +``` + + +%prep +%autosetup -n pygeofilter-0.2.1 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "\"/%h/%f\"\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "\"/%h/%f.gz\"\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-pygeofilter -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Tue Jun 20 2023 Python_Bot <Python_Bot@openeuler.org> - 0.2.1-1 +- Package Spec generated @@ -0,0 +1 @@ +ebe241057499f7c1dabc89ecb811d5da pygeofilter-0.2.1.tar.gz |