diff options
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | python-fabric-fim.spec | 790 | ||||
| -rw-r--r-- | sources | 1 |
3 files changed, 792 insertions, 0 deletions
@@ -0,0 +1 @@ +/fabric_fim-1.4.12.tar.gz diff --git a/python-fabric-fim.spec b/python-fabric-fim.spec new file mode 100644 index 0000000..d62b74a --- /dev/null +++ b/python-fabric-fim.spec @@ -0,0 +1,790 @@ +%global _empty_manifest_terminate_build 0 +Name: python-fabric-fim +Version: 1.4.12 +Release: 1 +Summary: FABRIC Information Model Library +License: MIT License +URL: https://github.com/fabric-testbed/InformationModel +Source0: https://mirrors.nju.edu.cn/pypi/web/packages/bc/ac/5dc414a323f764a9c454e6e71d014910b1e4d5c08b599f680b03491fd944/fabric_fim-1.4.12.tar.gz +BuildArch: noarch + +Requires: python3-matplotlib +Requires: python3-neo4j +Requires: python3-networkx +Requires: python3-networkx-query +Requires: python3-PyYAML +Requires: python3-recordclass +Requires: python3-requests + +%description +[](https://pypi.org/project/fabric-fim/) + +# Information Model + +## Basic Graph Abstractions + +FABRIC Information Model library, containing class definitions and methods for operating +on different types of information model representations (sliver and slice). + +The implementation covers the following models and model transformations: +- ARM (Aggregate Resource Models) - generated by aggregate owner, may contain multiple resource delegations intended + for different brokers +- ADM (Aggregate Delegation Model) - a single ARM may be partitioned into one or more ADMs intended for different + brokers +- CBM (Combined Broker Model) - multiple ADMs sent to a broker from different aggregates are merged into a single CBM. +If needed ADMs can be 'unmerged' from a CBM (when e.g. an aggregate goes off-line) +- BQM (Broker Query Model) - multiple types of BQMs can be produced by a broker in response to `listResources`-like +queries from the information contained in CBM and live allocation information stored by the Broker. +The returned models can contain different levels of details of resources. BQMs can be used by user tools, + portals or other entities which need to inspect available testbed resources +- ASM (Abstract Slice Mode) - model generated by user tools, passed to the Orchestrator, which describes a slice + topology. This model can be annotated by Orchestrator and Aggregate Managers with details of the provisioned (vs. + requested) infrastructure and returned back to the user. ASMs encompass what is commonly referred to as Slice Request + and Slice Manifest. + +These models are based on a common framework of Abstract Property Graph class `fim.graph.abc_property_graph.py`, which are +then subclassed into a NetworkX-based implementation and a Neo4j-based implementation of abstract functionalities. +Further types of models are built on top of these two implementations, thus allowing for easy interchange of the +underlying graph implementation - in-memory (via NetworkX) or persistent (via Neo4j). + +There is a generic graph validation framework built-in - validation is stronger/more thorough when operating on Neo4j +implementations due to stronger expressivity of Neo4j Cypher query language. Graph validation rules can be found under +`fim.graph.data`. + +ARM, ADM, CBM definitions and implementations can be found under `fim.graph.resources`, ASM - under `fim.graph.slices`. +Abstract definitions of BQM are found under `fim.graph.resources`, however multiple subtypes of BQM (with different +levels of topology details presented to the requestor). Generation of different types of BQMs is done via broker plugins +intended to be external to FIM (perhaps inside the Control Framework). See `fim.pluggable.py` for more details on +plugins. + +## Higher-level Abstractions + +On top of basic property graph models described above, FIM offers additional more convenient abstractions that +interchangeably use the property graph implementations (NetworkX or Neo4j) underneath to allow for easier manipulation, +inspection of data. + +The main set of abstractions is implemented under `fim.user` package, which provides +[additional documentation](fim/user/README.md). + +Supporting intermediate models are implemented under `fim.sliver`, however these models are largely internal to FIM and +Control Framework, and generally are not exposed to end-users. + +## Structure of the code + +This figure reflects the overall structure of the code: + + + +Follow this link into [fim/README.md](fim/README.md) to explore. + +## Development environment + +The recommended way is to set up your development environment using `virtualenvwrapper` after checking +out the code: +```bash +$ git clone git@github.com:fabric-testbed/InformationModel.git +$ cd InformationModel +$ mkvirtualenv -r requirements.txt infomodel +$ workon infomodel +(infomodel) $ +``` + +You can also use the built-in `venv` tool that comes with newer versions of Python as follows: +``` +$ git clone git@github.com:fabric-testbed/InformationModel.git +$ cd InformationModel +$ python3 -m venv venv +$ source ./venv/bin/activate +$ pip install --upgrade pip setuptools +(venv) $ +``` + +Depending on which parts of FIM you are developing you may need to have +[Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc) running. +Working on any models whose names start with `Neo4j` generally requires using the Neo4j Docker. + +Models which start with `NetworkX` operate on in-memory models using NetworkX toolkit and don't require the Neo4j Docker. +Higher-level abstractions under `fim.user` can all be debugged and tested with in-memory NetworkX models, although they +operate on both types of models, depending on the situation. + +Follow the instructions with the container to start it. + +## Installation + +Multiple installation options possible. For CF development the recommended method is to +[install from GitHub MASTER branch](https://codeinthehole.com/tips/using-pip-and-requirementstxt-to-install-from-the-head-of-a-github-branch/): +```bash +$ pip install git+https://github.com/fabric-testbed/InformationModel.git +``` + +For developing and testing the FIM code itself use editable install (from top-level directory) and +install build, twine and pytest: +```bash +(infomodel) $ pip install -e . +(infomodel) $ pip install build +(infomodel) $ pip install pytest +(infomodel) $ pip install twine +``` + +For inclusion in tools, etc, use PyPi +```bash +$ pip install fabric-fim +``` + +### Compatibility with Neo4j + +Below is the compatibility matrix showing which major versions of FIM support which versions of Neo4j +(tested with the corresponding versions of [Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc)) + +| FIM Major version | Neo4j-APOC docker version | +|--------------------|---------------------------| + | 1.0-1.2 | 4.0.3 | + | 1.3-1.4 | 4.1.6 | + | 1.5 | 5.3.0 | + +## Test Graphs + +### Original Test Graphs + +Original test graphs are located under [tests/models](tests/models). All graphs were created using +[yEd](https://www.yworks.com/products/yed) +desktop graph editor (note that on-line version does not provide the same flexibility for creating custom node +and link properties). + +### Generating New Test Graphs + +Depending on the type of graph, many test graphs can be generated using higher-level abstraction libraries. + +For example ASMs can be generated starting from `fim.user.topology.ExperimentTopology` object, while ARMs can be +generated using `fim.user.topology.SubstrateTopology`. Other models are generally derived from these types of models, +as described in the overview. + +Test graphs are generated by unit tests: +- ASMs are generated in `test/slice_topology_test.py` +- ARMs are generated in `test/substrate_topology_test.py` +- ARM-ADM-CBM operations are tested in `test/zz_neo4j_pg_test.py` based on serializations produced in + `test/substrate_topology_test.py`. + +## Testing and building + +In order to run tests, you will need to use [pytest](https://pypi.org/project/pytest/). +Also, in order to support tests of Neo4j-implemented models, Neo4j-APOC should be running in a Docker container: + +```console +$ docker run -p7474:7474 -p7687:7687 -e NEO4J_AUTH=neo4j/password \ + -v $(pwd)/neo4j/data:/data -v $(pwd)/neo4j/imports:/imports \ + fabrictestbed/neo4j-apoc +``` + +Note that the directories `neo4j/data` and `neo4j/imports` must exist for the above command to work as expected. + +Wait for neo4j-apoc Docker container to start up, and then run pytest: + +```console +$ pytest [-s] test +``` + +This will produce substrate ARM models and save them into file in project root folder. + +## Graph validation + +All graphs loaded into Neo4j (whether from files or being passed in as part of query or delegation) +must conform to a set of rules expressed as Cypher queries. The basic set of rules for all +types of graphs are located in [fim/graph/graph_validation_rules.json](fim/graph/data/graph_validation_rules.json). + +Prior to ingestion graphs are also tested on general syntax validity of JSON-formatted fields. + +Additional rule files specific to model types may govern the validity of specific models. + +NetworkX-based graphs also undergo validation, but more limited in scope due to lack of tools. + +## Using fim_util.py utility + +The utility supports a number of operations on GraphML files - enumerating nodes (for graphs) +coming out of yEd, loading into an instance of Neo4j, deleting graphs from Neo4j. + +Start the [Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc) + +Create fim_config.yaml with the following structure under `util/`: +``` +# default fim_config configuration file for FIM utilities +neo4j: + url: neo4j://0.0.0.0:7687 + user: neo4j + pass: password + import_host_dir: /host/directory/seen/by/neo4j/docker/as/imports + import_dir: /imports +``` +Parameters `password` and `import_host_dir` depend on how the Docker container is started in +the procedure above. Other parameters should remain unchanged from what is shown. + +Run the utility for detailed help for the various operations: +``` +(infomodel) $ python fim_util.py -h +``` + +Generally the utility is good for e.g. loading a graph file: `python fim_util.py -l -f -r <graphml file>` or +merging multiple advertisements: `python fim_util.py -m -f <file1> -f <file2>`. Most options take multiple `-f` +and related options so can e.g. load multiple files at once. + +When testing large graphs, note that by default Neo4j visualizes the first 300 nodes in a browser. If you +want to see a full graph, increase this limit (at neo4j prompt): +``` +:config initialNodeDisplay: 5000 +``` + +## Neo4j Performance Considerations +For performance reasons it is critical that every instance of Neo4j has appropriate indexes created. +Neo4j label `GraphNode` is hard-coded within FIM - every graph node has this label and another label is created +from the Class property and is meaningful to FIM. +This is done for performance reasons to make it easier to create indexes and query models using those indexes. + +The following indexes are required (indexes are created automatically by Neo4jGraphImporter whenever it is used): +``` +CREATE INDEX graphid FOR (n:GraphNode) ON (n.GraphID) +CREATE INDEX graphid_nodeid FOR (n:GraphNode) ON (n.GraphID, n.NodeID) +CREATE INDEX graphid_nodeid_type FOR (n:GraphNode) ON (n.GraphID, n.NodeID, n.Type) +CREATE INDEX graphid_type FOR (n:GraphNode) ON (n.GraphID, n.Type) +``` +Available indexes can be checked via console by using the `:schema` command. + +An index can be dropped using the following command (substitute appropriate index name): +``` +DROP INDEX graphid +``` + +See [additional documentation](https://neo4j.com/docs/cypher-manual/current/administration/indexes-for-search-performance/). + + +%package -n python3-fabric-fim +Summary: FABRIC Information Model Library +Provides: python-fabric-fim +BuildRequires: python3-devel +BuildRequires: python3-setuptools +BuildRequires: python3-pip +%description -n python3-fabric-fim +[](https://pypi.org/project/fabric-fim/) + +# Information Model + +## Basic Graph Abstractions + +FABRIC Information Model library, containing class definitions and methods for operating +on different types of information model representations (sliver and slice). + +The implementation covers the following models and model transformations: +- ARM (Aggregate Resource Models) - generated by aggregate owner, may contain multiple resource delegations intended + for different brokers +- ADM (Aggregate Delegation Model) - a single ARM may be partitioned into one or more ADMs intended for different + brokers +- CBM (Combined Broker Model) - multiple ADMs sent to a broker from different aggregates are merged into a single CBM. +If needed ADMs can be 'unmerged' from a CBM (when e.g. an aggregate goes off-line) +- BQM (Broker Query Model) - multiple types of BQMs can be produced by a broker in response to `listResources`-like +queries from the information contained in CBM and live allocation information stored by the Broker. +The returned models can contain different levels of details of resources. BQMs can be used by user tools, + portals or other entities which need to inspect available testbed resources +- ASM (Abstract Slice Mode) - model generated by user tools, passed to the Orchestrator, which describes a slice + topology. This model can be annotated by Orchestrator and Aggregate Managers with details of the provisioned (vs. + requested) infrastructure and returned back to the user. ASMs encompass what is commonly referred to as Slice Request + and Slice Manifest. + +These models are based on a common framework of Abstract Property Graph class `fim.graph.abc_property_graph.py`, which are +then subclassed into a NetworkX-based implementation and a Neo4j-based implementation of abstract functionalities. +Further types of models are built on top of these two implementations, thus allowing for easy interchange of the +underlying graph implementation - in-memory (via NetworkX) or persistent (via Neo4j). + +There is a generic graph validation framework built-in - validation is stronger/more thorough when operating on Neo4j +implementations due to stronger expressivity of Neo4j Cypher query language. Graph validation rules can be found under +`fim.graph.data`. + +ARM, ADM, CBM definitions and implementations can be found under `fim.graph.resources`, ASM - under `fim.graph.slices`. +Abstract definitions of BQM are found under `fim.graph.resources`, however multiple subtypes of BQM (with different +levels of topology details presented to the requestor). Generation of different types of BQMs is done via broker plugins +intended to be external to FIM (perhaps inside the Control Framework). See `fim.pluggable.py` for more details on +plugins. + +## Higher-level Abstractions + +On top of basic property graph models described above, FIM offers additional more convenient abstractions that +interchangeably use the property graph implementations (NetworkX or Neo4j) underneath to allow for easier manipulation, +inspection of data. + +The main set of abstractions is implemented under `fim.user` package, which provides +[additional documentation](fim/user/README.md). + +Supporting intermediate models are implemented under `fim.sliver`, however these models are largely internal to FIM and +Control Framework, and generally are not exposed to end-users. + +## Structure of the code + +This figure reflects the overall structure of the code: + + + +Follow this link into [fim/README.md](fim/README.md) to explore. + +## Development environment + +The recommended way is to set up your development environment using `virtualenvwrapper` after checking +out the code: +```bash +$ git clone git@github.com:fabric-testbed/InformationModel.git +$ cd InformationModel +$ mkvirtualenv -r requirements.txt infomodel +$ workon infomodel +(infomodel) $ +``` + +You can also use the built-in `venv` tool that comes with newer versions of Python as follows: +``` +$ git clone git@github.com:fabric-testbed/InformationModel.git +$ cd InformationModel +$ python3 -m venv venv +$ source ./venv/bin/activate +$ pip install --upgrade pip setuptools +(venv) $ +``` + +Depending on which parts of FIM you are developing you may need to have +[Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc) running. +Working on any models whose names start with `Neo4j` generally requires using the Neo4j Docker. + +Models which start with `NetworkX` operate on in-memory models using NetworkX toolkit and don't require the Neo4j Docker. +Higher-level abstractions under `fim.user` can all be debugged and tested with in-memory NetworkX models, although they +operate on both types of models, depending on the situation. + +Follow the instructions with the container to start it. + +## Installation + +Multiple installation options possible. For CF development the recommended method is to +[install from GitHub MASTER branch](https://codeinthehole.com/tips/using-pip-and-requirementstxt-to-install-from-the-head-of-a-github-branch/): +```bash +$ pip install git+https://github.com/fabric-testbed/InformationModel.git +``` + +For developing and testing the FIM code itself use editable install (from top-level directory) and +install build, twine and pytest: +```bash +(infomodel) $ pip install -e . +(infomodel) $ pip install build +(infomodel) $ pip install pytest +(infomodel) $ pip install twine +``` + +For inclusion in tools, etc, use PyPi +```bash +$ pip install fabric-fim +``` + +### Compatibility with Neo4j + +Below is the compatibility matrix showing which major versions of FIM support which versions of Neo4j +(tested with the corresponding versions of [Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc)) + +| FIM Major version | Neo4j-APOC docker version | +|--------------------|---------------------------| + | 1.0-1.2 | 4.0.3 | + | 1.3-1.4 | 4.1.6 | + | 1.5 | 5.3.0 | + +## Test Graphs + +### Original Test Graphs + +Original test graphs are located under [tests/models](tests/models). All graphs were created using +[yEd](https://www.yworks.com/products/yed) +desktop graph editor (note that on-line version does not provide the same flexibility for creating custom node +and link properties). + +### Generating New Test Graphs + +Depending on the type of graph, many test graphs can be generated using higher-level abstraction libraries. + +For example ASMs can be generated starting from `fim.user.topology.ExperimentTopology` object, while ARMs can be +generated using `fim.user.topology.SubstrateTopology`. Other models are generally derived from these types of models, +as described in the overview. + +Test graphs are generated by unit tests: +- ASMs are generated in `test/slice_topology_test.py` +- ARMs are generated in `test/substrate_topology_test.py` +- ARM-ADM-CBM operations are tested in `test/zz_neo4j_pg_test.py` based on serializations produced in + `test/substrate_topology_test.py`. + +## Testing and building + +In order to run tests, you will need to use [pytest](https://pypi.org/project/pytest/). +Also, in order to support tests of Neo4j-implemented models, Neo4j-APOC should be running in a Docker container: + +```console +$ docker run -p7474:7474 -p7687:7687 -e NEO4J_AUTH=neo4j/password \ + -v $(pwd)/neo4j/data:/data -v $(pwd)/neo4j/imports:/imports \ + fabrictestbed/neo4j-apoc +``` + +Note that the directories `neo4j/data` and `neo4j/imports` must exist for the above command to work as expected. + +Wait for neo4j-apoc Docker container to start up, and then run pytest: + +```console +$ pytest [-s] test +``` + +This will produce substrate ARM models and save them into file in project root folder. + +## Graph validation + +All graphs loaded into Neo4j (whether from files or being passed in as part of query or delegation) +must conform to a set of rules expressed as Cypher queries. The basic set of rules for all +types of graphs are located in [fim/graph/graph_validation_rules.json](fim/graph/data/graph_validation_rules.json). + +Prior to ingestion graphs are also tested on general syntax validity of JSON-formatted fields. + +Additional rule files specific to model types may govern the validity of specific models. + +NetworkX-based graphs also undergo validation, but more limited in scope due to lack of tools. + +## Using fim_util.py utility + +The utility supports a number of operations on GraphML files - enumerating nodes (for graphs) +coming out of yEd, loading into an instance of Neo4j, deleting graphs from Neo4j. + +Start the [Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc) + +Create fim_config.yaml with the following structure under `util/`: +``` +# default fim_config configuration file for FIM utilities +neo4j: + url: neo4j://0.0.0.0:7687 + user: neo4j + pass: password + import_host_dir: /host/directory/seen/by/neo4j/docker/as/imports + import_dir: /imports +``` +Parameters `password` and `import_host_dir` depend on how the Docker container is started in +the procedure above. Other parameters should remain unchanged from what is shown. + +Run the utility for detailed help for the various operations: +``` +(infomodel) $ python fim_util.py -h +``` + +Generally the utility is good for e.g. loading a graph file: `python fim_util.py -l -f -r <graphml file>` or +merging multiple advertisements: `python fim_util.py -m -f <file1> -f <file2>`. Most options take multiple `-f` +and related options so can e.g. load multiple files at once. + +When testing large graphs, note that by default Neo4j visualizes the first 300 nodes in a browser. If you +want to see a full graph, increase this limit (at neo4j prompt): +``` +:config initialNodeDisplay: 5000 +``` + +## Neo4j Performance Considerations +For performance reasons it is critical that every instance of Neo4j has appropriate indexes created. +Neo4j label `GraphNode` is hard-coded within FIM - every graph node has this label and another label is created +from the Class property and is meaningful to FIM. +This is done for performance reasons to make it easier to create indexes and query models using those indexes. + +The following indexes are required (indexes are created automatically by Neo4jGraphImporter whenever it is used): +``` +CREATE INDEX graphid FOR (n:GraphNode) ON (n.GraphID) +CREATE INDEX graphid_nodeid FOR (n:GraphNode) ON (n.GraphID, n.NodeID) +CREATE INDEX graphid_nodeid_type FOR (n:GraphNode) ON (n.GraphID, n.NodeID, n.Type) +CREATE INDEX graphid_type FOR (n:GraphNode) ON (n.GraphID, n.Type) +``` +Available indexes can be checked via console by using the `:schema` command. + +An index can be dropped using the following command (substitute appropriate index name): +``` +DROP INDEX graphid +``` + +See [additional documentation](https://neo4j.com/docs/cypher-manual/current/administration/indexes-for-search-performance/). + + +%package help +Summary: Development documents and examples for fabric-fim +Provides: python3-fabric-fim-doc +%description help +[](https://pypi.org/project/fabric-fim/) + +# Information Model + +## Basic Graph Abstractions + +FABRIC Information Model library, containing class definitions and methods for operating +on different types of information model representations (sliver and slice). + +The implementation covers the following models and model transformations: +- ARM (Aggregate Resource Models) - generated by aggregate owner, may contain multiple resource delegations intended + for different brokers +- ADM (Aggregate Delegation Model) - a single ARM may be partitioned into one or more ADMs intended for different + brokers +- CBM (Combined Broker Model) - multiple ADMs sent to a broker from different aggregates are merged into a single CBM. +If needed ADMs can be 'unmerged' from a CBM (when e.g. an aggregate goes off-line) +- BQM (Broker Query Model) - multiple types of BQMs can be produced by a broker in response to `listResources`-like +queries from the information contained in CBM and live allocation information stored by the Broker. +The returned models can contain different levels of details of resources. BQMs can be used by user tools, + portals or other entities which need to inspect available testbed resources +- ASM (Abstract Slice Mode) - model generated by user tools, passed to the Orchestrator, which describes a slice + topology. This model can be annotated by Orchestrator and Aggregate Managers with details of the provisioned (vs. + requested) infrastructure and returned back to the user. ASMs encompass what is commonly referred to as Slice Request + and Slice Manifest. + +These models are based on a common framework of Abstract Property Graph class `fim.graph.abc_property_graph.py`, which are +then subclassed into a NetworkX-based implementation and a Neo4j-based implementation of abstract functionalities. +Further types of models are built on top of these two implementations, thus allowing for easy interchange of the +underlying graph implementation - in-memory (via NetworkX) or persistent (via Neo4j). + +There is a generic graph validation framework built-in - validation is stronger/more thorough when operating on Neo4j +implementations due to stronger expressivity of Neo4j Cypher query language. Graph validation rules can be found under +`fim.graph.data`. + +ARM, ADM, CBM definitions and implementations can be found under `fim.graph.resources`, ASM - under `fim.graph.slices`. +Abstract definitions of BQM are found under `fim.graph.resources`, however multiple subtypes of BQM (with different +levels of topology details presented to the requestor). Generation of different types of BQMs is done via broker plugins +intended to be external to FIM (perhaps inside the Control Framework). See `fim.pluggable.py` for more details on +plugins. + +## Higher-level Abstractions + +On top of basic property graph models described above, FIM offers additional more convenient abstractions that +interchangeably use the property graph implementations (NetworkX or Neo4j) underneath to allow for easier manipulation, +inspection of data. + +The main set of abstractions is implemented under `fim.user` package, which provides +[additional documentation](fim/user/README.md). + +Supporting intermediate models are implemented under `fim.sliver`, however these models are largely internal to FIM and +Control Framework, and generally are not exposed to end-users. + +## Structure of the code + +This figure reflects the overall structure of the code: + + + +Follow this link into [fim/README.md](fim/README.md) to explore. + +## Development environment + +The recommended way is to set up your development environment using `virtualenvwrapper` after checking +out the code: +```bash +$ git clone git@github.com:fabric-testbed/InformationModel.git +$ cd InformationModel +$ mkvirtualenv -r requirements.txt infomodel +$ workon infomodel +(infomodel) $ +``` + +You can also use the built-in `venv` tool that comes with newer versions of Python as follows: +``` +$ git clone git@github.com:fabric-testbed/InformationModel.git +$ cd InformationModel +$ python3 -m venv venv +$ source ./venv/bin/activate +$ pip install --upgrade pip setuptools +(venv) $ +``` + +Depending on which parts of FIM you are developing you may need to have +[Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc) running. +Working on any models whose names start with `Neo4j` generally requires using the Neo4j Docker. + +Models which start with `NetworkX` operate on in-memory models using NetworkX toolkit and don't require the Neo4j Docker. +Higher-level abstractions under `fim.user` can all be debugged and tested with in-memory NetworkX models, although they +operate on both types of models, depending on the situation. + +Follow the instructions with the container to start it. + +## Installation + +Multiple installation options possible. For CF development the recommended method is to +[install from GitHub MASTER branch](https://codeinthehole.com/tips/using-pip-and-requirementstxt-to-install-from-the-head-of-a-github-branch/): +```bash +$ pip install git+https://github.com/fabric-testbed/InformationModel.git +``` + +For developing and testing the FIM code itself use editable install (from top-level directory) and +install build, twine and pytest: +```bash +(infomodel) $ pip install -e . +(infomodel) $ pip install build +(infomodel) $ pip install pytest +(infomodel) $ pip install twine +``` + +For inclusion in tools, etc, use PyPi +```bash +$ pip install fabric-fim +``` + +### Compatibility with Neo4j + +Below is the compatibility matrix showing which major versions of FIM support which versions of Neo4j +(tested with the corresponding versions of [Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc)) + +| FIM Major version | Neo4j-APOC docker version | +|--------------------|---------------------------| + | 1.0-1.2 | 4.0.3 | + | 1.3-1.4 | 4.1.6 | + | 1.5 | 5.3.0 | + +## Test Graphs + +### Original Test Graphs + +Original test graphs are located under [tests/models](tests/models). All graphs were created using +[yEd](https://www.yworks.com/products/yed) +desktop graph editor (note that on-line version does not provide the same flexibility for creating custom node +and link properties). + +### Generating New Test Graphs + +Depending on the type of graph, many test graphs can be generated using higher-level abstraction libraries. + +For example ASMs can be generated starting from `fim.user.topology.ExperimentTopology` object, while ARMs can be +generated using `fim.user.topology.SubstrateTopology`. Other models are generally derived from these types of models, +as described in the overview. + +Test graphs are generated by unit tests: +- ASMs are generated in `test/slice_topology_test.py` +- ARMs are generated in `test/substrate_topology_test.py` +- ARM-ADM-CBM operations are tested in `test/zz_neo4j_pg_test.py` based on serializations produced in + `test/substrate_topology_test.py`. + +## Testing and building + +In order to run tests, you will need to use [pytest](https://pypi.org/project/pytest/). +Also, in order to support tests of Neo4j-implemented models, Neo4j-APOC should be running in a Docker container: + +```console +$ docker run -p7474:7474 -p7687:7687 -e NEO4J_AUTH=neo4j/password \ + -v $(pwd)/neo4j/data:/data -v $(pwd)/neo4j/imports:/imports \ + fabrictestbed/neo4j-apoc +``` + +Note that the directories `neo4j/data` and `neo4j/imports` must exist for the above command to work as expected. + +Wait for neo4j-apoc Docker container to start up, and then run pytest: + +```console +$ pytest [-s] test +``` + +This will produce substrate ARM models and save them into file in project root folder. + +## Graph validation + +All graphs loaded into Neo4j (whether from files or being passed in as part of query or delegation) +must conform to a set of rules expressed as Cypher queries. The basic set of rules for all +types of graphs are located in [fim/graph/graph_validation_rules.json](fim/graph/data/graph_validation_rules.json). + +Prior to ingestion graphs are also tested on general syntax validity of JSON-formatted fields. + +Additional rule files specific to model types may govern the validity of specific models. + +NetworkX-based graphs also undergo validation, but more limited in scope due to lack of tools. + +## Using fim_util.py utility + +The utility supports a number of operations on GraphML files - enumerating nodes (for graphs) +coming out of yEd, loading into an instance of Neo4j, deleting graphs from Neo4j. + +Start the [Neo4j-APOC docker container](https://github.com/fabric-testbed/fabric-docker-images/tree/master/neo4j-apoc) + +Create fim_config.yaml with the following structure under `util/`: +``` +# default fim_config configuration file for FIM utilities +neo4j: + url: neo4j://0.0.0.0:7687 + user: neo4j + pass: password + import_host_dir: /host/directory/seen/by/neo4j/docker/as/imports + import_dir: /imports +``` +Parameters `password` and `import_host_dir` depend on how the Docker container is started in +the procedure above. Other parameters should remain unchanged from what is shown. + +Run the utility for detailed help for the various operations: +``` +(infomodel) $ python fim_util.py -h +``` + +Generally the utility is good for e.g. loading a graph file: `python fim_util.py -l -f -r <graphml file>` or +merging multiple advertisements: `python fim_util.py -m -f <file1> -f <file2>`. Most options take multiple `-f` +and related options so can e.g. load multiple files at once. + +When testing large graphs, note that by default Neo4j visualizes the first 300 nodes in a browser. If you +want to see a full graph, increase this limit (at neo4j prompt): +``` +:config initialNodeDisplay: 5000 +``` + +## Neo4j Performance Considerations +For performance reasons it is critical that every instance of Neo4j has appropriate indexes created. +Neo4j label `GraphNode` is hard-coded within FIM - every graph node has this label and another label is created +from the Class property and is meaningful to FIM. +This is done for performance reasons to make it easier to create indexes and query models using those indexes. + +The following indexes are required (indexes are created automatically by Neo4jGraphImporter whenever it is used): +``` +CREATE INDEX graphid FOR (n:GraphNode) ON (n.GraphID) +CREATE INDEX graphid_nodeid FOR (n:GraphNode) ON (n.GraphID, n.NodeID) +CREATE INDEX graphid_nodeid_type FOR (n:GraphNode) ON (n.GraphID, n.NodeID, n.Type) +CREATE INDEX graphid_type FOR (n:GraphNode) ON (n.GraphID, n.Type) +``` +Available indexes can be checked via console by using the `:schema` command. + +An index can be dropped using the following command (substitute appropriate index name): +``` +DROP INDEX graphid +``` + +See [additional documentation](https://neo4j.com/docs/cypher-manual/current/administration/indexes-for-search-performance/). + + +%prep +%autosetup -n fabric-fim-1.4.12 + +%build +%py3_build + +%install +%py3_install +install -d -m755 %{buildroot}/%{_pkgdocdir} +if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi +if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi +if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi +if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi +pushd %{buildroot} +if [ -d usr/lib ]; then + find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/lib64 ]; then + find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/bin ]; then + find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst +fi +if [ -d usr/sbin ]; then + find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst +fi +touch doclist.lst +if [ -d usr/share/man ]; then + find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst +fi +popd +mv %{buildroot}/filelist.lst . +mv %{buildroot}/doclist.lst . + +%files -n python3-fabric-fim -f filelist.lst +%dir %{python3_sitelib}/* + +%files help -f doclist.lst +%{_docdir}/* + +%changelog +* Wed May 10 2023 Python_Bot <Python_Bot@openeuler.org> - 1.4.12-1 +- Package Spec generated @@ -0,0 +1 @@ +d018653280c88d40aa6ddf0fc1a48f1d fabric_fim-1.4.12.tar.gz |