Documentation update: architecture, backendapi

docs/Makefile

@@ -3,7 +3,7 @@
 
 # You can set these variables from the command line.
 SPHINXOPTS    =
-SPHINXBUILD   = sphinx-build
+SPHINXBUILD   = /bin/sphinx-build
 PAPER         =
 BUILDDIR      = _build
 

docs/architecture.rst

@@ -4,5 +4,120 @@
 Architecture
 ************
 
+Jetconf is an implementation of the RESTCONF_ protocol for remote
+management of network devices and services.
 
+`YANG 1.1`_  data modelling language is also fully supported.
 
+JetConf is written in Python 3 language and available as open source
+software under the terms of the `GNU GPLv3`_ license.
+
+Requirements and Restrictions
+=============================
+
+JetConf is a compliant RESTCONF_ implementation supporting all mandatory features.
+
+Although it is written in Python, it should be fast enough to support
+large configuration databases with moderate rate of changes. A typical
+use can may be an authoritative TLD name server in which Jetconf
+covers both server management and domain provisioning.
+
+Jetconf supports only the **JSON** data encoding, i.e. media types with
+the ``+json`` structured syntax suffix, such as ``application/yang.data+json``.
+
+Jetconf supports only `HTTP/2`_ transport. Entity tags (ETag headers) can
+be generated for all data resources, whereas timestamps (Last-Modified
+headers) are supported for all container-like resources, i.e. not for
+individual leaf and leaf-list instances.
+
+Datastore
+=========
+
+Jetconf uses YANGSON_ library, which is responsible for *storage*,
+*validation* and *manipulation* with YANG data. This library utilizes an
+in-memory persistent structure called *"Zipper"* where the YANG data
+are kept in.
+
+Jetconf also provides an option to serialize data into ``.json``
+file on each commit, which ensures that all configuration data will
+be persistent among server startups.
+
+Additionally, the datastore can have an access control module
+associated with it. If so, every read/write operation will be verified
+with this ACM.
+
+Access Control
+==============
+
+The current version of Jetconf implements NACM_ access control
+system, which enables to specify fine-grained access permissions to
+particular data resources.
+
+The NACM_ data can only be edited by privileged users in startup configuration.
+
+Jetconf Server Loop
+===================
+
+#. The client opens a secure TLS connection.
+
+
+#. The client is authenticated via a client certificate. The
+   certificate of the CA that issued the client certificate needs to
+   be specified in the configuration file. The *e-mail* or *commonName* field obtained
+   from the client certificate is henceforth used as the username,
+   in particular for access control. If the client cannot be
+   authenticated, for example because his certificate has expired or
+   because it was not issued by correct CA, the connection is terminated.
+
+#. The server waits for an incoming client request.
+
+#. A received request is parsed and handed over to the appropriate
+   component. If the media type specified is not supported (in
+   particular, is not ``+json``), ``415 Unsupported Media Type`` is sent,
+   If the message is otherwise invalid, ``400 Bad Request`` is sent.
+
+#. The NACM data is queried to determine which groups the user is a
+   member of.
+
+#. Depending on the type of the request (read, write or RPC operation
+   invocation) and the Request-URI, the required permissions are
+   determined, and the NACM database is checked to verify that the
+   user possess all of them. If not, ``403 Forbidden`` is sent.
+
+#. If the request is an RPC operation, it is invoked and an
+   appropriate reply or error message generated.
+
+#. If the request is a read operation, the corresponding data are retrieved
+   from the datastore and formatted into a reply, or an error status
+   code is returned.
+
+#. If the request is a write operation, the changes are applied using
+   a persistent structure API (so that the original unmodified
+   configuration remains available). The new configuration is passed to
+   the *YANGSON* library for validation. If the validation succeeds, the
+   new configuration is written to non-volatile memory, and passed to
+   server instrumentation that applies the necessary changes. An
+   appropriate response or error code is generated and sent.
+
+#. After finishing one of the steps 7, 8 or 9, the server returns to step 3.
+
+Python Modules
+==============
+
+* ``rest_server``: a module providing the HTTP/2 and user authentication
+  functionality for REST operations,
+* ``http_handlers``: handlers connecting HTTP requests to datastore
+  operations,
+* ``data``: datastore implementation,
+* ``nacm``: basic NACM implementation,
+* ``config``: a module for reading and parsing the config file,
+* ``helpers``: static helper classes shared across modules,
+* ``op_internal``: implementation of Jetconf internal RPCs,
+* ``errors``: definition of exceptions used in Jetconf.
+
+.. _YANG 1.1: https://tools.ietf.org/html/rfc7950
+.. _RESTCONF: https://tools.ietf.org/html/rfc8040
+.. _GNU GPLv3: https://www.gnu.org/licenses/gpl.html
+.. _HTTP/2: https://tools.ietf.org/html/rfc7540
+.. _YANGSON: https://github.com/CZ-NIC/yangson
+.. _NACM: https://tools.ietf.org/html/rfc8341
\ No newline at end of file

docs/backendapi.rst

@@ -2,4 +2,420 @@
 
 ***********
 Backend API
-***********
\ No newline at end of file
+***********
+
+As there can be various use-case scenarios for Jetconf, bindings to a user application
+are not part of Jetconf server itself, but instead they are implemented in a separate package,
+so called *"Jetconf backend"*.
+
+The basic idea of Jetconf's backend architecture is that every node of the YANG schema
+(i.e. container, list, leaf-list) can have a custom handler object assigned to it.
+When a specific event affecting this node occurs , like configuration data being rewritten
+or RESCONF operation is called, an appropriate member function of this node handler is invoked.
+
+As there are some major differences between YANG configuration data, state data and RPCs,
+the architecture of corresponding node handlers in Jetconf also has to follow these differences.
+
+Backend package architecture
+============================
+
+Every backend package for Jetconf server has to provide implementation of following modules.
+
+- ``usr_conf_data_handlers`` (Handlers for configuration data)
+- ``usr_state_data_handlers`` (Handlers for state data)
+- ``usr_op_handlers`` (Handlers for RESTCONF operations - RPCs)
+- ``us_action_handlers`` (Handlers for RESTCONF actions - operation on node)
+- ``usr_datastore`` (Datastore initialization and save/load functions can be customized here)
+- ``usr_init`` (Jetconf initialization)
+
+In addition to this, backend package can also contain any other resources if necessary.
+When you consider writing a custom backend, looking at the very basic demo package
+jetconf_jukebox_ is a good way to start.
+
+usr_init
+========
+
+usr_datastore
+=============
+
+Basic ``usr_datastore`` module without any customization.
+
+.. code-block:: python
+
+    from jetconf.data import JsonDatastore
+
+    class UserDatastore(JsonDatastore):
+        pass
+
+
+Customizing ``load()`` and ``save()`` functions
+
+.. code-block:: python
+
+    from jetconf.data import JsonDatastore
+
+
+    class UserDatastore(JsonDatastore):
+
+        def load(self):
+
+            # load method can be customized here
+
+        def save(self):
+
+            # save method can be customized here
+
+
+usr_conf_data_handlers
+======================
+
+The main purpose of configuration data handlers is to project all changes performed on a
+particular data node, like creation, modification or deletion, to the user application.
+
+A configuration node handler is implemented by creating a custom class which inherits
+from either ``ConfDataObjectHandler`` or ``ConfDataListHandler`` base class depending on
+the type of YANG node. The former must be used when implementing a handler for ``Container``
+or ``Leaf`` data nodes, while the latter is used for list-like types, specifically ``List``
+and ``Leaf-List``.
+
+ConfDataObjectHandler:
+^^^^^^^^^^^^^^^^^^^^^^
+
+**Attributes**:
+
+.. code-block:: python
+
+    self.ds             # type: jetconf.data.BaseDatastore
+                        # Can be used for accessing the datastore content from handler functions
+
+    self.schema_path    # type: str
+                        # Contains the YANG schema path to which this handler object is registered (as string)
+
+    self.schema_node    # type: yangson.schemanode.SchemaNode
+                        # Contains the YANG schema path to which this handler object is registered (parsed)
+
+**Arguments**:
+
+.. code-block:: python
+
+    ii:         # type: yangson.instance.InstanceRoute
+                # Contains parsed instance identifier of the data node. Useful for determining list keys if this data node is a child of some list node.
+    ch:         # type: jetconf.data.DataChange
+                # Can be used for accessing additional edit information, like HTTP input data if needed
+
+Handlers derived from this base class has to implement the following interface:
+
+.. code-block:: python
+
+    from jetconf.handler_base import ConfDataObjectHandler
+    from yangson.instance import InstanceRoute
+    from jetconf.data import BaseDatastore, DataChange
+
+
+    class MyConfDataHandler(ConfDataObjectHandler):
+        def create(self, ii: InstanceRoute, ch: DataChange):
+
+            # Called when a new node is created
+
+        def replace(self, ii: InstanceRoute, ch: DataChange):
+
+            # Called when the node is being rewritten by new data
+
+        def delete(self, ii: InstanceRoute, ch: DataChange):
+
+            # Called when the node is deleted
+
+ConfDataListHandler:
+^^^^^^^^^^^^^^^^^^^^
+
+**Attributes**:
+
+.. code-block:: python
+
+    self.ds             # type: jetconf.data.BaseDatastore
+                        # Can be used for accessing the datastore content from handler functions
+
+    self.schema_path    # type: str
+                        # Contains the YANG schema path to which this handler object is registered (as string)
+
+    self.schema_node    # type: yangson.schemanode.SchemaNode
+                        # Contains the YANG schema path to which this handler object is registered (parsed)
+
+**Arguments**:
+
+.. code-block:: python
+
+    ii:     # type: yangson.instance.InstanceRoute
+            # Contains parsed instance identifier of the data node. Useful for determining list keys if this data node is a child of some list node.
+
+    ch:     # type: jetconf.data.DataChange
+            # Can be used for accessing additional edit information, like HTTP input data if needed
+
+Handlers derived from this base class has to implement the following interface:
+
+.. code-block:: python
+
+    from jetconf.handler_base import ConfDataListHandler
+    from yangson.instance import InstanceRoute
+    from jetconf.data import BaseDatastore, DataChange
+
+
+    class MyConfDataHandler(ConfDataListHandler):
+        def create_item(self, ii: InstanceRoute, ch: DataChange):
+
+            # Called when a new item is added to the list or leaf-list
+
+        def replace_item(self, ii: InstanceRoute, ch: DataChange):
+
+            # Called when specific list item is being rewritten
+
+        def delete_item(self, ii: InstanceRoute, ch: DataChange):
+
+            # Called when an item is being deleted from the list
+
+Handler registration
+^^^^^^^^^^^^^^^^^^^^
+
+Assignation of handler objects to the specific data nodes is done via registering them in
+``jetconf.handler_list.CONF_DATA_HANDLES`` handler list. Every ``usr_conf_data_handlers``
+backend module must implement the global function ``register_conf_handlers``,
+where the instantiation and registration of handler objects is done. This function is
+called on Jetconf startup after datastore initialization and has the following signature.
+
+.. code-block:: python
+
+    def register_conf_handlers(ds: BaseDatastore):
+
+        ds.handlers.conf.register(MyConfHandler(ds, "/ns:schema-path/to-desired-node"))
+
+
+usr_state_data_handlers
+=======================
+
+YANG state data, in contrast to the configuration data, represents more of a current
+state of the backend application. This means that they are not actually stored in
+Jetconf's datastore, but instead they has to be generated on the go. Generation of
+state data is the purpose of state data handlers.
+
+A state data handler has to acquire actual state data from backend application and generate data
+content of the node where it's assigned. The output data are formatted in Python's representation
+of *JSON* (using lists, dicts etc.) and their structure must be compliant with the standardized
+JSON encoding of YANG data (see RFC7951_).
+
+A state node handler is implemented by creating a custom class which inherits from either
+``StateDataContainerHandler`` or ``StateDataListHandler``, depending on the YANG node type.
+This is similar to he configuration data handlers.
+
+StateDataContainerHandler
+^^^^^^^^^^^^^^^^^^^^^^^^^
+**Attributes**:
+
+.. code-block:: python
+
+    self.ds             # type: jetconf.data.BaseDatastore
+                        # Can be used for accessing the datastore content from handler functions
+
+    self.data_model     # type: yangson.datamodel.DataModel
+                        # Reference to the current data model object
+
+    self.sch_pth        # type: str
+                        # YANG schema path to which this handler object is registered (as string)
+
+    self.schema_node    # type: yangson.schemanode.DataNode
+                        # Reference to the Yangson schema node object
+
+
+
+.. code-block:: python
+
+    from yangson.instance import InstanceRoute
+    from jetconf.handler_base import StateDataContainerHandler
+    from jetconf.data import BaseDatastore
+
+    class MyStateDataHandler(StateDataContainerHandler):
+        def generate_node(self, node_ii: InstanceRoute, username: str, staging: bool)
+
+            # This method has to generate content of the state data node
+
+            return generated_content
+
+
+StateDataListHandler
+^^^^^^^^^^^^^^^^^^^^
+
+**Attributes**:
+
+.. code-block:: python
+
+    self.ds             # type: jetconf.data.BaseDatastore
+                        # Can be used for accessing the datastore content from handler functions
+
+    self.data_model     # type: yangson.datamodel.DataModel
+                        # Reference to the current data model object
+
+    self.sch_pth        # type: str
+                        # YANG schema path to which this handler object is registered (as string)
+
+    self.schema_node    # type: yangson.schemanode.DataNode
+                        # Reference to the Yangson schema node object
+
+**Methods**:
+
+.. code-block:: python
+
+    from yangson.instance import InstanceRoute
+    from jetconf.helpers import JsonNodeT
+    from jetconf.handler_base import StateDataListHandler
+    from jetconf.data import BaseDatastore
+
+    class MyStateDataHandler(StateDataListHandler):
+        def generate_list(self, node_ii: InstanceRoute, username: str, staging: bool) -> JsonNodeT:
+
+            # This method has to generate entire list
+
+            return generated_content
+
+        def generate_list(self, node_ii: InstanceRoute, username: str, staging: bool) -> JsonNodeT:
+
+            # Generates only one specific item of the list. The list key(s) of the item which needs to be generated can be resolved by processing the instance identifier passed in 'node_ii' argument.
+
+            return generated_content
+
+Handler registration
+^^^^^^^^^^^^^^^^^^^^
+
+Assignation of state data handler objects to the specific data nodes is done via registering
+them in ``jetconf.handler_list.STATE_DATA_HANDLERS`` handler list. This is similar to the configuration data.
+Every ``usr_state_data_handlers`` backend module must implement the global function ``register_state_handlers``,
+where the instantiation and registration of handler objects is done. This function is called on Jetconf
+startup after datastore initialization and has the following signature:
+
+.. code-block:: python
+
+    def register_state_handlers(ds: BaseDatastore):
+
+        ds.handlers.state.register(MyStateDataHandler(ds, "/ns:schema-path/to/state/node"))
+
+
+usr_op_handlers
+===============
+Handlers for RESTCONF operations.
+
+**Arguments**:
+
+.. code-block:: python
+
+     input_args:        # type: JSON
+                        # Operation input arguments with structure defined by YANG model
+
+     username:          # type: jetconf.data.BaseDatastore
+                        # Name of the user who invoked the operation
+
+An operation handlers are implemented by adding a custom method to
+the class ``OpHandlersContainer``. Finally, this class is instantiated and its methods are assigned
+to specific operation names.
+
+.. code-block:: python
+
+    from yangson.instance import InstanceRoute
+    from jetconf.helpers import JsonNodeT
+    from jetconf.data import BaseDatastore
+
+    class OpHandlersContainer:
+        def __init__(self, ds: BaseDatastore):
+            self.ds = ds
+
+        def my_op_handler(self, input_args: JsonNodeT, username: str) -> JsonNodeT:
+
+            # RPC operation Body
+
+            # Operation output data as defined by YANG data model
+            # output is not mandatory
+            return output_data
+
+Handler registration
+^^^^^^^^^^^^^^^^^^^^
+
+Every ``usr_op_handlers`` backend module must implement the global function ``register_op_handlers``,
+where the class ``OpHandlersContainer`` is instantiated and its methods are tied to individual
+operations. This function with following signature is called on Jetconf startup after datastore
+initialization.
+
+.. code-block:: python
+
+    def register_op_handlers(ds: BaseDatastore):
+
+        op_handlers_obj = OpHandlersContainer(ds)
+        ds.handlers.op.register(op_handlers_obj.my_op_handler, "ns:operation")
+
+
+
+us_action_handlers
+==================
+Handlers for RESTCONF actions.
+
+**Arguments**:
+
+.. code-block:: python
+
+     ii:     # type: yangson.instance.InstanceRoute
+            # Contains parsed instance identifier of the data node. Useful for determining list keys if this data node is a child of some list node.
+
+     input_args:        # type: JSON
+                        # Operation input arguments with structure defined by YANG model
+
+     username:          # type: jetconf.data.BaseDatastore
+                        # Name of the user who invoked the operation
+
+
+An action handlers are implemented by adding a custom method to
+the class ``ActionHandlersContainer``. Finally, this class is instantiated and its methods are assigned
+to specific action names and node path.
+
+.. code-block:: python
+
+    from yangson.instance import InstanceRoute
+    from jetconf.helpers import JsonNodeT
+    from jetconf.data import BaseDatastore
+
+    class ActionHandlersContainer:
+        def __init__(self, ds: BaseDatastore):
+            self.ds = ds
+
+        def my_action_handler(self, ii: InstanceRoute, input_args: JsonNodeT, username: str) -> JsonNodeT:
+
+            # Action Body
+
+            # Action output data as defined by YANG data model
+            # output is not mandatory
+            return output_data
+
+
+Handler registration
+^^^^^^^^^^^^^^^^^^^^
+Every ``usr_action_handlers`` backend module must implement the global function ``register_action_handlers``,
+where the class ``ActionHandlersContainer`` is instantiated and its methods are tied to individual
+actions. This function with following signature is called on Jetconf startup after datastore
+initialization.
+
+.. code-block:: python
+
+    def register_action_handlers(ds: BaseDatastore):
+        act_handlers_obj = ActionHandlersContainer(ds)
+        ds.handlers.action.register(act_handlers_obj.my_action_handler, "/ns:schema-path/to/action/node")
+
+
+Handler inheritance
+===================
+Because some data models can be quite large, it would be difficult to manually assign
+handler objects to all schema nodes. Because of this, for configuration and state data handlers,
+Jetconf offers a feature called **Handler inheritance**.
+
+If a node without its own handler is edited, Jetconf finds a nearest
+parent node which has the handler assigned and then it calls its ``replace`` or ``replace_item``
+method. It's up to backend developer's decision where to place handler objects, a more fine-grained
+placement will usually mean better performance (less data rewriting), at the cost of more work.
+
+
+.. _jetconf_jukebox: https://gitlab.labs.nic.cz/jetconf/jetconf-jukebox
+.. _RFC7951: https://tools.ietf.org/html/rfc7951
\ No newline at end of file

docs/backends.rst

-.. _backends:
-
-********
-Backends
-********
-
-==============
-Jukebox (demo)
-==============
-
-=======
-KnotDNS
-=======
\ No newline at end of file

docs/clients.rst

-.. _clients:
-
-***************
-Clients
-***************
-
-====
-curl
-====
-
-
-=========
-JetScreen
-=========
\ No newline at end of file

docs/developers.rst

+.. _developers:
+
+**************
+For Developers
+**************
\ No newline at end of file

docs/features.rst

-.. _features:
-
-********
-Features
-********
-
-curl --http2-prior-knowledge -H "X-SSL-Client-CN: ales.mrazek@nic.cz" -X GET http://localhost:8443/.well-known/host-meta
-
-curl --http2-prior-knowledge -H "X-SSL-Client-CN: ales.mrazek@nic.cz" -X GET http://localhost:8443/restconf/data
\ No newline at end of file

docs/index.rst

@@ -10,7 +10,6 @@ The `RESTCONF <https://tools.ietf.org/html/rfc8040>`_ protocol implementation wr
 
    startguide
    certificates
-   developers
 
 .. toctree::
    :maxdepth: 2
@@ -19,6 +18,7 @@ The `RESTCONF <https://tools.ietf.org/html/rfc8040>`_ protocol implementation wr
    architecture
    configuration
    backendapi
+   developers
 
 .. toctree::
    :maxdepth: 1