itk-demo-registry microservice¶

Slides on the itk-demo-registry HTTP interface to the service registry can be found here.

Service Registry¶

The services registry is build on a distributed KV data store (etcd) and enables service discovery and synchronisation.

The services registry mainly consists of three parts:

The registrator which constantly looks for changes on the Docker socket. It reads the currently available services from Docker labels on the running containers and writes this information into the registry. It also constructs a set of key-value pairs based on information in the environment and in Docker inspect, like the url under which the ressources in the container are reachable.
The registry module which is used to put and get key-value pairs into the backend KV-store. Mostly the registrator puts keys, and the services retrieve keys to find related services.
The distributed KV-store which makes the key-value database available to all machines on the site. Currently we use etcd as KV-store backend ¹.

The sequence diagram explains how microservices are decoupled by way of the service registry:

The services are configured via environment variables
Docker labels are set based on environment variables. The format is:

Text Only

demi.{site}.{context}.{stack}.{service}[.key]=[value]

The registrator scans all Docker containers for labels starting with the root key (demi) and loads them.
Based on the system settings, environment and the Docker inspect JSON it translates the labels in to keys populated with additional information, eg:

Text Only

demi/{site}/{context}/{stack}/{service}/url=http://{host}:{port}

²

Automatically populated keys may be overridden or supplemented by manually supplied Docker labels. The latter is neccessary if the required information cannot be deduced automatically, for example the URL of GUIs from third-party tools.
Based on the keys provided via labels at start-up time, client services can then look up information about the ressource services in the registry.

sequenceDiagram
    Environment->>A: $A_LABEL=demi.label.A
    Environment->>B: $A_URL=demi.key.A.url   
    Registrator->>A: demi labels?
    Registrator->>B: demi labels?
    A-->>Registrator: demi.label.A
    B-->>Registrator: demi.label.B
    Note over Registrator: constructs set of standard kv-pairs
    Registrator->>Registry: demi/key/A/url
    Note over Environment, Registry: time passes ... at some point B needs to call A
    B->>Registry: demi/key/A/url?
    Registry-->>B: demi/key/A/url

How to implement¶

To ensure the service registry copies the docker labels into the key-value store, a specific format is required by the service registry. If at least one label with the following format exists, the docker container will be registered:

demi.{...}.{SERVICE}.{...}

Where SERVICE corresponds to the service defined in the docker-compose.yml.

The full naming convention is (used by eg. the Dashboard):

demi.{SITE}.{CONTEXT}.{STACK}.{SERVICE}.{KEY}[.{SUBKEY(S)}]={VALUE}

We call the part before the {KEY} the NAMESPACE of the service.

An example of a correct label:

demi.cern.wupp-charon.itk-demo-configdb-01.api.port=5000

The label can have an empty value if it is just there to signal the presence of the container to the service registry.

This format corresponds to the official Docker label format recommendations (even if DeMi does not have it's own domain just yet so there is no inverse-domain prefix).

While Docker labels are separated with dots, these will be converted to slashes when creating the KV-pair for the backend KV-database, eg.:

demi/cern/wupp-charon/itk-demo-configdb-01/api/port=5000

Keys¶

In order for a microservice to work with the dashboard some keys are required (tbc):

key	required?	label	value
category	yes	demi.{...}.type	Microservice, Microservice-UI, Tool
icon	no	demi.{...}.icon	path or url to an icon for the service
description	no	demi.{...}.description	a short description of the service
health	no	demi.{...}.health	health endpoint of the service

eg.:

type	value
Label	`demi.cern.wupp-charon.itk-demo-configdb-01.api.type=Microservice`
KV-pair	`demi/cern/wupp-charon/itk-demo-configdb-01/api/type=Microservice`

Additional keys¶

Additional labels can be set to enable the service registry to create a URL key that provides access to the microservice or GUI. If some of these labels are not set the service registry will try to detect the host and port automatically. The URL prefix will be omitted.

key	label	value
host	demi.{...}.host	wupp-charon.cern.ch
hostport	demi.{...}.hostport	5000
url_prefix	demi.{...}.url_prefix	/api
protocol	demi.{...}.protocol	http

Resulting URL created by the service registry:

Text Only

http://wupp-charon.cern.ch:5000/api

Connecting GUI and backend¶

The GUI requires the backend URL in order to connect to its backend. The backend URL will be retrieved from the service registry using the API_URL_KEY environment variable.

API_URL_KEY=demi.cern.wupp-charon.itk-demo-configdb-01.api.url

The starup-time configuration of the UI is currently implemented with nginx. The nginx-config template

Text Only

Using envsubst, one can insert the values of environment variables into this config at startup-time. needs to be provided with a /config endpoint. This endpoint should have the following format: id="__span-3-1">http { ... server { ... location /config { default_type application/json; return 200 '{ "status": 200, "etcdHost":"${ETCD_HOST}", "etcdPort":"${ETCD_PORT}", "etcdNamespace":"${ETCD_NAMESPACE}", "urlKey":"${API_URL_KEY}", "backendUrl":"${BACKEND_URL}" }'; } } } The backendUrl is an optional fallback-url that is used if no connection to the service registry can be established.

In the UI, the backend URL can be retrieved using the useConfigureValue hook:

Text Only

import { useConfigureValue } from '@itk-demo-sw/hooks'
...
const backendUrl = useConfigureValue('/config', 'urlKey', 'http://default_url:1234/api', 'backendUrl')

The hook returns a state-variable that automatically updates whenever the urlKey is changed in the service registry. The first two arguments of the useConfigureValue hook are self-explanatory. The third argument is an optional default value and the fourth provides the aforementioned optional fallback-key.

The /config endpoint can be used for any kind of startup-time configuration one wants to provide to the react ui.

Connecting to other microservices¶

Microservices can watch the URL of other microservices in the same way.

CONFIGDB_KEY=demi.cern.wupp-charon.itk-demo-configdb-01.api.url

To watch an URL in Python, you can use the following code snippet:

Python

import os
from service_registry import ServiceRegistry

CONFIGDB_KEY = os.environ.get("CONFIGDB_KEY")

def start_watch():
    registry = ServiceRegistry(namespace="")
    cancel = registry.watch(CONFIGDB_KEY, callback_function)
    return cancel

def callback_function(event_type, key, value):
    if event_type == "PUT":
       #Code to execute when watched key is created
    elif event_type == "DELETE":
       #Code to execute when watched key is deleted

The function given as the callback_function must have event_type, key, and value as parameters. These can then be used to execute code based on the type of the etcd-event. Note that the callback function will be executed in a separate thread and will therefor have no access to the current flask-app.

The returned value from the watch function can be used to cancel the watcher.

Circuit Breaker¶

The circuit breaker allows microservices to communicate as usual and monitor the number of failures occurring during a process.At his point the circuit breaker is still closed. If the failure count exceeds 3, the circuit breaker will move to the Open state. If not, it will reset the failure count and timeout period of 6 seconds. For the use of the circuit breaker, a decorator has to be infront of the function, which should be monitored by the circuit breaker. In this case for example: @cbreaker def get_greeting(url, port): ''' Get the main index page response ''' response = requests.get("http://{}:{}/".format(url, port)) return response.text

later we may replace this by IS, OKS or whatever central TDAQ facility is required. or by a more lightweight solution like DBM if etcd turns out to be overkill. ↩
note the different separator in label (dotted) and KV-stored (slash) to conform to the respective standard. ↩