How to use database api-package¶
To use the configdb-server-package it has to be added to your python project first. This can be done via the following poetry commands:
poetry source add itk-demo-sw_configdb-server https://gitlab.cern.ch/api/v4/projects/148643/packages/pypi/simple/
poetry add configdb-server
After that you are able to import the Database class which includes all the functionality needed to access the staging area and the backend database
Creating an instance of the database class¶
First an instance of the database class has to be created. For this you will need to give the staging area uri and the backend uri. In addition to this the type of the backend database has to be given. Currently on SQLAlchemy is supported, but this will be expanded in the future. Available backends can be found in the Backends class.
# Type of the backend. Available types are listed in the Backends class
backend = Backends.SQLALCHEMY
# URL and protocol for the backend database
# Here an in memory (temporary) database is chosen
# You can use an sqlite database with "sqlite:///backend.db" for example
backend_uri = "sqlite:///file:a.db?mode=memory&uri=true"
# URL and protocol for the staging area database
stage_uri = "sqlite:///file:b.db?mode=memory&uri=true"
# Initialize database object which is used to access databases
db = Database(backend, backend_uri, stage_uri)
The staging area¶
Currently the staging area is just an sqlite file with the same datamodel as the backend. It is used to edit runkeys before they are committed to the backend database. The path of the sqlite file must be given while initializing the Database object. The staging database and its tables are automatically created when creating an object of the Database class.
Initializing the database backend¶
If the backend database and its tables are not yet created you can do so by using the following function.
# Create the backend database and its tables (only necessary when using a new not initialized database)
db.init_backend()
Creating a runkey¶
There are multiple ways to create new runkeys in the staging area. - Manually by using the database-tools functions - Automatically from a list - Clone from the backend database - Import from the file system
Manual runkey creation¶
def create_new_runkey(db: Database):
# Create a new runkey node to node, starting with the root node
root_uuid = db.create_root(
"root_tag", [{"data": "root_node payload", "type": "root", "name": "root_1"}, {"data": "another_payload", "type": "root", "name": "root2"}]
)
# Adding two felix nodes to the root node
felix1_uuid = db.add_to_tree(
"felix",
True,
parents=[root_uuid],
payloads=[
{"data": "payload_felix01", "type": "felix", "name": "felix01"},
{"data": "additional_payload_felix01", "type": "felix", "name": "felix01_2"},
],
)
felix2_uuid = db.add_to_tree("felix", True, parents=[root_uuid], payloads=[{"data": "payload_felix02", "type": "felix", "name": "felix02"}])
# Adding three lpgbts to felix1 and one to felix2
lpgbt1_uuid = db.add_to_tree("felix", True, parents=[felix1_uuid], payloads=[{"data": "payload_lpgbt01", "type": "lpgbt", "name": "lpgbt01"}])
db.add_to_tree("lpgbt", True, parents=[felix1_uuid], payloads=[{"data": "payload_lpgbt02", "type": "lpgbt", "name": "lpgbt02"}])
db.add_to_tree("lpgbt", True, parents=[felix1_uuid], payloads=[{"data": "payload_lpgbt03", "type": "lpgbt", "name": "lpgbt03"}])
db.add_to_tree("lpgbt", True, parents=[felix2_uuid], payloads=[{"data": "payload_lpgbt04", "type": "lpgbt", "name": "lpgbt04"}])
# Adding two fes to lpgbt1
db.add_to_tree("frontend", True, parents=[lpgbt1_uuid], payloads=[{"data": "payload_fe01", "type": "frontend", "name": "fe01"}])
db.add_to_tree("frontend", True, parents=[lpgbt1_uuid], payloads=[{"data": "payload_fe02", "type": "frontend", "name": "fe02"}])
Creation from a list¶
def create_full_runkey(db: Database):
# Create full runkey in staging area from one nested list (e.g. imported from a json file)
# nested lists define children and dict entries in a list define a payload
# The following dict creates a root node with 2 payloads and 2 felix objects as children
data = {
# root node
"type": "root",
"payloads": [{"data": "root_node payload", "type": "config", "name": "test"}, {"data": "another_payload", "type": "root", "name": "test"}],
"children": [
# root node children (felix1 and felix2)
{
# felix1
"type": "felix",
"payloads": [
{"data": "payload_felix01", "type": "felix", "name": "test"},
{"data": "additional_payload_felix01", "type": "felix", "name": "test"},
],
"children": [
# felix1 children (lpgbt01, lpgbt02 and lpgbt03)
{
# lpgbt01
"type": "lpgbt",
"payloads": [{"data": "payload_lpgbt01", "type": "lpgbt", "name": "test"}],
"children": [
# lpgbt01 children (fe01 and fe02)
{
# fe01
"type": "frontend",
"payloads": [{"data": "payload_fe01", "type": "frontend", "name": "test"}],
},
{
# fe02
"type": "frontend",
"payloads": [{"data": "payload_fe02", "type": "frontend", "name": "test"}],
},
],
},
{
# lpgbt02
"type": "lpgbt",
"payloads": [{"data": "payload_lpgbt02", "type": "lpgbt", "name": "test"}],
},
{
# lpgbt03
"type": "lpgbt",
"payloads": [{"data": "payload_lpgbt03", "type": "lpgbt", "name": "test"}],
},
],
},
{
# felix2
"type": "felix",
"payloads": [{"data": "payload_felix02", "type": "felix", "name": "test"}],
"children": [
{
# lpgbt04
"type": "lpgbt",
"payloads": [{"data": "payload_lpgbt04", "type": "lpgbt", "name": "test"}],
},
],
},
],
}
db.create_full_tree(data, "root_tag")
Cloning¶
def clone_runkey(db: Database):
# In order to clone a runkey from the backend database first one has to exist there
# Here one of the other creation methods is used to achieve this
create_full_runkey(db, "root_tag")
db.commit_tree("root_tag", "already_existing_runkey", "my_name", "runkey")
# Now one can use this existing runkey to clone it in the staging area and edit it
db.clone_tag("already_existing_runkey", "root_tag")
Importing¶
def import_runkey(db: Database):
# Imports a runkey from a local directory, here the /config directory is used
# The current implementation of the import is based on the configuration format used in SR1
# It looks for a connectivity config which names the frontend config files
import_path = os.getcwd() + "/config/"
con_name = "connect_warm.json"
db.import_tree(import_path, con_name, "root_tag")
Editing a runkey¶
After creating a new runkey by any of these methods shown above one can add additional children and payloads. To do so you have to know the uuid of the object of the new object or payload should be associated with. If you want to add an additional frontend to the first lpgbt of the first felix you can do this like this:
current_tree = db.get_tag("root_tag", True, verbosity=3)["objects"][0]
felix_dict = current_tree["children"][0]
lpgbt_dict = felix_dict["children"][0]
lpgbt_uuid = lpgbt_dict["id"]
fe_uuid = db.add_to_tree("felix", True, parents=[lpgbt_uuid], payloads=[{"data": "payload for new frontend object", "type": "felix"}])
payload_uuid = db.stage.create_payload(data="payload for new frontend object", type="frontend", name="fe_new")
db.stage.add_to_object(fe_uuid, [], [payload_uuid])
Commiting a runkey¶
When the runkey is finished it should be committed to the backend database in order to be saved permanently. Committing the current staging area runkey will clear it.
Accessing a runkey¶
Runkeys from the backend database can be read and writen into a python dict like this:
They can also be exported to the file system in one of two ways. - Flat export into one directory: - Export into nested directories To print a tree on can use the format_tag method.