Configuration Schema¶
This is the schema of the configuration database, effectively the control plane of the SDP.
Design Principles¶
Uses a key-value store
Uses watches on a key or range of keys to monitor for any updates
Objects are represented as JSON
We will likely want to define schemas and validation eventually, but for the moment this will be by example
Scheduling Block¶
Path /sb/[sbi_id]
Dynamic state information of the scheduling block instance.
Contents:
{
"id": "sbi-mvp01-20200425-00000",
"max_length": 21600.0
"scan_types": [
{ "id": "science", ... },
{ "id": "calibration", ... }
]
"pb_realtime": [ "pb-mvp01-20200425-00000", ... ]
"pb_batch": [ ... ]
"pb_receive_addresses": "pb-mvp01-20200425-00000"
"current_scan_type": "science"
"status": "SCANNING"
"scan_id": 12345
}
When the scheduling block instance is being executed, the status
field is
set to the observation state (obsState
) of the subarray. When the scheduling
block is ended, status
is set to FINISHED
.
Processing Block¶
Path: /pb/[pb_id]
Static definition of processing block information.
Contents:
{
"id": "pb-mvp01-20200425-00000",
"sbi_id": "sbi-mvp01-20200425-00000",
"workflow": {
"type": "realtime",
"id": "vis_receive",
"version": "0.2.0"
}
"parameters": { ... }
}
There are two types of processing, real-time processing and batch (offline) processing. Real-time processing starts immediately, as it directly corresponds to an observation that is about to start. Batch processing will be inserted into a scheduling queue managed by the SDP, where it will typically be executed according to resource availability.
Valid types are realtime
and batch
. The workflow tag identifies the
workflow script version as well as the required underlying software (e.g.
execution engines, processing components). ...
stands for arbitrary
workflow-defined parameters.
Processing Block State¶
Path: /pb/[pb_id]/state
Dynamic state information of the processing block. If it does not exist, the processing block is still starting up.
Contents:
{
"resources_available": True
"status": "RUNNING",
"receive_addresses": [
{ "scan_type": "science", ... },
{ "scan_type": "calibration", ... },
]
}
Tracks the current state of the processing block. This covers both the SDP-internal state (as defined by the Execution Control Data Model) as well as information to publish via Tango for real-time workflows, such as the status and receive addresses (for ingest).
Processing Block Owner¶
Path: /pb/[pb_id]/owner
Identifies the process executing the workflow. Used for leader election/lock as well as a debugging aid.
Contents:
{
"command": [
"vis_receive.py",
"pb-mvp01-20200425-00000"
],
"hostname": "pb-mvp01-20200425-00000-workflow-2kxfz",
"pid": 1
}