Analytics and external DBs
Introduction
If you want to perform advanced analytics or attach a dashboarding product (such as Grafana or Splunk) to visualize data generated by step, in the enterprise version of step we provide plugins that write the data to databases other than MongoDB.
At this moment, the following plugins are available:
- JDBC (SQL)
- Prometheus
- Logback
In addition to these 3 plugins, step contains also an RTM plugin using the RTM measurements accessor and an Abstract Measurements plugin, responsible for processing generated measurements and sending them to the relevant exposing plugin.
In all cases, the step user is responsible for setting up and maintaining the external database of choice. Below we provide details on required configuration on the step side.
Installation and usage
Plugins can be enabled by uncommenting relevant lines in step.properties
of step Controller (CONTROLLER_DIR\conf\step.properties
). By default, the RTM plugin is enabled, and other plugins have following configuration:
plugins.JdbcControllerPlugin.enabled=false
plugins.PrometheusControllerPlugin.enabled=true
plugins.LogMeasurementControllerPlugin.enabled=false
JDBC (SQL)
With the JDBC plugin step may write data in SQL-based databases, such as PostgreSQL or MySQL.
A note on exporting noSQL-like data into SQL databases. Since the keywords used in step can be highly customized, each measurement generated by step contains a set of required fields (in bold) and may contain arbitrary amount of extra fields:
{ begin: ... , eId: ... , name: ... , planId: ... , rnStatus: ... , taskId: ... , type: ... , value: ... , customField1: ... , customField2: ... , customField3: ... , customField4: ... , ... }
Since SQL databases are not designed for dynamic schema adjustments, the JDBC plugin deals with this by exposing the required fields as separate columns and additionally exposing the custom fields as a json and writing them into a json-type column. PostgreSQL and MySQL support json as valid types, but other SQL databases may not, therefore the option to expose json with the custom measurement data can be enabled or disabled separately.
The plugin has been tested with PostgreSQL 12 and MySQL 8.
Enabling the JDBC plugin
To use the plugin:
-
the user must add the JDBC driver library corresponding to its database to the
CONTROLLER_DIR\lib
folder -
e.g. for PostgreSQL, download the relevant
.jar
toCONTROLLER_DIR\lib
folder -
enable the plugin in
step.properties
by setting
plugins.JdbcControllerPlugin.enabled=true
- configure the database url, user and password in
step.properties
, for example:
plugins.measurements.jdbc.url=jdbc:postgresql://localhost/step
plugins.measurements.jdbc.user=postgres
plugins.measurements.jdbc.password=test
Providing and configuration the database with proper indexing is the user responsibility, the table schema must be respected.
- optionally: enable writing of the custom fields in a json column:
plugins.measurements.jdbc.jsonColumn=true