Using step with Grafana
Intro
This tutorial will walk you through the steps needed to connect Grafana to the data generated by step. At the end of the tutorial, you will have:
- configured Grafana to connect to data generated by step in a database of your choice
- imported a performance dashboard
- visualized step data in Grafana
Why Grafana
Grafana is a well-known open-source dashboarding product with versatile functionality, broad community and vast ecosystem. Moreover, it is frequently updated and very devops friendly. It is especially well-suited for analyzing time-series data, exactly such as the data generated by step. We believe that enabling users to access step data in Grafana alongisde other metrics that they have already available will allow users to generate much more value out of their data.
Grafana relies on “data source plugins” to ingest data from various databases and applications. While there is no “step data source” plugin, step users can connect Grafana to the databases that contain step measurement data. In the open-source step measurements are stored in MongoDB, and in step Enterprise there is a possibility to use a plugin to concurrently store measurements in a different database.
Grafana dashboard is built out of “panels”, and besides built-in panel types shipped with Grafana, there is a collection of community-developed panel plugins as well. In step dashboards we have included some of these plugins for better usability (see below), however the most basic version of the dashboard will also work without them.
Grafana and MongoDB
By default, step writes the measurement data into a MongoDB collection. At this moment, Grafana doesn’t ship with a MongoDB data source within the open-source edition. However, there is a community-developed open-source MongoDB plugin available, which relies on a node.js proxy server serving the aggregation requests from Grafana UI to the Mongo backend and back. This tutorial has been written for the users who are using an open-source setup and therefore it requires this community plugin to be installed, in order to connect to the step data stored in MongoDB.
If you have an Enterprise version of Grafana, you may be able to use an official MongoDB data source instead. In this case, slight re-working of the aggregation queries may be needed, as these two data source plugins expect slightly different data formatting (such as renaming some fields in the output of the last aggregation stage).
The dashboards require MongoDB v4.0.x or later.
step data and other databases
Through the step Enterprise analytics plugins the user can enable step to concurrently write data into databases of choice such as PostgreSQL or Prometheus. Both PostgreSQL and Prometheus data sources are available in the open-source edition of Grafana and will work out of the box.
Performance considerations
Grafana is designed to re-run all the active queries of all the open panels every time the dashboard is refreshed, time range changed or a different variable selected. This may lead to performance problems if you have a huge number of measurements already present in the database of your choice. While we cannot assist you in optimizing your database setup, there is a couple of things that you may want to try:
- optimize indexes in your database, especially for the fields which are declared as dashboard variables, as they are often used in queries (eId, name, status, type)
- in the dashboard, hide rows with the panels you don’t need at the moment - the queries inside the panels in hidden rows are not being fired
- use the dashboards provided as a starting point to create your own dashboards - remove panels and variables which you don’t use frequently
- consider using a different database - in particular, TimescaleDB may be significantly faster than MongoDB
Overview of the essential steps
The general steps needed to monitor step data in Grafana are the following:
- Install Grafana according to the official instructions. We recommend version 7.3.5+.
- Decide on the data source:
- If using other databases than MongoDB - enable step to write data to external databases
- Configure Grafana
- If using MongoDB as a data source, install the community plugin
- Install additional panel plugins as described below
- Configure the data source
- Import a step dashboard
- Run an execution in step to produce data
- Monitor and analyze your step data in Grafana
Below we elaborate on each of them in detail.
Install and configure Grafana to read step data
If you don’t have a running Grafana instance, install one following the official documentation. Once Grafana is installed, run the Grafana server and open Grafana in the browser. You will be asked to log in (admin/admin) and create a new user. With these initial steps, you should have Grafana up and running.
Plugins
While Grafana offers an automated way to install plugins using grafana-cli, in our experience many plugins hosted in the official repository are not up-to-date with their latest releases and the waiting time for a new release of a plugin to be accepted in the official repository is considerably long (months). Therefore for trusted plugins we recommend manual installation of the latest release, not yet visible in grafana-cli:
- Go to a relevant git repository
- Download the most recent plugin release
- Unpack the plugin to the plugin directory you have selected in the Grafana configuration (default is
GRAFANA_DIR/data/plugins
) - Restart Grafana server (and force refresh the Grafana UI if kept running)
Please make sure that using unsigned or manually installed plugins is compliant with the guidelines in your organization.
Plugins relevant for step dashboards
The following Grafana plugins are relevant for step dashboards:
- MongoDB data source
- Timepicker button panel v4.0.0
- (optionally for advanced dashboards) ae3e Plotly.js panel v0.3.3
Each of them can be installed following the steps described above.
Configure the data sources
Once you have installed the plugins, it is time to connect Grafana to step data by defining a Grafana data source. If you run your database at a different configuration than the standard one, have this information ready.
MongoDB with community plugin
Before the first start:
- Go to the directory where you installed the plugin (e.g.
GRAFANA_DIR/data/plugins/MONGO_PLUGIN_DIR
) - Install the node.js dependencies as described in the plugin documentation
To set up a data source:
- In the plugin directory, run the proxy server (
npm run server
). By default it listens to requests from Grafana onhttp://localhost:3333
- In Grafana:
- Open Configuration -> Data Sources:
- Select “Add Data Source”:
- Select “MongoDB datasource”:
- You will see the screen with data source settings.
- Now we have to fill them in. Use the following values:
- Name: step-mongo
- URL: PROXY_SERVER_URL (default
localhost:3333
) - MongoDB URL: MONGO_DB_URL (default
mongodb://localhost:27017
) - MongoDB Database: step
- Additionally configure your authentication details in the Auth section
- Click on “Save & Test” - you should see a green message confirming that the connection test was successful.