Toggle navigation

Michael Wadman

Home

About Me

CV

Tags

Monitoring

Grafana

LibreNMS

Viewing LibreNMS data in Grafana

Using RRDReST as a Grafana data source

Posted by Michael Wadman on January 2, 2022

Overview

LibreNMS is a great network device monitoring system, and Grafana is a wonderful tool for visualising and alerting on data from multiple sources.

Both are great examples of open-source software that you should be considering for your environment.

However, while LibreNMS does provide official integration with many external databases that you can then use as a data source in Grafana, if you don’t already have one of these in your environment it requires you to deploy these as well.

I’ve also personally run into performance issues with the Prometheus Pushgateway integration when pushing a large number of metrics, probably because Pushgateway isn’t designed for that use case.

Recently I stumbled upon a Reddit post by user /u/wjhinz which described a method for “(almost) natively” using LibreNMS as a data source in Grafana by using RRDReST as an API endpoint to access the RRD files that LibreNMS creates.

After testing this and confirming it works very well, I wanted to add to the Reddit post by documenting what a containerised deployment of this solution looks like.

Major kudos to both /u/tbotnz (for their work on RRDReST) and /u/wjhinz (for documenting this solution).

Containerised Deployment

LibreNMS

I’ll be using Docker Compose for this deployment, and starting with the LibreNMS docker example compose file as the base.

I’ve trimmed some services that aren’t needed (mstpd, syslog and SNMP traps), and included my compose file in an appendix section.

RRDReST

Next, we want to add an RRDReST container. As a container image doesn’t currently exist, we will need to create one.

Thanks to tbotnz, who has provided instructions on how to get RRDReST running in his Github repository, creating a container image is relatively easy.

Here’s the Dockerfile I’ve written:

FROM python:3.10-alpine
RUN apk add g++ make rrdtool
COPY requirements.txt /opt/RRDReST/requirements.txt
WORKDIR /opt/RRDReST
RUN pip3 install -r requirements.txt
COPY . /opt/RRDReST
ENTRYPOINT ["uvicorn", "rrdrest:rrd_rest"]
CMD ["--host", "0.0.0.0", "--port", "9000"]

I’ve built the container using this and pushed it to be pulled from Docker Hub as well.

To add this to our deployment, we simply need to add another service to the docker-compose.yml file and give the container access to the LibreNMS volume (so it can read the RRD files).

I’m using /opt/librenms as the mount point for the volume, as it mimics where data would be located in a non-containerised deployment.

services:
...
rrdrest:
image: michaelwadman/rrdrest:latest
container_name: rrdrest
volumes:
- "./librenms:/opt/librenms"

As /u/wjhinz notes in their post, RRDReST needs read access to the RRD files, which should be fine with a docker-compose set up like this, but could be a good place to start investigations if it isn’t working for you.

Grafana

Finally, we want to add a Grafana container.

services:
...
grafana:
image: grafana/grafana-oss:latest
container_name: grafana
ports:
- target: 3000
published: 3000
protocol: tcp
environment:
- "GF_INSTALL_PLUGINS=marcusolsson-json-datasource"

Note that I’m adding an environment variable for the installation of the JSON API plugin, as this is needed to interact with both the LibreNMS API as well as the RRDReST API.

docker-compose up

With a complete compose file, we can spin up the environment:

$ sudo docker-compose up -d
Creating network "librenms_grafana_default" with the default driver
Creating librenms ... done
Creating grafana ... done
Creating librenms_db ... done
Creating librenms_dispatcher ... done
Creating librenms_redis ... done
Creating librenms_memcached ... done
Creating rrdrest ... done

Adding data sources in Grafana

Creating LibreNMS user

To add LibreNMS as a data source in Grafana, we first need to create an API token in LibreNMS.

Visit the LibreNMS web interface and log in (in my case, “http://127.0.0.1:8000”, with the default credentials of librenms/librenms).

Because Grafana will only require read access, we’re going to create a new user with read-only access and then associate the token we create with that user.

Under ‘Settings -> Manage Users’, I created a user named “grafana”, changed the access level to “Global Read” and set a random password as we will only be using the API access.

Then, under ‘Settings -> API -> API Settings’, I created an API token for the grafana user:

Make sure to copy the token when creating it, as we’ll need that later when adding the data source into LibreNMS.

Adding LibreNMS data source in Grafana

After logging into the Grafana web interface (in my case, “http://127.0.0.1:3000”, with the default credentials of admin/admin), navigate to ‘Configuration -> Data Sources’ from the left pane.

Add a new data source and select the JSON API type from the list.

In the settings for the LibreNMS API data source, we need to change the URL and add the token authentication header.

The URL should be set to your LibreNMS host and port, with the API endpoint of “/api/v0” appended.

Then, under “Custom HTTP Headers”, select “Add Header”. The header name will be “X-Auth-Token” and the value will be the token string that you created above.

Save and test that the API endpoint is working for you, and you should get a green “Success” pop-up if everything is working.

My data source looked like the following when finished:

Adding RRDReST data source in Grafana

Like the above, we’re going to also add the RRDReST API as a new data source in Grafana too.

This will also be a JSON API, so select this from the data source type when creating it.

In the settings for the RRDReST API data source, we only need to change the URL.

Again, point this towards your RRDReST container host and port, and leave the endpoint blank.

Save and test that the API endpoint is working for you, and you should get a red “Unprocessable Entity” pop-up if everything is working. This error means the API is reachable, but the plugin can’t process the data returned (which is expected, because there isn’t any data returned, as we haven’t pointed it to an RRD file yet).

My data source looked like the following when finished:

Creating a dashboard

The last thing to do is to create a dashboard for the LibreNMS data to be visualised.

From the left pane in Grafana, select ‘Create -> Import’ and then upload the JSON file created by /u/wjhinz on Pastebin. I’ve also included it in an appendix section below in the case that the Pastebin link goes away for some reason.

After import, you’ll be prompted to name the dashboard, place it into a folder and select the data sources you created before. Here’s what mine looked like before import:

There was one last item that I needed to change to get data displayed on the graph panels, which was to set the timezone to UTC. I believe this is needed because LibreNMS stores (or RRDReST represents) data in UTC format, but Grafana queries it with the local browser timezone by default.

To do so, drop-down the time picker from the top right of the dashboard screen, select “Change Time Settings” and choose UTC.

If you’ve got devices in LibreNMS, they should appear up the top left of the page in a drop down menu along with a list of interfaces. After selecting a device and an interface, both the “Bandwidth In/Out” and “Errors In/Out” graphs will populate with information:

While the panels in the dashboard provided only show port throughput and errors, it’s pretty easy to pick up the other RRD files in a device directory and display those statistics as well.

For example, LibreNMS has created 64 other RRD files full of stats for my test device:

$ sudo docker exec -it rrdrest ls /opt/librenms/rrd/
Test_Device
$ sudo docker exec -it rrdrest ls /opt/librenms/rrd/Test_Device/ | grep -v port | wc -l
64

Conclusion

Providing RRDReST access to the RRD files that LibreNMS creates is not only easy, but it performs well too.

If you are already using Grafana in your environment to visualise, analyse and/or alert on data in a centralised fashion, then pulling SNMP metrics in through LibreNMS (and its’ device discovery feature) from devices that are otherwise unmonitored and using RRDReST as a data source is a great “quick win”.

Appendices

References

wjhinz’s Reddit Post

RRDReST Repository

RRDReST Container Image

Grafana JSON API Data source Plugin

Versions used

Desktop Machine: Ubuntu 20.04

Docker: 20.10.12

Docker Compose: 1.26.2

LibreNMS: 21.11.0

Grafana: 8.3.3

Full Code

Docker Compose file

version: "3.5"

services:
db:
image: mariadb:10.5
container_name: librenms_db
command:
- "mysqld"
- "--innodb-file-per-table=1"
- "--lower-case-table-names=0"
- "--character-set-server=utf8mb4"
- "--collation-server=utf8mb4_unicode_ci"
volumes:
- "./db:/var/lib/mysql"
environment:
- "TZ=Pacific/Auckland"
- "MYSQL_ALLOW_EMPTY_PASSWORD=yes"
- "MYSQL_DATABASE=librenms"
- "MYSQL_USER=librenms"
- "MYSQL_PASSWORD=asupersecretpassword"
restart: always

memcached:
image: memcached:alpine
container_name: librenms_memcached
environment:
- "TZ=Pacific/Auckland"
restart: always

redis:
image: redis:5.0-alpine
container_name: librenms_redis
environment:
- "TZ=Pacific/Auckland"
restart: always

librenms:
image: librenms/librenms:latest
container_name: librenms
hostname: librenms
cap_add:
- NET_ADMIN
- NET_RAW
ports:
- target: 8000
published: 8000
protocol: tcp
volumes:
- "./librenms:/data"
environment:
- "TZ=Pacific/Auckland"
- "PUID=1000"
- "PGID=1000"
- "DB_HOST=db"
- "DB_NAME=librenms"
- "DB_USER=librenms"
- "DB_PASSWORD=asupersecretpassword"
- "DB_TIMEOUT=60"
- "REDIS_HOST=redis"
- "REDIS_PORT=6379"
- "REDIS_DB=0"
- "MEMORY_LIMIT=256M"
- "UPLOAD_MAX_SIZE=16M"
- "OPCACHE_MEM_SIZE=128"
- "REAL_IP_FROM=0.0.0.0/32"
- "REAL_IP_HEADER=X-Forwarded-For"
- "LOG_IP_VAR=remote_addr"
- "MEMCACHED_HOST=memcached"
- "MEMCACHED_PORT=11211"
- "LIBRENMS_WEATHERMAP=false"
restart: always

dispatcher:
image: librenms/librenms:latest
container_name: librenms_dispatcher
hostname: librenms-dispatcher
cap_add:
- NET_ADMIN
- NET_RAW
volumes:
- "./librenms:/data"
environment:
- "TZ=Pacific/Auckland"
- "PUID=1000"
- "PGID=1000"
- "DB_HOST=db"
- "DB_NAME=librenms"
- "DB_USER=librenms"
- "DB_PASSWORD=asupersecretpassword"
- "DB_TIMEOUT=60"
- "DISPATCHER_NODE_ID=dispatcher1"
- "REDIS_HOST=redis"
- "REDIS_PORT=6379"
- "REDIS_DB=0"
- "SIDECAR_DISPATCHER=1"
- "MEMORY_LIMIT=256M"
- "UPLOAD_MAX_SIZE=16M"
- "OPCACHE_MEM_SIZE=128"
- "REAL_IP_FROM=0.0.0.0/32"
- "REAL_IP_HEADER=X-Forwarded-For"
- "LOG_IP_VAR=remote_addr"
- "MEMCACHED_HOST=memcached"
- "MEMCACHED_PORT=11211"
- "LIBRENMS_WEATHERMAP=false"
restart: always

rrdrest:
image: michaelwadman/rrdrest:latest
container_name: rrdrest
volumes:
- "./librenms:/opt/librenms"

grafana:
image: grafana/grafana-oss:latest
container_name: grafana
ports:
- target: 3000
published: 3000
protocol: tcp
environment:
- "GF_INSTALL_PLUGINS=marcusolsson-json-datasource"

Grafana Dashboard JSON

{
"__inputs": [
{
"name": "DS_RRDREST_API",
"label": "RRDRest API",
"description": "",
"type": "datasource",
"pluginId": "marcusolsson-json-datasource",
"pluginName": "JSON API"
},
{
"name": "DS_LIBRENMS_API",
"label": "LibreNMS API",
"description": "",
"type": "datasource",
"pluginId": "marcusolsson-json-datasource",
"pluginName": "JSON API"
}
],
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana"
},
{
"type": "panel",
"id": "graph",
"name": "Graph",
"version": ""
},
{
"type": "datasource",
"id": "marcusolsson-json-datasource",
"name": "JSON API",
"version": "1.2.1"
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": null,
"iteration": 1626152089476,
"links": [],
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "${DS_RRDREST_API}",
"fieldConfig": {
"defaults": {
"custom": {},
"unit": "bps"
},
"overrides": []
},
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 15,
"w": 24,
"x": 0,
"y": 0
},
"hiddenSeries": false,
"id": 2,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "7.3.5",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"body": "",
"cacheDurationSeconds": 0,
"fields": [
{
"jsonPath": "$.data[*].time",
"name": "time",
"type": "time"
},
{
"jsonPath": "$.data[*].inoctets",
"name": "inoctets",
"type": "number"
},
{
"jsonPath": "$.data[*].outoctets",
"name": "outoctets",
"type": "number"
}
],
"hide": false,
"method": "GET",
"params": [
[
"",
""
]
],
"queryParams": "",
"refId": "A",
"urlPath": "?rrd_path=/opt/librenms/rrd/$device/port-id$portid.rrd&epoch_start_time=${__from:date:seconds}&epoch_end_time=${__to:date:seconds}"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Bandwidth In/Out",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"transformations": [
{
"id": "calculateField",
"options": {
"alias": "inMbps",
"binary": {
"left": "inoctets",
"operator": "*",
"reducer": "sum",
"right": "8"
},
"mode": "binary",
"reduce": {
"include": [
"inoctets"
],
"reducer": "lastNotNull"
},
"replaceFields": false
}
},
{
"id": "calculateField",
"options": {
"alias": "outMbps",
"binary": {
"left": "outoctets",
"operator": "*",
"reducer": "sum",
"right": "8"
},
"mode": "binary",
"reduce": {
"reducer": "sum"
},
"replaceFields": false
}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"names": [
"time",
"inMbps",
"outMbps"
]
}
}
}
],
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"$$hashKey": "object:1546",
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"$$hashKey": "object:1547",
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "${DS_RRDREST_API}",
"fieldConfig": {
"defaults": {
"custom": {},
"unit": "bps"
},
"overrides": []
},
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 15,
"w": 24,
"x": 0,
"y": 15
},
"hiddenSeries": false,
"id": 3,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "7.3.5",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"body": "",
"cacheDurationSeconds": 0,
"fields": [
{
"jsonPath": "$.data[*].time",
"name": "time",
"type": "time"
},
{
"jsonPath": "$.data[*].inerrors",
"name": "inerrors",
"type": "number"
},
{
"jsonPath": "$.data[*].outerrors",
"name": "outerrors",
"type": "number"
}
],
"hide": false,
"method": "GET",
"params": [
[
"",
""
]
],
"queryParams": "",
"refId": "A",
"urlPath": "?rrd_path=/opt/librenms/rrd/$device/port-id$portid.rrd&epoch_start_time=${__from:date:seconds}&epoch_end_time=${__to:date:seconds}"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Errors In/Out",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"transformations": [],
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"$$hashKey": "object:1546",
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"$$hashKey": "object:1547",
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"refresh": "30s",
"schemaVersion": 26,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"allValue": null,
"current": {},
"datasource": "${DS_LIBRENMS_API}",
"definition": "$.devices[*].hostname",
"error": null,
"hide": 0,
"includeAll": false,
"label": null,
"multi": false,
"name": "device",
"options": [],
"query": {
"cacheDurationSeconds": 300,
"fields": [
{
"jsonPath": "$.devices[*].hostname"
}
],
"method": "GET",
"queryParams": "",
"urlPath": "/devices"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {},
"datasource": "${DS_LIBRENMS_API}",
"definition": "$.ports[*].port_id",
"error": null,
"hide": 0,
"includeAll": false,
"label": "Interface",
"multi": false,
"name": "portid",
"options": [],
"query": {
"cacheDurationSeconds": 300,
"experimentalVariableTextField": "ifName",
"experimentalVariableValueField": "id",
"fields": [
{
"jsonPath": "$.ports[*].port_id",
"name": "id"
},
{
"jsonPath": "$.ports[*].ifName",
"name": ""
}
],
"method": "GET",
"queryParams": "",
"urlPath": "/devices/$device/ports?columns=ifName%2Cport_id"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-12h",
"to": "now"
},
"timepicker": {},
"timezone": "utc",
"title": "LibreNMS RRDReST",
"uid": "mFveL2mm1",
"version": 28
}

Previous

Resolving my DNS issues - Disabling systemd-resolved on Ubuntu 18.04

CATALOG

Overview

Containerised Deployment

LibreNMS

RRDReST

Grafana

docker-compose up

Adding data sources in Grafana

Creating LibreNMS user

Adding LibreNMS data source in Grafana

Adding RRDReST data source in Grafana

Creating a dashboard

Conclusion

Appendices

References

Versions used

Full Code

Docker Compose file

Grafana Dashboard JSON

Theme on GitHub