paulmataruso/docker-vuegraf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First Upload

Browse Source
This commit is contained in:
Paul Mataruso
2024-10-19 18:08:37 +00:00
commit c850b599b9
19 changed files with 1725 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
CHANGELOG.md Normal file
View File

@@ -0,0 +1,93 @@
# 1.TBD.TBD
## Breaking changes
- TBD
## New features
- TBD
## Other changes
- Upgrade to pyemvue 0.18.4 - @jertel
- Upgrade to influxdb_client 1.41.0 - @jertel
# 1.7.2
## Breaking changes
- None
## New features
- None
## Other changes
- Fixed daily metric collection after a new month cycles - @cdolghier
- Upgrade to pyemvue 0.18.1 - @jertel
# 1.7.1
## Breaking changes
- None
## New features
- None
## Other changes
- Added --debug arg to dump all points to console prior to sending to the database - @jertel
- On startup the new detailedDataHousesEnabled and detailedDataSecondsEnabled values will be printed to console - @jertel
- On startup the version will be printed to console - @jertel
- Fixed issue when hourly and historic data was collected it was discarding the previous minute data - @jertel
- Removed unused --quiet arg - @jertel
- Removed unnecessary --version arg - @jertel
- Refactored --verbose flag to properly use logging level - @jertel
- Removed unnecesary, duplicated queries - @jertel
# 1.7.0
## Breaking changes
- None
## New features
- Added `detailedDataSecondsEnabled` and `detailedDataHoursEnabled` to selectively fetch one or (or both) seconds- and hours- resolution data iff `detailedDataEnabled` = `true`
- Added `timezone` config to allow configuring the timezone according to which end-of-day is calculated.
## Other changes
- Fixed `--resetdatabase` (broken in 1.6.1)
# 1.6.1
## Breaking changes
- None
## New features
- None
## Other changes
- Upgrade to Python 3.12.1, replace deprecated datetime invocations - [#141](https://github.com/jertel/vuegraf/pull/141) - @jertel
- Fixed extractDataPoints to recurse correctly for nested devices - [#140](https://github.com/jertel/vuegraf/pull/140) - @cdolghier
# 1.6.0
## Breaking changes
- Replaced Minute with Hour as normal interval since history is limited to 7 days from Emporia on minute data
- argparse libary was added, must run `pip install -r requirements.txt` again in the src directory (or pip3 based on install)
## New features
- Hour / Day historic data retrieval: allows for history of up to two years to be pulled. Assists in clean numbers/graphs to see daily monthly usage to compare against utilities reports.
- Hour data runs with the get details time, default is 1 hour (3600 seconds). Based on when the program is started, you may be almost 2 hours behind for get hour.
- Moved one-time parameters out of the json config file. Those parameters are now specified as command line arguments (--historydays, --resetdatabase).
## Other changes
- Started Changelog for this and future releases
- Added project metadata to `vuegraf.py`, values can be updated through github automations
- Added command line pairing with help syntax for all values, via argparse lib.
- Updated `requirements.txt` and setup.py with `argparse>= 1.4.0`
- Updated `vuegraf.json.sample` as history and reset database was moved to command line
- Updated Readme.md with above changes
- ran pylint and fixed
Quote delimiter consistency to all '
Whitespaces
Extra lines
With special thanks to @gauthig for initiating these 1.6.0 changes!

34
Dockerfile Normal file
View File

@@ -0,0 +1,34 @@
# Fully qualified container name prevents public registry typosquatting
FROM docker.io/library/python:3-alpine
ARG UID=1012
ARG GID=1012
RUN addgroup -S -g $GID vuegraf
RUN adduser -S -g $GID -u $UID -h /opt/vuegraf vuegraf
WORKDIR /opt/vuegraf
# Install pip dependencies with minimal container layer size growth
COPY src/requirements.txt ./
RUN set -x && \
apk add --no-cache build-base libffi-dev rust cargo openssl-dev && \
pip install --upgrade pip && \
pip install --no-cache-dir -r requirements.txt && \
apk del build-base libffi-dev rust cargo openssl-dev && \
rm -rf /var/cache/apk /opt/vuegraf/requirements.txt
# Copying code in after requirements are built optimizes rebuild
# time, with only a marginal increate in image layer size; chmod
# is superfluous if "git update-index --chmod=+x ..." is done.
COPY src/vuegraf/*.py ./
RUN chmod a+x *.py
# A numeric UID is required for runAsNonRoot=true to succeed
USER $UID
VOLUME /opt/vuegraf/conf
ENTRYPOINT ["/opt/vuegraf/vuegraf.py" ]
CMD ["/opt/vuegraf/conf/vuegraf.json"]

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2020 Jason Ertel
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

350
README.md Normal file
View File

@@ -0,0 +1,350 @@
![Vuegraf Logo](https://github.com/jertel/vuegraf/blob/master/vuegraf.png?raw=true "Vuegraf Logo")
# Overview
The [Emporia Vue](https://emporiaenergy.com "Emporia's Homepage") energy monitoring kit allows homeowners to monitor their electrical usage. It monitors the main feed consumption and up to 8 (or 16 in the newer version) individual branch circuits, and feeds that data back to the Emporia API server.
This project, Vuegraf, fetches those metrics from the Emporia Vue API host and stores the metrics into your own InfluxDB. After installation you will be able to:
* View your energy usage across all circuits on a single graph
* Create alerts to notify when certain energy usage thresholds are exceeded
This project is not affiliated with _emporia energy_ company.
# Dependencies
* [Emporia Vue](https://emporiaenergy.com "Emporia Energy") Account - Username and password for the Emporia Vue system are required.
* [Python 3](https://python.org "Python") - With Pip.
* [InfluxDB 2](https://influxdata.com "InfluxDB") - Host, port, org, bucket, and token are all required.
# Influx
## Setup
If you do not yet have a running InfluxDB 2 instance, you will need to set one up. You can do this very quickly by launching an InfluxDB 2 Docker container as follows:
```
mkdir -p /home/myuser/influxdb2
docker run -v /home/myuser/influxdb2:/var/lib/influxdb2 -p 8086:8086 -e INFLUXD_SESSION_LENGTH=432000 --name influxdb influxdb
```
Substitute an appropriate host path for the `/home/myuser/influxdb2` location above. Once running, access the web UI at `http://localhost:8086`. It will prompt you for a username, password, organization name, and bucket name. The rest of this document assumes you have entered the word `vuegraf` for all of these inputs, except for the password; choose your own password that meets the minimum requirements.
Note that the default session timeout for Influx is only 60 minutes, so this command increases the login session to 300 days.
Once logged in, go to the _Load Data -> API Tokens_ screen and generate a new All Access token with the description of _vuegraf_. Copy the generated token for use in the rest of this document, specifically when referenced as `<my-influx-token>`.
## Dashboard
By default, a new InfluxDB instance will not have any dashboards loaded. You will need to import the included Influx JSON template, or create your own dashboard in order to visualize your energy usage. Because this template contains more than just the dashboard itself you will not be able to use the InfluxDB UI to perform the import. You will need to use the instructions included below.
The included template file named `influx_dashboard.json` includes the provided dashboard and accompanying variables to reproduce the visualizations shown below. This dashboard assumes your main/parent device name contains the word `Panel` (specifically cased as shown), such as `House Panel`, or `Right Panel`. If it does not, the Flux queries will need to be adjusted manually to look for your device's name. Note that nested devices should contain the word `Subpanel` (again using that specific upper/lower casing).
![Influx Dashboard Screenshot](https://github.com/jertel/vuegraf/blob/master/screenshots/influx_dashboard.png?raw=true "Influx Dashboard")
You will need to apply this template file to your running InfluxDB instance. First, copy the `influx_dashboard.json` file into your new InfluxDB container:
```
docker cp <path-to-vuegraf-project>/influx_dashboard.json influxdb:/var/lib/influxdb2/
```
Next, to import the dashboard, run the following command:
```
docker exec influxdb influx apply -f /var/lib/influxdb2/influx_dashboard.json --org vuegraf --force yes -t <my-influx-token>
```
Replace the `<my-influx-token>` with the All Access Token you generated in the Influx _Load Data -> API Tokens_ screen.
You're now ready to proceed with the Vuegraf configuration and startup.
# Configuration
The configuration allows for the definition of multiple Emporia Vue accounts. This will only be useful to users that need to pull metrics from multiple accounts. This is not needed if you have multiple Vue devices in a single account. Vuegraf will find multiple devices on its own within each account.
The email address and password must match the credentials used when creating the Emporia Vue account in their mobile app.
Important: Ensure that sufficient protection is in place on this configuration file, since it contains the plain-text login credentials into the Emporia Vue account.
A [sample configuration file](https://github.com/jertel/vuegraf/blob/master/vuegraf.json.sample "Sample Vuegraf Configuration File") is provided in this repository, and details are described below.
## Minimal Configuration
The minimum configuration required to start Vuegraf is shown below.
```json
{
"influxDb": {
"version": 2,
"url": "http://my.influxdb.hostname:8086",
"org": "vuegraf",
"bucket": "vuegraf",
"token": "<my-influx-token>"
},
"accounts": [
{
"name": "Primary Residence",
"email": "my@email.address",
"password": "my-emporia-password"
}
]
}
```
## Advanced Configuration
### Timezones
All data is stored in InfluxDB in UTC. To represent day-summary datapoints, vuegraf fetches a day's data at the end of the day in a certain timezone, configured by the configuration field `timezone`.
- if `timezone` is missing or null or its `upper()` is `"TZ"`, then the "default timezone" will be used
- the "default timezone" depends on the deployment method of the script
- If you are using Docker, the container has the timezone set to UTC unless the environment `TZ` is set.
- If you are running the script natively, it depends on your operating system. For example, in Ubuntu the timezone name is the contents of `/etc/timezone`
- for all values of `timezone` other than the ones named above, the string **SHOULD** be a valid timezone name.
The configured timezone is only relevant for collecting day-scoped data: the script fetches Emporia's "day to date" counter values, so if the account's timezone does not match the script one's, the last hours of the day will not be counted. For example, if your account is in the `America/Los_Angeles` timezone while the script runs its default UTC configuration in a Docker container, the daily summaries will miss the last 8 hours of every day.
For a list of timezones as of late 2023, consult the `TZ identifier` column of the table at [this wikipedia page](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones).
### Ingesting Historical Data
If desired, it is possible to have Vuegraf import historical data. To do so, run vuegraf.py with the optional `--historydays` parameter with a value between 1 and 720. When this parameter is provided Vuegraf will start and collect all hourly data points up to the specified parameter, or max history available. It will also collect one day's summary data for each day, storing it with the timestamp 23:59:59 for each day. It collects the time using the configured timezone, but stores it in influxDB in UTC.
IMPORTANT - If you restart Vuegraf with `--historydays` on the command line (or forget to remove it from the dockerfile) it will import history data _again_. This will likely cause confusion with your data since you will now have duplicate/overlapping data. For best results, only enable `--historydays` on a single run.
For Example:
```
python3 path/to/vuegraf.py vuegraf.json --historydays 365
```
### Channel Names
To provide more user-friendly names of each Vue device and branch circuit, the following device configuration can be added to the configuration file, within the account block. List each device and circuit in the order that you added them to the Vue mobile app. The channel names do not need to match the names specified in the Vue mobile app but the device names must match. The below example shows two 8-channel Vue devices for a home with two breaker panels.
Be aware that the included dashboard assumes your device name contains the word "Panel". For best results, consider renaming your Vue device to contain that word, otherwise you will need to manually adjust the included dashboards' queries.
```json
"devices": [
{
"name": "Right Panel",
"channels": [
"Air Conditioner",
"Furnace",
"Coffee Maker",
"Oven",
"Pool Vacuum",
"Pool Filter",
"Refrigerator",
"Office"
]
},
{
"name": "Left Panel",
"channels": [
"Dryer",
"Washer",
"Dishwasher",
"Water Heater",
"Landscape Features",
"Septic Pump",
"Deep Freeze",
"Sprinkler Pump"
]
}
]
```
# Running
Vuegraf can be run either as a container (recommended), or as a host process.
## Container (recommended)
A Docker container is provided at [hub.docker.com](https://hub.docker.com/r/jertel/vuegraf). Refer to the command below to launch Vuegraf as a container. This assumes you have created a folder called `/home/myuser/vuegraf` and placed the vuegraf.json file inside of it.
Normal run with docker
```sh
docker run --name vuegraf -d -v /home/myuser/vuegraf:/opt/vuegraf/conf jertel/vuegraf
```
Recreate database and load 25 days of history
```sh
docker run --name vuegraf -d -v /home/myuser/vuegraf:/opt/vuegraf/conf jertel/vuegraf --resetdatabase --historydays=24
```
## Host Process
Ensure Python 3 and Pip are both installed. Install the required dependencies:
```sh
pip install -r src/requirements.txt
```
or, on some Linux installations:
```sh
pip3 install -r src/requirements.txt
```
Then run the program via Python, specifying the JSON configuration file path as the only argument:
```sh
python src/vuegraf/vuegraf.py vuegraf.json
```
or, on some Linux installations:
```sh
python3 src/vuegraf/vuegraf.py vuegraf.json
```
Optional Command Line Parameters
```
usage: vuegraf.py [-h] [--version] [-v] [-q] [--historydays HISTORYDAYS] [--resetdatabase] configFilename
Retrieves data from cloud servers and inserts it into an InfluxDB database.
positional arguments:
configFilename JSON config file
options:
-h, --help show this help message and exit
--version Display version number
-v, --verbose Verbose output - summaries
--historydays HISTORYDAYS
Starts executing by pulling history of Hours and Day data for specified number of days.
example: --load-history-day 60
--resetdatabase Drop database and create a new one
```
## Alerts
The included dashboard template contains two alerts which will trigger when either a power outage occurs, or a loss of Vuegraf data. There are various reasons why alerts can be helpful. See the below screenshots which help illustrate how a fully functioning alert and notification rule might look. Note that the included alerts do not send out notifications. To enable outbound notifications, such as to Matrix or Slack, you can create a Notification Endpoint and Notification Rule.
This alert was edited via the text (Flux) interface since the alert edit UI does not yet accommodate advanced alerting inputs.
Side note: The logo at the top of this documentation satisfies Slack's icon requirements. Consider using it to help quickly distinguish between other alerts.
![Influx Alert Edit](https://github.com/jertel/vuegraf/blob/master/screenshots/alert_edit.png?raw=true "Influx Alert")
This notification rule provides an example of how you can have several alerts change the status to crit, but only a single notification rule is required to transmit notifications to external endpoints (such as email or Slack).
![Influx Notification Rule](https://github.com/jertel/vuegraf/blob/master/screenshots/notification_rule.png?raw=true "Influx Notification Rule")
To send alerts to a Matrix chat room hosted on a Synapse server, use a [Hookshot bot](https://github.com/matrix-org/matrix-hookshot) with appropriate URL path routing (via a reverse proxy), and in InfluxDB, specify the Alert Endpoint as follows:
- Destination: HTTP
- HTTP Method: POST
- Auth Method: None
- URL: https://my-matrix-host.net/webhook/1234abcd-4321-1234-abcd-abcdef123456
In the above example, /webhook/ routes to the Hookshot server. Then edit the Alert Notification Rule, and change the `body` variable assignment as follows:
```
body = {text: "🔴 ${r._notification_rule_name} -> ${r._check_name}"}
```
# Additional Topics
## Per-second and per-hour Data Details
By default, Vuegraf will poll every minute to collect the energy usage value over the past 60 seconds. This results in a single value being captured per minute per channel, or 60 values per hour per channel. If you also would like to also fetch per-second and/or per-hour values, you can enable the detailed collection, which is polled once per hour, and backfilled over the previous 3600 seconds. This API call is very expensive on the Emporia servers, so it should not be polled more frequently than once per hour. To enable this detailed data, add (or update) the top-level `detailedDataEnabled` configuration value with a value of `true`.
```
detailedDataEnabled: true
```
If `detailedDataEnabled` is set to `true`, the following two configuration fields become relevant. Notice that they are _not_ mutually exclusive and are actually both set to `true` unless overridden:
- `detailedDataSecondsEnabled` (default value is `true`): fetch and store per-second data every hour
- `detailedDataHoursEnabled` (default value is `true`): fetch and store per-hour data every hour
For every datapoint a tag is stored in InfluxDB for the type of measurement
- `detailed = True` represents backfilled per-second data that is optionally queried from Emporia once every hour.
- `detailed = False` represents the per-minute average data that is collected every minute.
- `detailed = Hour` represents the data summarized in hours
- `detailed = Day` represents a single data point to summarize the entire day
When building graphs that show a sum of the energy usage, be sure to only include the correct detail tag, otherwise your summed values will be higher than expected. Detailed data will take more time for the graphs to query due to the extra data involved. If you want to have a chart that shows daily data over a long period or even a full year, use the `detailed = Day` tag.
If you are running this on a small server, you might want to look at setting a RETENTION POLICY on your InfluxDB bucket to remove minute or second data over time. For example, it will reduce storage needs if you retain only 30 days of per-_second_ data.
## Vue Utility Connect Energy Monitor
As reported in [discussion #104](https://github.com/jertel/vuegraf/discussions/104), the Utility Connect device is supported without any custom changes.
## Smart Plugs
To include an Emporia smart plug in the configuration, add each plug as it's own device, without channels. Again, the name of the Smart Plug device must exactly match the name you gave the device in the Vue app during initial registration.
```json
devices: [
{
"name": "Main Panel",
"channels": [
"Air Conditioner",
"Furnace",
"Coffee Maker",
"Oven",
"Dishwasher",
"Tesla Charger",
"Refrigerator",
"Office"
]
},
{
"name": "Projector Plug"
},
{
"name": "3D-Printer Plug"
}
]
```
## Docker Compose
For those that want to run Vuegraf using Docker Compose, the following files have been included: `docker-compose.yaml.template` and `docker-compose-run.sh`. Copy the`docker-compose.yaml.template` file to a new file called `docker-compose.yaml`. In the newly copied file, `vuegraf.volumes` values will need to be changed to the same directory you have created your vuegraf.json file. Additionally, adjust the persistent host storage path for the InfluxDB data volume.
Finally run the `docker-compose-run.sh` script to start up the multi-container application.
```sh
./docker-compose-run.sh
```
## Upgrading from InfluxDB v1
Early Vuegraf users still on InfluxDB v1 can upgrade to InfluxDB 2. To do so, stop the Influx v1 container (again, assuming you're using Docker). Then run the following command to install InfluxDB 2 and automatically upgrade your data.
```
docker run --rm --pull always -p 8086:8086 \
-v /home/myuser/influxdb:/var/lib/influxdb \
-v /home/myuser/influxdb2:/var/lib/influxdb2 \
-e DOCKER_INFLUXDB_INIT_MODE=upgrade \
-e DOCKER_INFLUXDB_INIT_USERNAME=vuegraf \
-e DOCKER_INFLUXDB_INIT_PASSWORD=vuegraf \
-e DOCKER_INFLUXDB_INIT_ORG=vuegraf \
-e DOCKER_INFLUXDB_INIT_BUCKET=vuegraf \
-e DOCKER_INFLUXDB_INIT_RETENTION=1y \
influxdb
```
Adjust the host paths above as necessary, to match the old and new influxdb directories. The upgrade should complete relatively quickly. For reference, a 7GB database, spanning several months, upgrades in about 15 seconds on SSD storage.
Monitor the console output and once the upgrade completes and the Influx server finishes starting, shut it down (CTRL+C) and then restart the Influx DB using the command referenced earlier in this document.
Login to the new Influx DB 2 UI from your web browser, using the _vuegraf / vuegraf_ credentials. Go into the _Load Data -> Buckets_ screen and rename the `vue/autogen` bucket to `vuegraf` via the Settings button.
Finally, apply the dashboard template as instructed earlier in this document.
## Productionalizing the Server
There are additional steps necessary for making this configuration fault tolerant. Consider implementing the following:
- Configuring the container to always restart (such as after a reboot or a crash)
- Backing up the InfluxDB on a frequent basis
- Configure logging for rollover management, such as by file size or date
- Configure OS alerts to an admin when detecting crashes of critical software, such as InfluxDB
- Checking for low disk space on the host and alerting an admin
- Setting up calendar reminders for host OS updates and associated kernel reboots
- Updating Vuegraf and Influx on a schedule
- Much more!
These topics are out of scope of this project, but are intended to help new system administrators understand different areas that need to be considered for ensuring disaster recovery and prevention of vulnerabilities.
# License
Vuegraf is distributed under the MIT license.
See [LICENSE](https://github.com/jertel/vuegraf/blob/master/LICENSE) for more information.

11
docker-compose-run.sh Executable file
View File

@@ -0,0 +1,11 @@
# Running this script will bring up containers for InfluxDB and Vuegraf.
# Re-running this script will bring any previous containers down and bring up containers with any configuration changes you have made.
# Ran on MacOS with Docker Desktop for Mac 3.2.1
# vuegraf.json setting for host. "host": "host.docker.internal"
set -e
echo "Bringing up InfluxDB and Vuegraf"
docker-compose down && docker-compose -f docker-compose.yaml build --pull && docker-compose -f docker-compose.yaml up -d
echo "Done"

22
docker-compose.yaml Normal file
View File

@@ -0,0 +1,22 @@
version: "3.5"
services:
vue-influxdb:
image: influxdb
container_name: vue-influxdb
ports:
- 8086:8086
volumes:
- influxdb-storage:/var/lib/influxdb2
environment:
- INFLUXD_SESSION_LENGTH=432000
restart: always
vuegraf:
image: jertel/vuegraf
container_name: vuegraf
depends_on:
- vue-influxdb
volumes:
- ./:/opt/vuegraf/conf #This assumes you have created a folder called vuegraf and placed the vuegraf.json file inside of it.
restart: always
volumes:
influxdb-storage:

567
influx_dashboard.json Normal file
View File

@@ -0,0 +1,567 @@
[
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "Label",
"metadata": {
"name": "xenodochial-greider-36f001"
},
"spec": {
"color": "#009f5f",
"name": "outage"
}
},
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "CheckDeadman",
"metadata": {
"name": "noshing-borg-b6f003"
},
"spec": {
"every": "1m0s",
"level": "CRIT",
"name": "Energy Data Lost",
"query": "from(bucket: \"vue/autogen\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"energy_usage\")\n |\u003e filter(fn: (r) =\u003e r[\"_field\"] == \"usage\")",
"staleTime": "10m0s",
"status": "active",
"statusMessageTemplate": "Check: ${ r._check_name } is: ${ r._level }",
"timeSince": "1m30s"
}
},
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "CheckThreshold",
"metadata": {
"name": "competent-hopper-f6f003"
},
"spec": {
"every": "1m0s",
"name": "Power Outage",
"query": "from(bucket: \"vuegraf\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"energy_usage\")\n |\u003e filter(fn: (r) =\u003e r[\"_field\"] == \"usage\")\n |\u003e filter(fn: (r) =\u003e r[\"detailed\"] == \"False\")\n |\u003e filter(fn: (r) =\u003e r[\"device_name\"] =~ /Panel/)\n |\u003e aggregateWindow(every: 1m, fn: max, createEmpty: false)\n |\u003e yield(name: \"max\")",
"status": "active",
"statusMessageTemplate": "Check: ${ r._check_name } is: ${ r._level }",
"thresholds": [
{
"level": "CRIT",
"type": "lesser",
"value": 1
}
]
}
},
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "Variable",
"metadata": {
"name": "flamboyant-einstein-76f005"
},
"spec": {
"language": "flux",
"name": "Account",
"query": "import \"influxdata/influxdb/schema\"\nimport \"array\"\n\ndynamic = schema.tagValues(bucket: \"vuegraf\", tag: \"account_name\")\n\nstatic = array.from(\n rows: [\n {\n _value: \"(All)\",\n },\n ],\n)\n\nunion(tables: [static, dynamic])",
"selected": [
"(All)"
],
"type": "query"
}
},
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "Variable",
"metadata": {
"name": "funny-kapitsa-76f009"
},
"spec": {
"name": "DetailedDataEnabled",
"selected": [
"False"
],
"type": "constant",
"values": [
"True",
"False"
]
}
},
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "Variable",
"metadata": {
"name": "inspiring-shockley-b6f001"
},
"spec": {
"language": "flux",
"name": "Device",
"query": "import \"influxdata/influxdb/schema\"\nimport \"array\"\n\ndynamic = schema.tagValues(bucket: \"vuegraf\", tag: \"device_name\")\n\nstatic = array.from(\n rows: [\n {\n _value: \"(All)\",\n },\n ],\n)\n\nunion(tables: [static, dynamic])",
"selected": [
"(All)"
],
"type": "query"
}
},
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "Variable",
"metadata": {
"name": "realistic-kowalevski-76f007"
},
"spec": {
"name": "CostPerkWh",
"selected": [
"0.12"
],
"type": "constant",
"values": [
"0.10",
"0.11",
"0.12",
"0.13",
"0.14",
"0.15",
"0.16",
"0.17",
"0.18",
"0.19",
"0.20",
"0.21",
"0.22",
"0.23",
"0.24",
"0.25",
"0.26",
"0.27",
"0.28",
"0.29",
"0.30",
"0.31",
"0.32",
"0.33",
"0.34",
"0.35",
"0.36",
"0.37",
"0.38",
"0.39",
"0.40",
"0.41",
"0.42",
"0.43",
"0.44",
"0.45",
"0.46",
"0.47",
"0.48",
"0.49",
"0.50",
"0.51",
"0.52",
"0.53",
"0.54",
"0.55",
"0.56",
"0.57",
"0.58",
"0.59",
"0.60",
"0.61",
"0.62",
"0.63",
"0.64",
"0.65",
"0.66",
"0.67",
"0.68",
"0.69",
"0.70",
"0.71",
"0.72",
"0.73",
"0.74",
"0.75",
"0.76",
"0.77",
"0.78",
"0.79",
"0.80",
"0.81",
"0.82",
"0.83",
"0.84",
"0.85",
"0.86",
"0.87",
"0.88",
"0.89",
"0.90",
"0.91",
"0.92",
"0.93",
"0.94",
"0.95",
"0.96",
"0.97",
"0.98",
"0.99"
]
}
},
{
"apiVersion": "influxdata.com/v2alpha1",
"kind": "Dashboard",
"metadata": {
"name": "musing-gauss-b6f001"
},
"spec": {
"charts": [
{
"colors": [
{
"hex": "#7CE490",
"id": "0",
"name": "honeydew",
"type": "min"
},
{
"hex": "#FFD255",
"id": "EWB_hfTr0JyVEfRaI7TtX",
"name": "thunder",
"type": "threshold",
"value": 3
},
{
"hex": "#DC4E58",
"id": "k6XHqpWMc-pSrzoRu8pXD",
"name": "fire",
"type": "threshold",
"value": 5
},
{
"hex": "#BF3D5E",
"id": "1",
"name": "ruby",
"type": "max",
"value": 10
}
],
"decimalPlaces": 1,
"height": 4,
"kind": "Gauge",
"name": "Most Recent Combined Usage",
"queries": [
{
"query": "accountFilter = (tables=\u003c-) =\u003e\n if v.Account != \"(All)\" then\n tables |\u003e filter(fn: (r) =\u003e r[\"account_name\"] == v.Account)\n else\n tables\n\nfrom(bucket: \"vuegraf\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"energy_usage\")\n |\u003e accountFilter()\n |\u003e filter(fn: (r) =\u003e r[\"device_name\"] =~ /Panel/ and r[\"device_name\"] !~ /-Balance/)\n |\u003e filter(fn: (r) =\u003e (r[\"detailed\"] == \"False\"))\n |\u003e group(columns: [\"_time\"])\n |\u003e sum()\n |\u003e map(fn: (r) =\u003e ({ r with _value: r._value / 1000.0}))\n |\u003e group()\n |\u003e last()"
}
],
"staticLegend": {},
"suffix": " kW",
"width": 3
},
{
"axes": [
{
"base": "10",
"name": "x",
"scale": "linear"
},
{
"base": "10",
"name": "y",
"scale": "linear",
"suffix": "kW"
}
],
"colorizeRows": true,
"colors": [
{
"hex": "#31C0F6",
"id": "dNYnHkyoVodZBiY5Ah-1Y",
"name": "Nineteen Eighty Four",
"type": "scale"
},
{
"hex": "#A500A5",
"id": "N-KsOA9A4CWWm1TXAgmPp",
"name": "Nineteen Eighty Four",
"type": "scale"
},
{
"hex": "#FF7E27",
"id": "KXxluUZHVG6IGhsRq2QK0",
"name": "Nineteen Eighty Four",
"type": "scale"
}
],
"geom": "line",
"height": 4,
"hoverDimension": "auto",
"kind": "Xy",
"legendColorizeRows": true,
"legendOpacity": 1,
"legendOrientationThreshold": 100000000,
"name": "Combined Usage over Time Period",
"opacity": 1,
"orientationThreshold": 100000000,
"position": "overlaid",
"queries": [
{
"query": "accountFilter = (tables=\u003c-) =\u003e\n if v.Account != \"(All)\" then\n tables |\u003e filter(fn: (r) =\u003e r[\"account_name\"] == v.Account)\n else\n tables\n\nfrom(bucket: \"vuegraf\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"energy_usage\")\n |\u003e accountFilter()\n |\u003e filter(fn: (r) =\u003e r[\"device_name\"] =~ /Panel/ and r[\"device_name\"] !~ /-Balance/)\n |\u003e filter(fn: (r) =\u003e (r[\"detailed\"] == \"False\"))\n |\u003e group(columns: [\"_time\"])\n |\u003e sum()\n |\u003e map(fn: (r) =\u003e ({ r with _value: r._value / 1000.0}))\n |\u003e group()"
}
],
"shade": true,
"staticLegend": {
"colorizeRows": true,
"opacity": 1,
"orientationThreshold": 100000000,
"widthRatio": 1
},
"width": 12,
"widthRatio": 1,
"xCol": "_time",
"yCol": "_value",
"yPos": 4
},
{
"axes": [
{
"base": "10",
"name": "x",
"scale": "linear"
},
{
"base": "10",
"name": "y",
"scale": "linear",
"suffix": "W"
}
],
"colorizeRows": true,
"colors": [
{
"hex": "#74D495",
"id": "HtHVo9w6qtur77QlYwY2B",
"name": "Atlantis",
"type": "scale"
},
{
"hex": "#3F3FBA",
"id": "xVw_K2WdESvGIhqDSt1re",
"name": "Atlantis",
"type": "scale"
},
{
"hex": "#FF4D9E",
"id": "bH9L5-MEa6kMawcnqcqKY",
"name": "Atlantis",
"type": "scale"
}
],
"geom": "line",
"height": 4,
"heightRatio": 0.75,
"hoverDimension": "xy",
"kind": "Xy",
"legendColorizeRows": true,
"legendOpacity": 0.77,
"legendOrientationThreshold": 100000000,
"name": "Individual Device Usage over Time Period",
"opacity": 1,
"orientationThreshold": 100000000,
"position": "overlaid",
"queries": [
{
"query": "accountFilter = (tables=\u003c-) =\u003e\n if v.Account != \"(All)\" then\n tables |\u003e filter(fn: (r) =\u003e r[\"account_name\"] == v.Account)\n else\n tables\n\ndeviceFilter = (tables=\u003c-) =\u003e\n if v.Device == \"(All)\" then\n tables |\u003e filter(fn: (r) =\u003e r[\"device_name\"] !~ /Panel/)\n else\n tables |\u003e filter(fn: (r) =\u003e r[\"device_name\"] == v.Device)\n\nfrom(bucket: \"vuegraf\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"energy_usage\")\n |\u003e accountFilter()\n |\u003e deviceFilter()\n |\u003e filter(fn: (r) =\u003e (r[\"detailed\"] == \"False\" and r[\"device_name\"] =~ /-Balance/) or (r[\"detailed\"] == v.DetailedDataEnabled and r[\"device_name\"] !~ /-Balance/))\n |\u003e group(columns: [\"device_name\"])\n |\u003e aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |\u003e yield(name: \"mean\")"
}
],
"shade": true,
"staticLegend": {
"colorizeRows": true,
"heightRatio": 0.75,
"opacity": 1,
"orientationThreshold": 100000000,
"valueAxis": "y",
"widthRatio": 1
},
"valueAxis": "y",
"width": 12,
"widthRatio": 1,
"xCol": "_time",
"yCol": "_value",
"yPos": 8
},
{
"colors": [
{
"hex": "#FFD255",
"id": "base",
"name": "thunder",
"type": "text"
}
],
"decimalPlaces": 0,
"height": 2,
"kind": "Single_Stat",
"name": "Accumulated Combined Usage",
"queries": [
{
"query": "accountFilter = (tables=\u003c-) =\u003e\n if v.Account != \"(All)\" then\n tables |\u003e filter(fn: (r) =\u003e r[\"account_name\"] == v.Account)\n else\n tables\n\nfrom(bucket: \"vuegraf\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"energy_usage\")\n |\u003e accountFilter()\n |\u003e filter(fn: (r) =\u003e r[\"device_name\"] =~ /Panel/ and r[\"device_name\"] !~ /-Balance/)\n |\u003e filter(fn: (r) =\u003e (r[\"detailed\"] == \"False\"))\n |\u003e group(columns: [\"_time\"])\n |\u003e sum()\n |\u003e map(fn: (r) =\u003e ({ r with _value: (r._value / 60000.0)}))\n |\u003e group()\n |\u003e sum()"
}
],
"staticLegend": {},
"suffix": " kWh",
"width": 3,
"xPos": 3
},
{
"colors": [
{
"hex": "#BF3D5E",
"id": "base",
"name": "ruby",
"type": "text"
}
],
"decimalPlaces": 2,
"height": 2,
"kind": "Single_Stat",
"name": "Accumulated Combined Cost",
"prefix": "$",
"queries": [
{
"query": "accountFilter = (tables=\u003c-) =\u003e\n if v.Account != \"(All)\" then\n tables |\u003e filter(fn: (r) =\u003e r[\"account_name\"] == v.Account)\n else\n tables\n\nfrom(bucket: \"vuegraf\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"energy_usage\")\n |\u003e accountFilter()\n |\u003e filter(fn: (r) =\u003e r[\"device_name\"] =~ /Panel/ and r[\"device_name\"] !~ /-Balance/)\n |\u003e filter(fn: (r) =\u003e (r[\"detailed\"] == \"False\"))\n |\u003e group(columns: [\"_time\"])\n |\u003e sum()\n |\u003e map(fn: (r) =\u003e ({ r with _value: (r._value / 60000.0) * float(v: v.CostPerkWh)}))\n |\u003e group()\n |\u003e sum()"
}
],
"staticLegend": {},
"width": 3,
"xPos": 3,
"yPos": 2
},
{
"colors": [
{
"hex": "#4ED8A0",
"id": "base",
"name": "rainforest",
"type": "text"
},
{
"hex": "#4ED8A0",
"id": "WSyp2s3QMf552_5MxZ82q",
"name": "rainforest",
"type": "text"
},
{
"hex": "#00C9FF",
"id": "uq4MehfknGiAspmyZGYlo",
"name": "laser",
"type": "text",
"value": 1
},
{
"hex": "#F48D38",
"id": "wrpn7oZHPA5FbdpFjIiaO",
"name": "tiger",
"type": "text",
"value": 2
},
{
"hex": "#BF3D5E",
"id": "sAeFfBE5HDJfnIS_4EFWq",
"name": "ruby",
"type": "text",
"value": 3
}
],
"fieldOptions": [
{
"displayName": "Alarm",
"fieldName": "_check_name",
"visible": true
},
{
"displayName": "Severity",
"fieldName": "_value",
"visible": true
},
{
"displayName": "Status",
"fieldName": "_level",
"visible": true
},
{
"displayName": "Alarm",
"fieldName": "Alarm"
},
{
"displayName": "Status",
"fieldName": "Status"
},
{
"displayName": "_start",
"fieldName": "_start"
},
{
"displayName": "_stop",
"fieldName": "_stop"
},
{
"displayName": "Update Time",
"fieldName": "_time"
},
{
"displayName": "_check_id",
"fieldName": "_check_id"
},
{
"displayName": "_field",
"fieldName": "_field"
},
{
"displayName": "_measurement",
"fieldName": "_measurement"
},
{
"displayName": "_source_measurement",
"fieldName": "_source_measurement"
},
{
"displayName": "_type",
"fieldName": "_type"
},
{
"displayName": "account_name",
"fieldName": "account_name"
},
{
"displayName": "detailed",
"fieldName": "detailed"
},
{
"displayName": "device_name",
"fieldName": "device_name"
},
{
"displayName": "Update Time",
"fieldName": "Update Time"
}
],
"height": 4,
"kind": "Table",
"name": "Alarm Status",
"queries": [
{
"query": "from(bucket: \"_monitoring\")\n |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |\u003e filter(fn: (r) =\u003e r[\"_measurement\"] == \"statuses\")\n |\u003e filter(fn: (r) =\u003e r[\"_field\"] == \"_message\")\n |\u003e filter(fn: (r) =\u003e r[\"detailed\"] == \"False\")\n |\u003e drop(columns: [\"_value\"])\n |\u003e duplicate(column: \"_level\", as: \"_value\")\n |\u003e map(fn: (r) =\u003e ({ r with _value: if r._value == \"ok\" then 0 else if r._value == \"info\" then 1 else if r._value == \"warn\" then 2 else 3 }))\n |\u003e group(columns: [\"_check_id\"])\n |\u003e sort(columns: [\"_time\"])\n |\u003e last()\n |\u003e group()\n |\u003e keep(columns: [\"_check_name\",\"_level\",\"_value\"])"
}
],
"staticLegend": {},
"tableOptions": {
"sortBy": "Alarm",
"verticalTimeAxis": true
},
"timeFormat": "YYYY-MM-DD HH:mm:ss",
"width": 6,
"xPos": 6
}
],
"name": "Vuegraf Energy Dashboard"
}
}
]

85
k8s.yaml Normal file
View File

@@ -0,0 +1,85 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: vuegraf
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vue
namespace: vuegraf
spec:
replicas: 1
revisionHistoryLimit: 3
selector:
matchLabels:
app: vue
strategy:
type: Recreate
template:
metadata:
labels:
app: vue
spec:
containers:
- image: docker.io/jertel/vuegraf:latest
imagePullPolicy: IfNotPresent
name: vue
volumeMounts:
- mountPath: /opt/vuegraf/conf/vuegraf.json
subPath: vuegraf.json
name: config
securityContext:
runAsNonRoot: true
volumes:
- name: config
configMap:
name: vuegraf-configs
---
apiVersion: v1
kind: ConfigMap
metadata:
name: vuegraf-configs
namespace: vuegraf
data:
# Replace influx db configuration; emporia account information
# and channel names
vuegraf.json: |
{
"influxDb": {
"host": "influx-svc.influxdb.svc.cluster.local",
"port": 8086,
"user": "vue-write",
"pass": "XXX",
"database": "vue",
"reset": false
},
"accounts": [
{
"name": "accountNameHere",
"email": "user@example.com",
"password": "YYY",
"devices": [
{
"name": "panelNameHere",
"channels": [
"Kiln",
"Split AC",
"Outlets - left",
"Outlets - right",
"Outlets - central",
"Shedroom",
"Outlets - rear",
"Lights"
]
}
]
}
]
}

BIN
screenshots/alert_edit.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 77 KiB

BIN
screenshots/influx_dashboard.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 404 KiB

BIN
screenshots/notification_rule.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

36
setup.py Normal file
View File

@@ -0,0 +1,36 @@
import os
from setuptools import find_packages
from setuptools import setup
base_dir = os.path.dirname(__file__)
setup(
name='vuegraf',
version='1.7.2',
author='Jason Ertel',
long_description=open('README.md').read(),
long_description_content_type="text/markdown",
url='https://github.com/jertel/vuegraf',
description='Populate metrics from your Emporia Vue energy monitoring devices into an InfluxDB',
setup_requires='setuptools',
license='MIT',
project_urls={
"Documentation": "https://github.com/jertel/vuegraf",
"Source Code": "https://github.com/jertel/vuegraf",
"Discussion Forum": "https://github.com/jertel/vuegraf/discussions",
},
classifiers=[
'Programming Language :: Python :: 3.12',
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
],
entry_points={
'console_scripts': ['vuegraf=vuegraf.vuegraf']
},
packages=find_packages(),
install_requires=[
'influxdb>=5.3.1',
'influxdb_client>=1.41.0',
'pyemvue>=0.18.4',
'argparse>= 1.4.0'
]
)

4
src/requirements.txt Normal file
View File

@@ -0,0 +1,4 @@
influxdb >= 5.3.1
influxdb_client >= 1.41.0
pyemvue >= 0.18.4
argparse >= 1.4.0

6
src/vuegraf/.gitignore vendored Normal file
View File

@@ -0,0 +1,6 @@
vuegraf.json
int
docker-compose.yaml
dist/
vuegraf.egg-info/
vugraftobe.py

0
src/vuegraf/__init__.py Normal file
View File

403
src/vuegraf/vuegraf.py Normal file
View File

@@ -0,0 +1,403 @@
#!/usr/bin/env python3
__author__ = 'https://github.com/jertel'
__license__ = 'MIT'
__contributors__ = 'https://github.com/jertel/vuegraf/graphs/contributors'
__version__ = '1.7.2'
__versiondate__ = '2024/02/25'
__maintainer__ = 'https://github.com/jertel'
__github__ = 'https://github.com/jertel/vuegraf'
__status__ = 'Production'
import datetime
import json
import signal
import sys
import time
import traceback
from threading import Event
import argparse
import pytz
# InfluxDB v1
import influxdb
# InfluxDB v2
import influxdb_client
from pyemvue import PyEmVue
from pyemvue.enums import Scale, Unit
# flush=True helps when running in a container without a tty attached
# (alternatively, "python -u" or PYTHONUNBUFFERED will help here)
def log(level, msg):
now = datetime.datetime.now(datetime.UTC)
print('{} | {} | {}'.format(now, level.ljust(5), msg), flush=True)
def debug(msg):
if args.debug:
log('DEBUG', msg)
def error(msg):
log('ERROR', msg)
def info(msg):
log('INFO', msg)
def verbose(msg):
if args.verbose:
log('VERB', msg)
def handleExit(signum, frame):
global running
error('Caught exit signal')
running = False
pauseEvent.set()
def getConfigValue(key, defaultValue):
if key in config:
return config[key]
return defaultValue
def populateDevices(account):
deviceIdMap = {}
account['deviceIdMap'] = deviceIdMap
channelIdMap = {}
account['channelIdMap'] = channelIdMap
devices = account['vue'].get_devices()
for device in devices:
device = account['vue'].populate_device_properties(device)
deviceIdMap[device.device_gid] = device
for chan in device.channels:
key = '{}-{}'.format(device.device_gid, chan.channel_num)
if chan.name is None and chan.channel_num == '1,2,3':
chan.name = device.device_name
channelIdMap[key] = chan
info('Discovered new channel: {} ({})'.format(chan.name, chan.channel_num))
def lookupDeviceName(account, device_gid):
if device_gid not in account['deviceIdMap']:
populateDevices(account)
deviceName = '{}'.format(device_gid)
if device_gid in account['deviceIdMap']:
deviceName = account['deviceIdMap'][device_gid].device_name
return deviceName
def lookupChannelName(account, chan):
if chan.device_gid not in account['deviceIdMap']:
populateDevices(account)
deviceName = lookupDeviceName(account, chan.device_gid)
name = '{}-{}'.format(deviceName, chan.channel_num)
try:
num = int(chan.channel_num)
if 'devices' in account:
for device in account['devices']:
if 'name' in device and device['name'] == deviceName:
if 'channels' in device and len(device['channels']) >= num:
name = device['channels'][num - 1]
break
except:
if chan.channel_num == '1,2,3':
name = deviceName
return name
def createDataPoint(account, chanName, watts, timestamp, detailed):
dataPoint = None
if influxVersion == 2:
dataPoint = influxdb_client.Point('energy_usage') \
.tag('account_name', account['name']) \
.tag('device_name', chanName) \
.tag('detailed', detailed) \
.field('usage', watts) \
.time(time=timestamp)
else:
dataPoint = {
'measurement': 'energy_usage',
'tags': {
'account_name': account['name'],
'device_name': chanName,
'detailed': detailed,
},
'fields': {
'usage': watts,
},
'time': timestamp
}
return dataPoint
def dumpPoints(label, usageDataPoints):
if args.debug:
debug(label)
for point in usageDataPoints:
debug(' {}'.format(point.to_line_protocol()))
def extractDataPoints(device, usageDataPoints, pointType=None, historyStartTime=None, historyEndTime=None):
excludedDetailChannelNumbers = ['Balance', 'TotalUsage']
minutesInAnHour = 60
secondsInAMinute = 60
wattsInAKw = 1000
for chanNum, chan in device.channels.items():
if chan.nested_devices:
for gid, nestedDevice in chan.nested_devices.items():
extractDataPoints(nestedDevice, usageDataPoints, pointType, historyStartTime, historyEndTime)
chanName = lookupChannelName(account, chan)
kwhUsage = chan.usage
if kwhUsage is not None:
if pointType is None:
watts = float(minutesInAnHour * wattsInAKw) * kwhUsage
timestamp = stopTime
usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, False))
elif pointType == 'Day' or pointType == 'Hour' :
watts = kwhUsage * 1000
timestamp = historyStartTime
usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, pointType))
if chanNum in excludedDetailChannelNumbers:
continue
if collectDetails and detailedSecondsEnabled and historyStartTime is None:
# Collect seconds (once per hour, never during history collection)
verbose('Get second details; device="{}"; start="{}"; stop="{}"'.format(chanName, detailedStartTime, stopTime ))
usage, usage_start_time = account['vue'].get_chart_usage(chan, detailedStartTime, stopTime, scale=Scale.SECOND.value, unit=Unit.KWH.value)
index = 0
for kwhUsage in usage:
if kwhUsage is None:
continue
timestamp = detailedStartTime + datetime.timedelta(seconds=index)
watts = float(secondsInAMinute * minutesInAnHour * wattsInAKw) * kwhUsage
usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, True))
index += 1
# fetches historical minute data
if historyStartTime is not None and historyEndTime is not None:
verbose('Get historic details; device="{}"; start="{}"; stop="{}"'.format(chanName, historyStartTime,historyEndTime ))
#Hours
usage, usage_start_time = account['vue'].get_chart_usage(chan, historyStartTime, historyEndTime, scale=Scale.HOUR.value, unit=Unit.KWH.value)
index = 0
for kwhUsage in usage:
if kwhUsage is None:
continue
timestamp = historyStartTime + datetime.timedelta(hours=index)
watts = kwhUsage * 1000
usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, 'Hour'))
index += 1
#Days
usage, usage_start_time = account['vue'].get_chart_usage(chan, historyStartTime, historyEndTime, scale=Scale.DAY.value, unit=Unit.KWH.value)
index = 0
for kwhUsage in usage:
if kwhUsage is None:
continue
timestamp = historyStartTime + datetime.timedelta(days=index-1)
timestamp = timestamp.replace(hour=23, minute=59, second=59,microsecond=0)
timestamp = timestamp.astimezone(pytz.UTC)
watts = kwhUsage * 1000
usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, 'Day'))
index += 1
startupTime = datetime.datetime.now(datetime.UTC)
try:
#argparse includes default -h / --help as command line input
parser = argparse.ArgumentParser(
prog='vuegraf.py',
description='Veugraf retrieves energy usage data from commerical cloud servers and inserts it into a self-hosted InfluxDB database.',
epilog='For more information visit: ' + __github__
)
parser.add_argument(
'configFilename',
help='JSON config file',
type=str
)
parser.add_argument(
'-v',
'--verbose',
help='Verbose output - shows additional collection information',
action='store_true')
parser.add_argument(
'-d',
'--debug',
help='Debug output - shows all point data being collected and written to the DB (can be thousands of lines of output)',
action='store_true')
parser.add_argument(
'--historydays',
help='Starts execution by pulling history of Hours and Day data for specified number of days. example: --historydays 60',
type=int,
default=0
)
parser.add_argument(
'--resetdatabase',
action='store_true',
default=False,
help='Drop database and create a new one. USE WITH CAUTION - WILL RESULT IN COMPLETE VUEGRAF DATA LOSS!')
args = parser.parse_args()
info('Starting Vuegraf version {}'.format(__version__))
config = {}
with open(args.configFilename) as configFile:
config = json.load(configFile)
influxVersion = 1
if 'version' in config['influxDb']:
influxVersion = config['influxDb']['version']
bucket = ''
write_api = None
query_api = None
sslVerify = True
if 'ssl_verify' in config['influxDb']:
sslVerify = config['influxDb']['ssl_verify']
if influxVersion == 2:
info('Using InfluxDB version 2')
bucket = config['influxDb']['bucket']
org = config['influxDb']['org']
token = config['influxDb']['token']
url= config['influxDb']['url']
influx2 = influxdb_client.InfluxDBClient(
url=url,
token=token,
org=org,
verify_ssl=sslVerify
)
write_api = influx2.write_api(write_options=influxdb_client.client.write_api.SYNCHRONOUS)
query_api = influx2.query_api()
if args.resetdatabase:
info('Resetting database')
delete_api = influx2.delete_api()
start = '1970-01-01T00:00:00Z'
stop = startupTime.isoformat(timespec='seconds').replace("+00:00", "") + 'Z'
delete_api.delete(start, stop, '_measurement="energy_usage"', bucket=bucket, org=org)
else:
info('Using InfluxDB version 1')
sslEnable = False
if 'ssl_enable' in config['influxDb']:
sslEnable = config['influxDb']['ssl_enable']
# Only authenticate to ingress if 'user' entry was provided in config
if 'user' in config['influxDb']:
influx = influxdb.InfluxDBClient(host=config['influxDb']['host'], port=config['influxDb']['port'], username=config['influxDb']['user'], password=config['influxDb']['pass'], database=config['influxDb']['database'], ssl=sslEnable, verify_ssl=sslVerify)
else:
influx = influxdb.InfluxDBClient(host=config['influxDb']['host'], port=config['influxDb']['port'], database=config['influxDb']['database'], ssl=sslEnable, verify_ssl=sslVerify)
influx.create_database(config['influxDb']['database'])
if args.resetdatabase:
info('Resetting database')
influx.delete_series(measurement='energy_usage')
historyDays = min(args.historydays, 720)
history = historyDays > 0
running = True
signal.signal(signal.SIGINT, handleExit)
signal.signal(signal.SIGHUP, handleExit)
pauseEvent = Event()
intervalSecs = getConfigValue('updateIntervalSecs', 60)
detailedIntervalSecs = getConfigValue('detailedIntervalSecs', 3600)
detailedDataEnabled = getConfigValue('detailedDataEnabled', False)
detailedSecondsEnabled = detailedDataEnabled and getConfigValue('detailedDataSecondsEnabled', True)
detailedHoursEnabled = detailedDataEnabled and getConfigValue('detailedDataHoursEnabled', True)
info('Settings -> updateIntervalSecs: {}, detailedDataEnabled: {}, detailedIntervalSecs: {}, detailedDataHoursEnabled: {}, detailedDataSecondsEnabled: {}'.format(intervalSecs, detailedDataEnabled, detailedIntervalSecs, detailedHoursEnabled, detailedSecondsEnabled))
lagSecs = getConfigValue('lagSecs', 5)
accountTimeZoneName = getConfigValue('timezone', None)
accountTimeZone = pytz.timezone(accountTimeZoneName) if accountTimeZoneName is not None and accountTimeZoneName.upper() != "TZ" else None
info('Settings -> timezone: {}'.format(accountTimeZone))
detailedStartTime = startupTime
pastDay = datetime.datetime.now(accountTimeZone)
pastDay = pastDay.replace(hour=23, minute=59, second=59, microsecond=0)
while running:
usageDataPoints = []
now = datetime.datetime.now(datetime.UTC)
curDay = datetime.datetime.now(accountTimeZone)
stopTime = now - datetime.timedelta(seconds=lagSecs)
secondsSinceLastDetailCollection = (stopTime - detailedStartTime).total_seconds()
collectDetails = detailedDataEnabled and detailedIntervalSecs > 0 and secondsSinceLastDetailCollection >= detailedIntervalSecs
verbose('Starting next event collection; collectDetails={}; secondsSinceLastDetailCollection={}; detailedIntervalSecs={}'.format(collectDetails, secondsSinceLastDetailCollection, detailedIntervalSecs))
for account in config['accounts']:
if 'vue' not in account:
account['vue'] = PyEmVue()
account['vue'].login(username=account['email'], password=account['password'])
info('Login completed')
populateDevices(account)
try:
deviceGids = list(account['deviceIdMap'].keys())
usages = account['vue'].get_device_list_usage(deviceGids, stopTime, scale=Scale.MINUTE.value, unit=Unit.KWH.value)
if usages is not None:
for gid, device in usages.items():
extractDataPoints(device, usageDataPoints)
if collectDetails and detailedHoursEnabled:
pastHour = stopTime - datetime.timedelta(hours=1)
pastHour = pastHour.replace(minute=00, second=00,microsecond=0)
verbose('Collecting previous hour: {} '.format(pastHour))
historyStartTime = pastHour
usages = account['vue'].get_device_list_usage(deviceGids, pastHour, scale=Scale.HOUR.value, unit=Unit.KWH.value)
if usages is not None:
for gid, device in usages.items():
extractDataPoints(device, usageDataPoints, 'Hour', historyStartTime)
if pastDay.day != curDay.day:
usages = account['vue'].get_device_list_usage(deviceGids, pastDay, scale=Scale.DAY.value, unit=Unit.KWH.value)
historyStartTime = pastDay.astimezone(pytz.UTC)
verbose('Collecting previous day: {}Local - {}UTC, '.format(pastDay, historyStartTime))
if usages is not None:
for gid, device in usages.items():
extractDataPoints(device, usageDataPoints,'Day', historyStartTime)
pastDay = datetime.datetime.now(accountTimeZone)
pastDay = pastDay.replace(hour=23, minute=59, second=00, microsecond=0)
if history:
info('Loading historical data: {} day(s) ago'.format(historyDays))
historyStartTime = stopTime - datetime.timedelta(historyDays)
historyStartTime = historyStartTime.replace(hour=00, minute=00, second=00, microsecond=000000)
while historyStartTime <= stopTime:
historyEndTime = min(historyStartTime + datetime.timedelta(20), stopTime)
historyEndTime = historyEndTime.replace(hour=23, minute=59, second=59,microsecond=0)
verbose(' {} - {}'.format(historyStartTime,historyEndTime))
for gid, device in usages.items():
extractDataPoints(device, usageDataPoints, 'History', historyStartTime, historyEndTime)
if not running:
break
historyStartTime = historyEndTime + datetime.timedelta(1)
historyStartTime = historyStartTime.replace(hour=00, minute=00, second=00, microsecond=000000)
pauseEvent.wait(5)
history = False
if not running:
break
info('Submitting datapoints to database; account="{}"; points={}'.format(account['name'], len(usageDataPoints)))
dumpPoints("Sending to database", usageDataPoints)
if influxVersion == 2:
write_api.write(bucket=bucket, record=usageDataPoints)
else:
influx.write_points(usageDataPoints,batch_size=5000)
except:
error('Failed to record new usage data: {}'.format(sys.exc_info()))
traceback.print_exc()
if collectDetails:
detailedStartTime = stopTime + datetime.timedelta(seconds=1)
pauseEvent.wait(intervalSecs)
info('Finished')
except SystemExit as e:
#If sys.exit was 2, then normal syntax exit from help or bad command line, no error message
if e.code == 0 or e.code == 2:
quit(0)
else:
error('Fatal error: {}'.format(sys.exc_info()))
traceback.print_exc()

44
vuegraf.json Normal file
View File

@@ -0,0 +1,44 @@
{
"influxDb": {
"version": 2,
"url": "http://vue-influxdb:8086",
"org": "DHI",
"bucket": "vuegraf",
"token": "zzvcytd060kXkZrR1jPmgDe7e_OnZwa8CcZl91YuvD8Mqz89wHpUUJxtJquNxQMDq2Auh-RlgS-IEsDYCqt6Aw=="
},
"updateIntervalSecs": 60,
"detailedDataEnabled": false,
"detailedDataSecondsEnabled": true,
"detailedDataHoursEnabled": true,
"timezone": "TZ",
"accounts": [
{
"name": "9 Putnam Panel",
"email": "mataruso@gmail.com",
"password": "Gx180d132!23456",
"devices": [
{
"name": "Main Panel",
"channels": [
"Datacenter Top",
"Datacenter Bottom",
"Furnace",
"C4",
"C5",
"Datacenter Left",
"C7",
"C8",
"C9",
"C10",
"C11",
"Office",
"Tank Outlet",
"R2 Breaker",
"Fridge",
"R1 Breaker"
]
}
]
}
]
}

49
vuegraf.json.bak Normal file
View File

@@ -0,0 +1,49 @@
{
"influxDb": {
"version": 2,
"url": "http://vue-influxdb:8086",
"org": "DHI",
"bucket": "vuegraf",
"token": "zzvcytd060kXkZrR1jPmgDe7e_OnZwa8CcZl91YuvD8Mqz89wHpUUJxtJquNxQMDq2Auh-RlgS-IEsDYCqt6Aw=="
},
"updateIntervalSecs": 60,
"detailedDataEnabled": false,
"detailedDataSecondsEnabled": true,
"detailedDataHoursEnabled": true,
"timezone": "TZ",
"accounts": [
{
"name": "9 Putnam Panel",
"email": "mataruso@gmail.com",
"password": "Gx180d132!23456",
"devices": [
{
"name": "Panel Phase A",
"channels": [
"Datacenter Top",
"Datacenter Bottom",
"Furnace",
"C4",
"C5",
"Datacenter Left",
"C7",
"C8"
]
},
{
"name": "Panel Phase B",
"channels": [
"C9",
"C10",
"C11",
"Office",
"Tank Outlet",
"R2 Breaker",
"Fridge",
"R1 Breaker"
]
}
]
}
]
}

BIN
vuegraf.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB