smurphy
Full Member
Posts: 169
Raspberry Pi: Yes
Other Device: many ...
|
Post by smurphy on Nov 26, 2020 7:49:43 GMT -8
Hi all,
here is my first attempt to graph the weatherdata using Grafana Note that the setup is not that easy.
To understand what I do - you need to know that I try to reduce to a max whatever is downloaded from the raspberry PI Zero W. So - only thing that gets out is WeatherStats.csv (little addon from me) and the skycam picture already reduced in side.
After that, I import the data into mrtg (RRD Files) for long time storage. This actually can be used by various other tools. With the grafana_rrd_server I can then connect to it through grafana as it provides the rrd data sources as regular json output.
I will have to work on the lightning data stuff as that one is not as I want it to be. Note - the temp color will change with the temperature. Will get blue below zero, and bright red when getting above 35 ⁰C
|
|
|
Post by SDL on Nov 28, 2020 9:33:51 GMT -8
That's beautiful!
We startend out with Graphana on our Python3 SkyWeather2 and got it up and kind of working. Our feeling was that it was waaaayy too much for people to set up.
I'm very impressed that you got this to work. How about writing a tutorial on it and we will post it as a guest blog?
In Python3 SkyWeather2 we went to a dash_app. Not quite as elegant of an output, but a whale of a lot easy for the end customer.
BP
|
|
|
Post by Jason on Nov 28, 2020 16:21:49 GMT -8
Very cool!
Jason
|
|
smurphy
Full Member
Posts: 169
Raspberry Pi: Yes
Other Device: many ...
|
Post by smurphy on Nov 29, 2020 3:12:57 GMT -8
That's beautiful! We startend out with Graphana on our Python3 SkyWeather2 and got it up and kind of working. Our feeling was that it was waaaayy too much for people to set up. I'm very impressed that you got this to work. How about writing a tutorial on it and we will post it as a guest blog? In Python3 SkyWeather2 we went to a dash_app. Not quite as elegant of an output, but a whale of a lot easy for the end customer. BP Well - the grafana installation is kind of easy if you have a kubernetes cluster :} (And I so happen to have one ). In fact - the issue we have with our Weatherstation is the data itself. Grafana expects the data to be available, also in the past. So - whatever you do, you will have to provide long time storage. 2 options come to mind: 1. Mysql DB 2. RRD files
MySQL DB will require some sort of algorythm that will make sure the data hold for a while. From my experience, MySQL requires way too much storage to hold the data for 2 years. This is where the rrd part comes into play. It holds the data by default to up to 2 years (Of course, granularity will decrease depending on time into the past), and that data can be polled directly by a small agent that translates the RRD data sources into JSON data - that in turn can be polled directly by grafana. And that is the approach I use. Because I have the "infrastructure" to sustain it.
For the RRD files to be populated, all I do is poll the csv file (modification of the static text file provided out of the box), and with a script suitable for mrtg, I extract it via "./weather.sh rainmm" - and it returns me the rain as stored. I then use the grafana-rrd-server to provide the data out of the rrd-files to grafana. Note that one RRD file holding data for 2 years has a fixed file size of 203K.
I think if we want to use grafana or any other type of tool, and keep it small within our raspberry PI's, we'll have to store the data on the PI's in rrd format (or any other format suitable to keep data for a certain period of time without requiring too many resources). And then make sure we can provide that data using json format.
Maybe we should start thinking about that somehow
|
|
|
Post by frenchie on Dec 3, 2020 10:15:23 GMT -8
Very nice !
|
|
smurphy
Full Member
Posts: 169
Raspberry Pi: Yes
Other Device: many ...
|
Post by smurphy on Dec 10, 2020 2:43:19 GMT -8
Updated my grafana stuff. Happy now This is the weather-view for a week. hardest was how to visualize the wind direction - and opted for a plasma heat map. The more into the yellow, the more entries you have.
A shame the screens are not big enough 4K here
Still an issue with the correct table display for rain values. I however think that it's an error in the grafan plugin for rrd as rrd recomputes the values for a bigger delta, and grafane applies a delta based on the view. Maybe I'll find a way to circumvent that.
|
|
|
Post by SDL on Dec 10, 2020 8:35:22 GMT -8
Beautiful! You are driving a 4K Monitor with the Raspberry Pi at 4K resolution? Have a link for that? I've got a 4K TV hooked up displaying Solar Data for my SkyWeather2 prototype Solar System. Have you looked at using a wind rose to display wind direction? BP
|
|
smurphy
Full Member
Posts: 169
Raspberry Pi: Yes
Other Device: many ...
|
Post by smurphy on Dec 11, 2020 1:37:54 GMT -8
Lol. No. I just have a 4K Monitor at my Workstation. and under linux, everything is "local" through the network. The Wind-Rose I did not check. Maybe I'll check in the grafana Plugin direction. Even though the heat-map is very good (from experience).
Update: The Windrose plugins for grafana are either totally out of date, or not there yet to be subnmitted. Thing is - I have so much running on my systems that maintaining myself the code for these things is unfortunately not really an option.
|
|
|
Post by SDL on Dec 11, 2020 13:12:57 GMT -8
Totally get it. I ended up having to do some incredible data queries to get our windrose to work right!
BP
|
|
smurphy
Full Member
Posts: 169
Raspberry Pi: Yes
Other Device: many ...
|
Post by smurphy on May 26, 2021 0:07:15 GMT -8
So - I finally managed, while migrating my entire setup, to change Monitoring and graphing entirely. What I did was patch SkyWeather.py to provide a metric file, actually dump a metric file in the static directory. This will be served by lighthttpd so prometheus can actually poll it.
So - the path to SkyWeather:
# write weather stats out to Metrics (prometheus) def writeWeatherMetrics():
f = open("/home/pi/SDL_Pi_SkyWeather/static/metrics", "w") raintext = "# HELP Total rain in mm since station restart\n# TYPE weather_rain counter\n" f.write(raintext + "weather_rain " + str(totalRain) + '\n') lightningcnt = "# HELP Detected lightning discharges \n# TYPE weather_as3935_Lightning_Count counter\n" f.write(lightningcnt + "weather_as3935_Lightning_Count " + str(as3935LightningCount) + '\n') lastinterrupt = "# HELP lightning invalidators \n# TYPE weather_as3935_Last_Interrupt counter\n" f.write(lastinterrupt + "weather_as3935_Last_Interrupt " + str(as3935LastInterrupt) + '\n') lastdistance = "# HELP Distance of last lightning strike \n# TYPE weather_as3935_Last_Distance gauge\n" f.write(lastdistance + "weather_as3935_Last_Distance " + str(as3935LastDistance) + '\n') windspeed = "# HELP Last measured wind speed in Kilometers per hour \n# TYPE weather_current_Wind_Speed gauge\n" f.write(windspeed + "weather_current_Wind_Speed " + str(currentWindSpeed) + '\n') windgust = "# HELP highest measured wind gust in Kilometers per hour since last measurement\n# TYPE weather_current_Wind_Gust gauge\n" f.write(windgust + "weather_current_Wind_Gust " + str(currentWindGust) + '\n') insidetemp = "# HELP Inside temperature in degrees celcius\n# TYPE weather_bmp180_Temperature gauge\n" f.write(insidetemp + "weather_bmp180_Temperature " + str(bmp180Temperature) + '\n') insidepressure = "# HELP Inside pressure in hPa\n# TYPE weather_bmp180_Pressure gauge\n" f.write(insidepressure + "weather_bmp180_Pressure " + str(bmp180Pressure) + '\n') insidealtitude = "# HELP Station altitude in m\n# TYPE weather_bmp180_Altitude gauge\n" f.write(insidealtitude + "weather_bmp180_Altitude " + str(bmp180Altitude) + '\n') sealevel = "# HELP Sea-level altitude in m\n# TYPE weather_bmp180_Sea_Level gauge\n" f.write(sealevel + "weather_bmp180_Sea_Level " + str(bmp180SeaLevel) + '\n') outtemp = "# HELP Temperature in degree celcius\n# TYPE weather_outside_Temperature gauge\n" f.write(outtemp + "weather_outside_Temperature " + str(outsideTemperature) + '\n') outhumidity = "# HELP Humidity percent\n# TYPE weather_outside_Humidity gauge\n" f.write(outhumidity + "weather_outside_Humidity " + str(outsideHumidity) + '\n') currwinddir = "# HELP Current wind direction in degrees\n# TYPE weather_current_Wind_Direction gauge\n" f.write(currwinddir + "weather_current_Wind_Direction " + str(currentWindDirection) + '\n') currwinddirvolt = "# HELP Current wind direction voltage in V\n# TYPE weather_current_Wind_Direction_Voltage gauge\n" f.write(currwinddirvolt + "weather_current_Wind_Direction_Voltage " + str(currentWindDirectionVoltage) + '\n') htutemp = "# HELP HTU Temperature in degree celcius\n# TYPE weather_HTU_temperature gauge\n" f.write(htutemp + "weather_HTU_temperature " + str(HTUtemperature) + '\n') htuhumidity = "# HELP HTU humidity in percent\n# TYPE weather_HTU_humidity gauge\n" f.write(htuhumidity + "weather_HTU_humidity " + str(HTUhumidity) + '\n') visiblelight = "# HELP Visible light in Lux \n# TYPE weather_Visible_light gauge\n" f.write(visiblelight + "weather_Visible_light " + str(SunlightVisible) + '\n') irlight = "# HELP Sunlight IR in Lux \n# TYPE weather_Sun_light_IR gauge\n" f.write(irlight + "weather_Sun_light_IR " + str(SunlightIR) + '\n') uvlight = "# HELP Ultra violet light in Lux \n# TYPE weather_Sun_light_UV gauge\n" f.write(uvlight + "weather_Sun_light_UV " + str(SunlightUV) + '\n') uvlindex = "# HELP Ultra violet light index \n# TYPE weather_Sun_light_UV_Index gauge\n" f.write(uvlindex + "weather_Sun_light_UV_Index " + str(SunlightUVIndex) + '\n') outdooraqi = "# HELP Air Quality index in ppm \n# TYPE weather_Outdoor_AirQuality_Sensor_Value gauge\n" f.write(outdooraqi + "weather_Outdoor_AirQuality_Sensor_Value " + str(state.Outdoor_AirQuality_Sensor_Value) + '\n') f.close() and in the scheduler section add this to be executed
scheduler.add_job(writeWeatherMetrics, 'interval', seconds=60)
Install the light httpd with:
apt install lighttpd and configure it to point to the location for the metric file. I did dump everything into the /etc/lighttpd.conf file:
server.document-root = "/home/pi/SDL_Pi_SkyWeather/static"
server.port = 80 server.errorlog = "/var/tmp/log/lighttpd_error.log" server.stat-cache-engine = "disable"
mimetype.assign = ( ".html" => "text/html", ".txt" => "text/plain", ".jpg" => "image/jpeg", ".png" => "image/png" )
static-file.exclude-extensions = ( ".fcgi", ".php", ".rb", "~", ".inc" ) index-file.names = ( "index.html" )
server.username = "www-data" server.groupname = "www-data"
From now on - I should be able to poll all files from the static directory (if I know the file names) using curl.
Prometheus setup (This assumes you have it running). Add the following snippet to the configuration file:
- job_name: 'weather-metrics' static_configs: - targets: ['10.0.0.220:80']
The IP is fake, but that's the IP of your weather Station.
reload prometheus for reading the changed config, and check if it sees any errors in the status page. If not - you're good to go.
Go to grafana (Assuming that grafana uses prometheus as data-source, make sure it uses "Windrose panel" plugin,
In the end - this is what you get:
The one thing I have to fix, is the rain display. It is a rain-gauge, but it comes in as counter. Grafana and prometheus seem to have an issue to scale the data right. But that may come once I get the details sorted out.
|
|
|
Post by SDL on May 27, 2021 11:23:39 GMT -8
Awesome!
BP
|
|
smurphy
Full Member
Posts: 169
Raspberry Pi: Yes
Other Device: many ...
|
Post by smurphy on Sept 14, 2021 6:24:11 GMT -8
So - I was not happy with prometheus to collect the data. Problem is mostly thatn it aggregates/averages the data and finding the right configuration for grafana is a pain. On top of that, it will phase out old data too. Which makes it difficult to actually look at weather data in the past. I went over to a more basic approach. Write all data into a mysql-DB, and configured grafana to show the data. You can see details on my web-site in the build-blogs: www.solsys.org/dyntbl.php?mod=docs&upd_menu_id=53&op=View&id=1Check out at the end to see the current graphing. PS: I do not write the data directly into a DB. Problem is that python will bail out if a DB write error occurs.
|
|
|
Post by SDL on Sept 15, 2021 12:08:39 GMT -8
Can't you build a try except black around the DB write error?
Maybe I don't understand the problem.
You do really good work. I love the Graphana board!
BP
|
|
smurphy
Full Member
Posts: 169
Raspberry Pi: Yes
Other Device: many ...
|
Post by smurphy on Sept 15, 2021 23:20:08 GMT -8
When the network is not running (interruptions with WiFi are quite common), the python script tries to force the connection. Most of the time it works. But sometimes, it only returns garbage. Even the logs (nohup) get filled with binary garbage and it stops. As this happens only randomly, it is very difficult to troubleshoot. So I didn't bother and went for something that works And thx for the dashboards. They start to be usable
|
|
|
Post by joejvg on Oct 11, 2023 12:33:58 GMT -8
Hello.
I was curious if use of your code I think produces an excellent page of graphically represented weather data has a copyright. I am a developer and have a customer's website I developed along with a designer I partner with that provides daily and yearly weather data but only in tabular format and your page provides a beautifully done representation in a visual format that is perfect for my needs. I would need to rewrite it in PHP using a MySQL database but I would ensure a credit along with a link to your work is on the page.
Please let me know.
Sincerely, Stony Creek Consulting, LLC Joseph J. Geller joejvg@gmail.com
Sincerely
|
|