A Watchdog for SkyWeather

impactoz
Full Member

Posts: 142
Raspberry Pi: Yes

A Watchdog for SkyWeather Oct 30, 2020 15:52:22 GMT -8

Quote

Post by impactoz on Oct 30, 2020 15:52:22 GMT -8

So in my ever evolving quest of a remote system and reliability... I noticed my test bed did hang once while getting it all properly configured - which would have been disastrous for me - as it could have been weeks before I actually physically visit the unit. On the hang - there was no way to connect with VNC / SSH or anything to reboot it...

So reluctantly I started investigating a hardware watchdog (knowing that the Pi has something - but unreliable for all situations)...

Looked at the switchdoc solution... yup thats what I want - but it just seemed messy - soldering / wires etc... So kept on googling...

Found this elegant solution! PiWatcher -> www.omzlo.com/articles/the-piwatcher

Love the simplicity - its a miniture hat - just connects tto 6 pins - Pins 2/4/6 (+5V/+5V/Gnd) and Pins 1/3/5 (+3V/GPIO2/GPIO3). The GPIO Pins being also really used in this situation as I2CA SDA and I2CA SDL

SO its controllable using I2C - something we are all familiar with on the SkyWeather

Lots of features on this unit - but the only one I am interested in is to power cycle the Raspberry Pi if it misses a heart beat. Which sounds simple enough - pat the dog routine could just send a heart beat, and if the SkyWeather hangs, no pat the dog, no heartbeat, and the piwatcher cuts the power to reboot the PI - Perfect!

What I like about this unit is the thinking out of the circle... instead of using the GPIO pins to receive power - its actually powering the unit through those pins - its kind of reversed - and I just think that was brilliant thinking so that pi watcher gets the opportunity to control the power to the raspberry pi... And the unit is so cheap!!! And very easy to install and use!

Maybe SDL could redesign their hardware watchdog to a similar design !!!

I need to investigate what I2C channels its using in the software / how it works to see how compatible it is with the SkyWeather - first glance seems no issues - PiWatcher is not compatible with Pi4..... and it uses at I2C address 0x62 which I think is safe... Will keep looking at the code....

But I wanted to mention it... I just thought it was a brilliant design!

Last Edit: Oct 30, 2020 16:00:34 GMT -8 by impactoz

My SkyWeather Stations;
www.wunderground.com/dashboard/pws/I90582147
skyweather.weatherstem.com/malmsbury

Currently offline - be back Late Nov 19
www.wunderground.com/dashboard/pws/IMELBO387
skyweather.weatherstem.com/thornbury

SDL
Global Moderator

Posts: 5,655
Raspberry Pi: Yes
Other Device: PC, arduino, beagleBone

A Watchdog for SkyWeather Oct 31, 2020 7:07:42 GMT -8

Quote

Post by SDL on Oct 31, 2020 7:07:42 GMT -8

I just read over the PiWatcher Design and you have given me an idea for a watchdog board that could be used for a future system. We have to have a solution that works for ESP32, Arduino and Raspberry Pi.

I'm going to buy one of these and play with it. Very interesting. I like the I2C control. It does violate on of my rules of watchdog reliability. You should not have ANY way that the software can stop the watchdog timer, except by patting the watchdog.

BP

impactoz
Full Member

Posts: 142
Raspberry Pi: Yes

A Watchdog for SkyWeather Nov 20, 2020 21:41:45 GMT -8

Quote

Post by impactoz on Nov 20, 2020 21:41:45 GMT -8

So cheap - I bought one of these to try out... Finally arrived - think Greek Airmail = Seamail.... but at under $10 landed its not bad...

At first I plugged it in sandwiched between the Raspberry Pi board and the SDL Board - boots fine... So it gets power all good... Then I try to run Skyweather - and it all fails... Cant find the I2C stuff...

The reason I plugged it in between the two boards, as the connectors are in the way on the SDL board - but luckily you do get a small extender (Id probably be paying $5 for that - so bargain).

Using the small edge conneector, the watchdog board sits above the SDL board - and the raspberry Pi powers beautifully... Run the SkyWeather app - all works fine.... so far there are no conflicts on I2c - its all working well....

Sandwiching it between the two boards must have meant not all the connectors were properly making the right connection...

The supplied software installed, However at the moment I am getting a read error on the unit with the supplied software... Reboot the Pi - it works... run the supplied piwatcher status program and get an error Failed to read i2c data from register 0x00
ERR
. This is with all the SDL hardware installed but have not run any of the software... Going to have to look at the source code of piwatcher program to see if I can work out what it does - might be a hardware conflict somehow with the SDL I2C stuff

I need to work out now how to send the heartbeat from within SkyWeather - what is the command to what channel... A little bit of experimenting...

I have noticed in the last few weeks - 2 occasions when this watchdog will help.
- Once the whole raspberry pi just hang - could not connect to it, could not run any commands - it was dead to the world, but was still alive - but doing nothing... So the heartbeat would have not been sent, and the Pi should have done a reboot - which is exactly what I did...

- The other occasion - which was strange... and I dont have the full error log any more - but it started erroring on the schedules, telling me that there was too many things scheduled, the Pi kept running, and I would get the scheduling error again and again each time it ran itself and retried to reschedule itself. BUT the sensors did not run, it was not taking readings any more, but would reschedule itself again and again...

So in the second instant - the routine sampleAndDisplay was attempted to be scheduled, but never ran - yet the Pi was continue to run SkyWeather - I am thinking that this is probably the best routine to send the heartbeat from - I currently have that scheduled for every 300 seconds... so I can live with a wait of 5 mins for a reset...

So far so good - now to integrate it!

Last Edit: Nov 21, 2020 2:17:24 GMT -8 by impactoz

My SkyWeather Stations;
www.wunderground.com/dashboard/pws/I90582147
skyweather.weatherstem.com/malmsbury

Currently offline - be back Late Nov 19
www.wunderground.com/dashboard/pws/IMELBO387
skyweather.weatherstem.com/thornbury

impactoz
Full Member

Posts: 142
Raspberry Pi: Yes

A Watchdog for SkyWeather Nov 21, 2020 3:45:10 GMT -8

Quote

Post by impactoz on Nov 21, 2020 3:45:10 GMT -8

I had a bent pin 3 - which is SDLA - so no wonder the I2C could not see it. All fixed and everything works - no interference in the SDL software - all working fine...

So in my startup I have put in a couple new commands before running the SkyWeather software;
piwatcher wake 30

Which should tell the Pi, after shutting down, reapply the power after 30 seconds, and

piwatcher watch 330

Which should tell the watchdog to shutdown by removing power if a heartbeat is not sensed for 330 Seconds...

I have put the command in SkyWeather.py in the SampleandDisplay routine which should send the heartbeat...
os.system("piwatcher status")

All looking good now - lets see what happens over the next week - do I get better reliability

?

My SkyWeather Stations;
www.wunderground.com/dashboard/pws/I90582147
skyweather.weatherstem.com/malmsbury

Currently offline - be back Late Nov 19
www.wunderground.com/dashboard/pws/IMELBO387
skyweather.weatherstem.com/thornbury

SDL Global Moderator Posts: 5,655 Raspberry Pi: Yes Other Device: PC, arduino, beagleBone	A Watchdog for SkyWeather Nov 21, 2020 8:20:13 GMT -8 Quote Select Post Deselect Post Link to Post Member Give Gift Back to Top Post by SDL on Nov 21, 2020 8:20:13 GMT -8 Outstanding report. That's for filling us in! BP

Post by impactoz on Oct 30, 2020 15:52:22 GMT -8

Post by SDL on Oct 31, 2020 7:07:42 GMT -8

Post by impactoz on Nov 20, 2020 21:41:45 GMT -8

Post by impactoz on Nov 21, 2020 3:45:10 GMT -8

Post by SDL on Nov 21, 2020 8:20:13 GMT -8

Quick Reply