n7qnm
Junior Member
Posts: 80
|
Post by n7qnm on Jan 8, 2022 14:00:54 GMT -8
I'm using MQTT to publish data to weewx; works great; but, I'm finding MQTT fails silently after somewhere between 1 and 4 days. The call to mqtt publish returns; but no data shows up at the broker. Restarting SkyWeather2 will "clear" problem.
I'm pretty sure it's because the paho mqtt publish method expects a persistent connection AND uses callbacks. There's a "publish single" method that the paho docs say does NOT require the callbacks and just "publishes and disconnects". So, I replaced the calls "publish" calls in wirelessSenors.py and publishMQTT.py with "publish.single"; it's been "cooking" for about a day now and appears to work. I'll report back after a week or so (I want to see if it survives a reboot).
I think the "ultimate" solution will be to add the callbacks and keep a persistent connection (to avoid connect/disconnects and also do better error handling; but I wanted to make sure this solves the problem first
Here's "git diff's" on the code - I've hardcoded the broker hostname for now; but that can be had in the config file.
diff --git a/wirelessSensors.py b/wirelessSensors.py
index 9b4f344..25943ea 100644
--- a/wirelessSensors.py
+++ b/wirelessSensors.py
@@ -22,8 +22,9 @@ import time
import os
import signal
import traceback
-sys.path.append('./SDP_Pi_HM3301/aqi')
+sys.path.append('./SDL_Pi_HM3301/')
import aqi
+import paho.mqtt.publish as publishSingle
import MySQLdb as mdb
# ---------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -68,7 +69,7 @@ def mqtt_publish_single(message, topic):
topic = '{0}/{1}'.format("skyweather2", topic)
#print ("topic=", topic)
try:
- state.mqtt_client.publish(topic, message)
+ publishSingle.single(topic, message, hostname = "weewx")
except:
traceback.print_exc()
print('Mosquitto not available')
===============================================
diff --git a/publishMQTT.py b/publishMQTT.py
index 60d56cf..f3aa689 100644
--- a/publishMQTT.py
+++ b/publishMQTT.py
@@ -3,12 +3,13 @@ import config
import state
import paho.mqtt.client
+import paho.mqtt.publish as publishMQTT
def publish():
if (config.SWDEBUG):
print("--->Sending MQTT Packet<---")
- state.mqtt_client.publish("skyweather2/state", state.StateJSON)
+ publishMQTT.single("skyweather2/state", state.StateJSON, hostname = "weewx")
|
|
|
Post by SDL on Jan 8, 2022 15:20:53 GMT -8
I'm looking forward to your results. Interesting problem.
BP
|
|
|
Post by Jason on Jan 10, 2022 10:58:09 GMT -8
I've been using the paho.mqtt.publish.single method for quite awhile in my SkyWeather2 setup without issues for exactly the reason mentioned. Network blips, etc. would knock out my MQTT messages. The overhead of opening/closing the MQTT connection in a local network for me is so minimal that I decided not to spend time figuring out how to handle the callbacks necessary to properly implement the pho.mqtt.publish method. Have not had any issues since. Sorry I didn't share when I discovered this issue and fixed it.
Thanks,
Jason
|
|
|
Post by SDL on Jan 10, 2022 17:17:13 GMT -8
Jason,
Publish the code and I'll add it in V028.
BP
|
|
|
Post by Jason on Jan 11, 2022 5:38:19 GMT -8
In the script below, you'll see where I attempted to troubleshoot the issue using try/except to no avail. This could've been a class but meh lol. I would definitely consider making the client a class if there is a desire to support the various callbacks etc as it would make the client much more portable within the code base.
import config import state
from paho.mqtt import publish import json
''' def publish(topic, payload): if config.SWDEBUG: print("--->Sending MQTT Packet<---") # state.mqtt_client.publish("skyweather2/state", state.StateJSON) try: state.mqtt_client.publish(topic, payload) except ConnectionError as e: print('MQTT unavailable') except OSError as oe: print('OS Error') '''
def mqtt_publish_single(topic, payload): # print('Current topic: {0}'.format(topic)) # debug # print('Current MQTT broker: {0}'.format(config.MQTT_Server_URL)) # debug try: publish.single( topic=topic, payload=json.dumps(payload), hostname=config.MQTT_Server_URL, port=config.MQTT_Port_Number ) except ConnectionError as e: print(e) except OSError as oe: print(oe)
Thanks,
Jason
|
|
n7qnm
Junior Member
Posts: 80
|
Post by n7qnm on Jan 11, 2022 16:27:33 GMT -8
Thanks, Jason!
The code I'm running is in the diffs in my original message. I can send the entire files if you'd like; I want to do a little more to "clean it up" (one publish function called by both SkyWeather2 and wirelessSensors, grab the host from the config and maybe a try/except block). When are you planning on publishing V28?
I also posted a question about single vs full implementation in the Paho forum, no response yet; I'd kinda like to wait for any response there.
The reason the try/accept didn't return anything useful is because the initial call to publish "succeeds", since that that does is pass the message to the ASYNCHRONOUS paho message handler. W/O the callbacks, there's no place for the message handler to report back success or failure.
Let me know on V28 and I can do the cleanup and send or post the whole file (or just the diffs)
Clay
|
|
|
Post by SDL on Jan 11, 2022 18:55:04 GMT -8
Clay,
Post the whole file and I can do the diffs and look at the changes (and understand them!).
I'd guess V28 will be in March at the speed that things are going. Don't quote me back to Laurie!
BP
|
|
|
Post by Jason on Jan 12, 2022 3:43:13 GMT -8
I changed my SkyWeather2 script to publish ALL readings to MQTT and ignore the built in functionally for publishing MQTT data every 15 minutes (I think?!?). Because I have quite a bit more data, I didn’t see value in worrying about a miss here or there so didn’t bother with the callback. As an aside, I did start to write a class for managing the connection and messaging for a higher volume use case and can share that script as a starting point if you’d like.
Thanks,
Jason
|
|
n7qnm
Junior Member
Posts: 80
|
Post by n7qnm on Jan 12, 2022 10:02:45 GMT -8
BP - OK, I'll do some more cleanup and then send you the result.
Jason, yes, I'd be interested in looking at the class - I was thinking along the same lines. So, in your current implementation, did you just put calls to publish.single in each of the sensor files (like wirelessSensor) and the "main"?
I'm also working on a few other changes; I want to put the "raw" particle counts in "state" (or at least publish 'em to MQTT); and then I'm gonna use the wireless AQ sensor as my INDOOR sensor (no panel/batteries, just power from a wall-wart, or perhaps even an ALkaline pack. Also working on some heat dissipation ideas; I've got the GeekWorm case on order, and then I'm looking at "container" heat sink, small heat pipes and phase change material.
|
|
n7qnm
Junior Member
Posts: 80
|
Post by n7qnm on Jan 14, 2022 9:59:37 GMT -8
I'm now 5 days (and a couple of reboots) and MQTT publish.single has been working non-stop; so I think we can call that issue quashed.
My plan is to do something along the same lines as Jason and publish everything. I'm going to pull the scheduled "MQTT Publish" and the publish calls for the different sensors, and just put one call to publish state after the BuildJSON call. That way, any time state gets rebuilt, a new message will get published.
Also, we'll be publishing SI or at least US "normalized unit" values, NOT the raw data from the sensors.
I also noticed that some of the "sensor" routines modify state but do NOT call the semaphore code, I'll fixe that.
Finally, I want to add "raw" particle counts to state for both the WS AQI and "internal" AQI code - I want to use the internal one for my outdoor sensor, and the WS one as an internal sensor.
I'm hoping to get the changes made this weekend (to SkyWeather, state and the various "sensor" files). I'll post 'em as soon as I get it tested. Should I just upload the whole source file as a message here; or is there a "better" way.
Clay
|
|
|
Post by Jason on Jan 14, 2022 15:04:36 GMT -8
I never posted the stub I promised. It's been crazy here the last couple of days so apologies. After digging it up, it is not as far along as I thought. I think i'm going to do some work on it and eventually share.
Thanks,
Jason
|
|
|
Post by Jason on Jan 16, 2022 13:34:45 GMT -8
BP - OK, I'll do some more cleanup and then send you the result. Jason, yes, I'd be interested in looking at the class - I was thinking along the same lines. So, in your current implementation, did you just put calls to publish.single in each of the sensor files (like wirelessSensor) and the "main"?I'm also working on a few other changes; I want to put the "raw" particle counts in "state" (or at least publish 'em to MQTT); and then I'm gonna use the wireless AQ sensor as my INDOOR sensor (no panel/batteries, just power from a wall-wart, or perhaps even an ALkaline pack. Also working on some heat dissipation ideas; I've got the GeekWorm case on order, and then I'm looking at "container" heat sink, small heat pipes and phase change material. I realized I missed your question while scanning through. I added it to publishMQTT.py which gets loaded by SkyCamera.py, SkyWeather2.py, wiredSensors.py, DustSensor.py, and wirelessSensors.py. I haven't spent any time trying to come up with a more elegant solution and am not sure it is possible with the jobs running in the background through APScheduler. However, I'll readily admit I have not spent a whole lot of time researching whether a class could be passed to the background jobs. My guess is no but am not sure. Thanks, Jason
|
|
n7qnm
Junior Member
Posts: 80
|
Post by n7qnm on Jan 17, 2022 16:33:01 GMT -8
yeah - I just got through "cleaning up" my stuff, at least some. I took a slightly different approach. I really want the "processed" values, so first thing I tried was to modify buildJSON.py to just "single publish" the state whenever it got called. buildJSON is invoked very early on in SkyWeather2 and for some reason I don't understand yet, when I did that, publish.single "hung" inside paho (only way out was control-c and I haven't analyzed what happened yet. So, what I did as an interim was to put a call to publishMQTT (and all the appropriate defs and stuff) in wiredSensors and wirelessSensors
|
|
n7qnm
Junior Member
Posts: 80
|
Post by n7qnm on Mar 29, 2022 19:41:32 GMT -8
When I was having issues with the cameras, I "reverted" back to the "master" 27.3 code; but, the MQTT publish issue has popped back up again, no doubt aggravated by the the fact that I'm doing WSAQI every 5 minutes. I'm gonna see if I can get my changes put back in tomorrow and will keep you posted.
I also ordered some solar panels and 8 of the ESP32 CAM boards. I'll keep you posted on THAT project.
Any ETA on 28?
|
|
|
Post by SDL on Mar 31, 2022 9:12:34 GMT -8
V28 is scheduled for May right now. We are collecting bugs and features. The main new feature will be support for the upcoming WeatherSense RadSense geiger counter that we are developing. Kickstarter will start in a couple of weeks. Here's a picture of the RadSense prototype: BP
|
|