maestro
New Member
Posts: 13
Raspberry Pi: Yes
|
Post by maestro on Feb 28, 2021 10:51:50 GMT -8
SkyWeather2.py has been giving me this error lately. It only seems to happen after several hours of operation, though not a consistent amount of time. I tried disconnecting the SkyCamera and this seemed to stop the error from happening but then I wasn't getting updates to WeatherStem.
Tick! The time is: 2021-02-23 19:26:57.100021
mySkyCameraText= SkyWeather2 V022 23-Feb-2021 19:27:00 Wind Speed: 0.0MPH Wind Gust: 0.0MPH Temp: 50.2 F
------->Sea Level 1013.6993
Tick! The time is: 2021-02-23 19:27:57.100189
trying database
CPUT= 53.069
trying database
mySkyCameraText= SkyWeather2 V022 23-Feb-2021 19:28:00 Wind Speed: 0.0MPH Wind Gust: 0.0MPH Temp: 50.3 F
------->Sea Level 1013.7505
Tick! The time is: 2021-02-23 19:28:57.099917
mySkyCameraText= SkyWeather2 V022 23-Feb-2021 19:29:00 Wind Speed: 0.0MPH Wind Gust: 0.0MPH Temp: 50.3 F
Tick! The time is: 2021-02-23 19:29:57.099871
Execution of job "takeSkyPicture (trigger: interval[0:01:00], next run at: 2021-02-23 19:29:57 EST)" skipped: maximum number of running instances reached (1)
Tick! The time is: 2021-02-23 19:30:57.099842
Execution of job "takeSkyPicture (trigger: interval[0:01:00], next run at: 2021-02-23 19:30:57 EST)" skipped: maximum number of running instances reached (1)
Tick! The time is: 2021-02-23 19:31:57.099885
|
|
|
Post by SDL on Mar 2, 2021 10:01:43 GMT -8
Sounds like the takeSkyPicture process is hanging in the data transfer. Could you turn SWDEBUG on and replicate the problem? Then post the results.
BP
|
|
maestro
New Member
Posts: 13
Raspberry Pi: Yes
|
Post by maestro on Mar 4, 2021 9:09:52 GMT -8
Ok, I can't see anything else wrong in the output and no other errors. The "takeSkyPicture..." error seems to happen during each Tick, but at a different interval each time. I didn't catch the first occurrence of the error this time. I'll try again to see if something different happens on the first error.
Tick! The time is: 2021-03-04 11:49:13.403288
-------------
Current State
-------------
-------------
------Patting The Dog-------
latest MainSensor Reading= 2021-03-04 11:49:13
MainDeviceNumber= 0
OutdoorTemperature = 16.44
OutdoorHumidity = 31
latest Indoor Sensor Reading= 2021-03-04 11:48:33
IndoorDeviceNumber= 1
Execution of job "takeSkyPicture (trigger: interval[0:01:00], next run at: 2021-03-04 11:49:13 EST)" skipped: maximum number of running instances reached (1)
IndoorTemperature = 23.78
IndoorHumidity = 37
Rain60Minutes = 0.0
SunlightVisible = 37425
SunlightUVIndex = 2.5
WindSpeed = 0.0
WindGust = 0.0
WindDirection = 214
TotalRain = 120.9
BarometricTemperature = 27.45
BarometricPressure = 101.31905
Altitude = 14.0
BarometricPressureSeaLevel = 101.48736
BarometricTemperature = 27.45
barometricTrend = False
pastBarometricReading = 101.32161
AQI = 0.0
Hour24_AQI = 0.0
Main Battery Status = OK
CPU Temperature = 40.407
-------------
runRainbow = False
flashStrip = False
runOLED = True
-------------
Last_Event = My Last Event
-------------
batteryVoltage 0
batteryCurrent 0
solarVoltage 0
solarCurrent 0
loadVoltage 0
loadCurrent 0
batteryPower 0
solarPower 0
loadPower 0
batteryCharge 0
SolarMAX Inside Temperature 0.0
SolarMAX Inside Humidity 0.0
SolarMAX Last Received None
-------------
-------------
-------------
fanState = False
-------------
------Patting The Dog-------
Processing F016TH data
This is the raw data: {"time" : "2021-03-04 11:49:26", "model" : "SwitchDoc Labs F016TH Thermo-Hygrometer", "device" : 98, "modelnumber" : 5, "channel" : 1, "battery" : "OK", "temperature_F" : 74.800, "humidity" : 37, "mic" : "CRC"}
processing FT020T Data
This is the raw data: {"time" : "2021-03-04 11:49:28", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 0, "gustwindspeed" : 3, "winddirection" : 295, "cumulativerain" : 1209, "temperature" : 1017, "humidity" : 31, "light" : 37981, "uv" : 28, "mic" : "CRC"}
------Patting The Dog-------
------Patting The Dog-------
processing FT020T Data
This is the raw data: {"time" : "2021-03-04 11:49:44", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 0, "gustwindspeed" : 3, "winddirection" : 275, "cumulativerain" : 1209, "temperature" : 1018, "humidity" : 31, "light" : 36032, "uv" : 21, "mic" : "CRC"}
processing FT020T Data
This is the raw data: {"time" : "2021-03-04 11:49:44", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 0, "gustwindspeed" : 3, "winddirection" : 275, "cumulativerain" : 1209, "temperature" : 1018, "humidity" : 31, "light" : 36032, "uv" : 21, "mic" : "CRC"}
------Patting The Dog-------
processing FT020T Data
This is the raw data: {"time" : "2021-03-04 11:50:00", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 0, "gustwindspeed" : 0, "winddirection" : 131, "cumulativerain" : 1209, "temperature" : 1019, "humidity" : 31, "light" : 37399, "uv" : 25, "mic" : "CRC"}
------Patting The Dog-------
Tick! The time is: 2021-03-04 11:50:13.403369
-------------
Current State
-------------
-------------
------Patting The Dog-------
latest MainSensor Reading= 2021-03-04 11:50:01
MainDeviceNumber= 0
OutdoorTemperature = 16.61
OutdoorHumidity = 31
latest Indoor Sensor Reading= 2021-03-04 11:49:26
IndoorDeviceNumber= 1
IndoorTemperature = 23.78
IndoorHumidity = 37
Rain60Minutes = 0.0
SunlightVisible = 37399
SunlightUVIndex = 2.5
WindSpeed = 0.0
WindGust = 0.0
WindDirection = 131
TotalRain = 120.9
BarometricTemperature = 27.49
BarometricPressure = 101.31459
Altitude = 14.0
BarometricPressureSeaLevel = 101.4829
BarometricTemperature = 27.49
barometricTrend = False
pastBarometricReading = 101.32161
AQI = 0.0
Hour24_AQI = 0.0
Main Battery Status = OK
CPU Temperature = 40.407
-------------
runRainbow = False
flashStrip = False
runOLED = True
-------------
Last_Event = My Last Event
-------------
batteryVoltage 0
batteryCurrent 0
solarVoltage 0
solarCurrent 0
loadVoltage 0
loadCurrent 0
batteryPower 0
solarPower 0
loadPower 0
batteryCharge 0
SolarMAX Inside Temperature 0.0
SolarMAX Inside Humidity 0.0
SolarMAX Last Received None
-------------
-------------
-------------
fanState = False
-------------
Execution of job "takeSkyPicture (trigger: interval[0:01:00], next run at: 2021-03-04 11:50:13 EST)" skipped: maximum number of running instances reached (1)
processing FT020T Data
This is the raw data: {"time" : "2021-03-04 11:50:16", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 4, "gustwindspeed" : 10, "winddirection" : 240, "cumulativerain" : 1209, "temperature" : 1019, "humidity" : 31, "light" : 37333, "uv" : 22, "mic" : "CRC"}
processing FT020T Data
This is the raw data: {"time" : "2021-03-04 11:50:16", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 4, "gustwindspeed" : 10, "winddirection" : 240, "cumulativerain" : 1209, "temperature" : 1019, "humidity" : 31, "light" : 37333, "uv" : 22, "mic" : "CRC"}
Processing F016TH data
This is the raw data: {"time" : "2021-03-04 11:50:19", "model" : "SwitchDoc Labs F016TH Thermo-Hygrometer", "device" : 98, "modelnumber" : 5, "channel" : 1, "battery" : "OK", "temperature_F" : 74.800, "humidity" : 37, "mic" : "CRC"}
|
|
|
Post by SDL on Mar 4, 2021 15:08:09 GMT -8
We need to see the data showing the SkyCam failing. The first time.
BP
|
|
dow4hurst
Full Member
SkyWeather2 Newbie
Posts: 117
|
Post by dow4hurst on Mar 11, 2021 14:59:04 GMT -8
KernelLinux SwitchDocLabs 5.10.17-v7l+ #1403 SMP Mon Feb 22 11:33:35 GMT 2021 armv7l GNU/Linux Firmwaresudo vcgencmd version Feb 25 2021 12:11:39 Copyright (c) 2012 Broadcom version 564e5f9b852b23a330b1764bcf0b2d022a20afd0 (clean) (release) (start_x) SDL softwareSkyWeather2 Weather Station Version 023 - SwitchDoc Labs Added the suggested edit by gorddel: The code in wirelessSensors.py (V023) to suppress the duplicate input lines from the FT020T station appears to be incomplete. I fixed it by adding the following line after line 90: 90 return "" 91 lastFT020TTimeStamp = var["time"] 92 if (config.MQTT_Enable == True):SDR/rtl-sdr is updated and recompiled as of March 10th from github. Errors found/var/log/daemon.log Mar 11 16:53:30 SwitchDocLabs rc.local[478]: Execution of job "takeSkyPicture (trigger: interval[0:01:00], next run at: 2021-03-11 16:53:30 EST)" skipped: maximum number of running instances reached (1) Error in nohup.out at line 2618 first shows up just after the previous F016TH measurement where the time was "2021-03-11 12:54:57" Checksum Error! 2 Checksum Errors! The nohup.out is attached. There were 239 Ticks between the first checksum error in the nohup.out file and the /var/log/daemon.log error about maximum number of running instances. I don't know if that is significant, but the updates to weatherstem stopped exactly when the daemon.log message showed up. Attachments:nohup.out (1.52 MB)
|
|
|
Post by SDL on Mar 11, 2021 16:35:06 GMT -8
Looks like you have some kind of a networking problem on your Pi.
exception in blynkTerminalUpdate HTTPConnectionPool(host='blynk-cloud.com', port=80): Max retries exceeded with url: /5pWJ4JeVhDdt9bmChfIVpWDrFAeh_kkj/update/V32 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb4a896f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
This is not good.
I wonder if the SkyCamera picture is running into the same network problem and that is why it hangs.
Regarding the checksum errors, different thread (HM3301). The dust sensor will occasionally throw an error, which is dealt with but doesn't affect the WeatherSTEM.
I'm thinking that it is hanging in transfer. Specifically in this line:
r = requests.post(url = API_ENDPOINT, json = data)
Put a print statement before this (such as print("Before requests.post") ) and run this test again until it fails. I am going to log this as the error as a bug, but not going to fix it until we verify it. I think it needs a time out.
I'll add the V023 but into the error list too. Thanks!
BP
|
|
maestro
New Member
Posts: 13
Raspberry Pi: Yes
|
Post by maestro on Mar 12, 2021 21:10:58 GMT -8
I just can't seem to catch this error in the act. It keeps recurring while I'm asleep or at work and I miss the first instance. I just updated to v024 and I'll keep trying to capture it.
|
|
|
Post by Jason on Mar 13, 2021 9:25:35 GMT -8
You can wrap the POST request in a try/except block something like this:
try: r = requests.post(url = API_ENDPOINT, json = data, timeout = 10) except requests.exceptions.Timeout as rto: # do something except requests.exceptions.ConnectionError as rce: # do something
The snippet above tells the requests library to abort the POST request if a response has not been received within 10s. There are a couple of options you could add to the exception handler such as sys.exit(). sys.exit() will abort the running program entirely which should prevent your log files from filling up. Another option could be to write your own log file with the exception information so you don't have to swim through tons of text to find the exception.
Jason
|
|
dow4hurst
Full Member
SkyWeather2 Newbie
Posts: 117
|
Post by dow4hurst on Mar 13, 2021 11:47:13 GMT -8
BP, Captured the data you wanted. At line 110128 in the nohup.out is the last time "Before requests.post" shows up. That is at March 13, at 10:11am. At line 330 in the kern_cut.log, where I cut out the relevant sections from boot to when files were captured, a crash occurs. That is is March 13, at 10:12am. The only way to rule out the BrosTrend USB wifi adapter is to remove it, which is my next step. Also, the reason you don't see in the nohup.out log file any problems with DNS lookup of blynk-cloud or skyweather is that I did not let the scripts start at boot. I waited until the wifi was stably operating. I now know that there is a delay between the boot sequence and the BrosTrend USB wifi adapter getting completely configured. My next step is to remove the BrosTrend USB wifi and retry with the exact same SkyWeather2 scripts. The current BrosTrend wifi adapter has these numbers when operating: Link Quality=95/100 Signal level=53/100 Noise level=0/100 The onboard wifi while contained in the metal fanless heatsink case provides: Link Quality=49/70 Signal level=-61 dBm (which will be fine if I put the WeatherRack2 close enough to the wifi access point) Dow Hurst Attachments:logs.tgz (414.32 KB)
|
|
|
Post by SDL on Mar 15, 2021 8:24:00 GMT -8
Dow,
So, I think you are heading the right direction. There is something wrong with the network and I definitely suspect the difference in your system, the USB WiFi dongle.
I suspect that the Raspberry Pi foundation has done little or no testing with external wifi dongle's. Another test to try is plug it into a wired network. Further isolating the problem.
BP
|
|
|
Post by Jason on Mar 15, 2021 15:17:21 GMT -8
Apologies if I missed it in an earlier post. What is the model of the adapter you are trying to install?
Thanks,
Jason
|
|
maestro
New Member
Posts: 13
Raspberry Pi: Yes
|
Post by maestro on Mar 16, 2021 20:32:48 GMT -8
I've finally captured the error. It's telling me there's a 504 gateway timeout just after it skips the first time. Since I've been having this error I've had the pi wired via ethernet cable and I don't seem to have any connectivity problem aside from this (though I suppose I can't rule it out completely.) I'm using a pi4 and currently running V024. Here's the output:
processing FT020T Data This is the raw data: {"time" : "2021-03-17 00:00:54", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 0, "gustwindspeed" : 0, "winddirection" : 77, "cumulativerain" : 1221, "temperature" : 826, "humidity" : 87, "light" : 0, "uv" : 0, "mic" : "CRC"}
duplicate found processing FT020T Data This is the raw data: {"time" : "2021-03-17 00:01:10", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 0, "gustwindspeed" : 0, "winddirection" : 68, "cumulativerain" : 1221, "temperature" : 826, "humidity" : 87, "light" : 0, "uv" : 0, "mic" : "CRC"}
Tick! The time is: 2021-03-17 00:01:13.756478 ------------- Current State ------------- ------------- latest MainSensor Reading= 2021-03-17 00:01:11 MainDeviceNumber= 0 OutdoorTemperature = 5.89 OutdoorHumidity = 87 latest Indoor Sensor Reading= 2021-03-17 00:00:30 IndoorDeviceNumber= 1 IndoorTemperature = 23.17 IndoorHumidity = 29 Rain60Minutes = 0.0 SunlightVisible = 0 SunlightUVIndex = 0.0 WindSpeed = 0.0 WindGust = 0.0 WindDirection = 68 TotalRain = 122.1 BarometricTemperature = 23.96 BarometricPressure = 102.0627 Altitude = 14.0 BarometricPressureSeaLevel = 102.23722 BarometricTemperature = 23.96 barometricTrend = True pastBarometricReading = 0 AQI = 0.0 Hour24_AQI = 0.0 WS_AQI = 0.0 WS_Hour24_AQI = 0.0 Main Battery Status = OK CPU Temperature = 0.0 ------------- runRainbow = False flashStrip = False runOLED = True ------------- Last_Event = My Last Event ------------- batteryVoltage 0 batteryCurrent 0 solarVoltage 0 solarCurrent 0 loadVoltage 0 loadCurrent 0 batteryPower 0 solarPower 0 loadPower 0 batteryCharge 0 SolarMAX Inside Temperature 0.0 SolarMAX Inside Humidity 0.0 SolarMAX Last Received Never ------------- ------------- ------------- fanState = False ------------- ------Patting The Dog------- Execution of job "takeSkyPicture (trigger: interval[0:01:00], next run at: 2021-03-17 00:01:13 EDT)" skipped: maximum number of running instances reached (1) Processing F016TH data This is the raw data: {"time" : "2021-03-17 00:01:23", "model" : "SwitchDoc Labs F016TH Thermo-Hygrometer", "device" : 98, "modelnumber" : 5, "channel" : 1, "battery" : "OK", "temperature_F" : 73.700, "humidity" : 29, "mic" : "CRC"}
The pastebin URL is (r.text):<html> <head><title>504 Gateway Time-out</title></head> <body bgcolor="white"> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx</center> </body> </html>
processing FT020T Data This is the raw data: {"time" : "2021-03-17 00:01:26", "model" : "SwitchDoc Labs FT020T AIO", "device" : 12, "id" : 0, "batterylow" : 0, "avewindspeed" : 0, "gustwindspeed" : 0, "winddirection" : 76, "cumulativerain" : 1221,
|
|
|
Post by SDL on Mar 17, 2021 8:56:48 GMT -8
Since you have been having no problems with the wired interface, it's your WIFi dongle.
Or POSSIBLY your Access Point.
BP
|
|
dow4hurst
Full Member
SkyWeather2 Newbie
Posts: 117
|
Post by dow4hurst on Apr 2, 2021 14:10:50 GMT -8
Jason: My wifi adapter is a BrosTrend AC650 Model AC5L for Linux. I've ditched using that for now. BP: I received the new SwitchDocLabs SD card. The changes I implemented were just updating with the Readme procedure to v24 of SkyWeather2, recompiling/install the rtl433 driver as recommended, and getting the OS to configure the onboard wifi. No OS updates or additional software installations or configurations were performed. With the new pi 4 I found I still would have a crash occur whether using onboard wifi or ethernet. The crash is the same as before: from /var/log/messages Apr 2 04:28:38 SwitchDocLabs kernel: [54164.326696] WARNING: CPU: 0 PID: 8242 at drivers/firmware/raspberrypi.c:64 rpi_firmware_transaction+0xec/0x128 Apr 2 04:28:38 SwitchDocLabs kernel: [54164.326710] Firmware transaction timeoutWith the original pi 4 all this just happens more quickly. I have a new unused AQI sensor. With all the troubleshooting, I'd like to just get a new enclosure so I can mount the pi as it was intended and without a metal case on it. The fresh SD card eliminated all the updates to the OS and firmware. The new pi 4 eliminated what I was thinking was a pi hardware problem. I'm down to the metal pi 4 case warming other components too much, the HAT, the AQI, the SDR, or the power supply as sources of error. When it gets this bad, I'd just prefer to start over. All I really want is to get it working and stable so I can learn the software side of capturing and presenting the data in cool ways like some of the others on the forum are doing. I'm part of the funding campaign for the Lightning sensor and added an extra enclosure in the purchase. Would that suffice to replace the current SkyWeather2 enclosure? If so, then I could get a new grove skyweather2 HAT and that might be enough. Attached are some pics of the original way I was going to have it put together and weatherproofed when I thought the BrosTrend wifi adapter would be useful. Dow Hurst
|
|
|
Post by SDL on Apr 4, 2021 9:54:07 GMT -8
Per our chat on the chat line, I have modified testSkyCamera.py to loop around and run the WeatherStem transfer which should give us a better idea on what the heck is going on regarding failures on your system.
Put this in a file called testLongSkyCamera.py in your SkyWeather2 Directory and the execute with "sudo python3 testLongSkyCamera.py"
It will send a picture to WeatherSTEM every 30 seconds. Let's see what and how it fails.
# imports # Check for user imports from __future__ import print_function import config
import state import SkyCamera import time
#Establish WeatherSTEMHash if (config.USEWEATHERSTEM == True): state.WeatherSTEMHash = SkyCamera.SkyWeatherKeyGeneration(config.STATIONKEY)
print("config.STATIONKEY=", config.STATIONKEY) # test SkyWeather Camera and WeatherSTEM
while (True): print ("taking SkyPicture") SkyCamera.takeSkyPicture() print ("sending SkyCamera") SkyCamera.sendSkyWeather()
time.sleep(30)
BP
|
|