|
|
Post by caughron on Sept 17, 2017 11:16:42 GMT -8
Hello, hoping I can get some assistance with my AM2315. I have a Grove WeatherPi build without the solar option. I have ensured that the power source is the 2+ amp supply as well. The build has has been working well for the last two months or so - other than known bugs and issues. Within the last two weeks however, I've noticed a complete failure of AM2315. It simply stops reporting and only returns crc of -1. This occurs at some point while the system is running and working as expected. It is not after a reboot or power failure, just mid-stream of normal operation. I have found that that the sensor will not revocer with a sudo reboot of the system. I have to physically remove power to the system and reapply. The sensor then immediately comes back online and functions. This lasts about three days and then the same situation occurs. This has been consistent now for four of these cycles. This seems similar to a post in another thread ( forum.switchdoc.com/post/1380/thread ). Is anyone else experiencing this problem? I think the sensor may be bad. I originally had problems with this sensor when setting up my unit. It frequently would not get recognized on boot up. I had to write a loop in the code to attempt detection up to a hundred times. Although the workaround worked well, it will still occasionally loop 100 times without recognizing the sensor. I did report these problems to SDL, but I never received the follow up I was told I would get. Any help would be great! Additionally I'm still having huge erroneous wind gust reporting problems even after the latest updates. I'll post those on a new thread as it doesn't fit here, but I'm hoping I can get these bugs worked out for reliability. Thanks in advance for any help. |
|
|
|
Post by caughron on Sept 18, 2017 6:18:35 GMT -8
Ok, so for a little more info on this. Looking at my Wunderground history, it looks like I'm only getting anywhere from 3-6 hours of reporting before this sensor completely fails. I have a catch setup to prevent reporting to Wunderground if the crc is -1 to prevent erroneous data being reported. By looking at the history of my wunder site the rate and consistency of failure can be seen. www.wunderground.com/personal-weather-station/dashboard?ID=KCABAKER102#history/s20170917/e20170917/mdailyThe last couple failures have been about 3 hours after reset. The data shows it goes several days without resetting. I'm not always able to go out and do a hard reset immediately, so that is why the span of downtime before coming back online. Hope that helps someone help me figure this thing out. |
|
|
|
Post by SDL on Sept 18, 2017 11:29:24 GMT -8
Hi Caughron, We have some suggestions. First of all, I hooked up an AM2315 to our test setup and ran it for days with no problems. That being said, John said he had some problems a couple of weeks ago that sounded similar to your problem. We know a couple of things at this point: 1) The manufacturer says there is no known problem like this. 2) We are running the AM2315 off of ESP8266 in the OurWeather kits for hundreds of thousands of reads with no problem. 3) John has seen a problem like yours in the past. The problem went away so he didn't pursue it. The problem seems to be focused on the I2C bus on the Raspberry Pi and that is what we are going to focus on in the next weeks (start later next week) to figure out. The first thing to do is for us to build a reliable failure mode so we can test fixes. The Am2315 has kind of a funky startup mode (it powers the device down between reads to reduce self heating), so it doesn't always show up on an i2cdetect. At this point, we think that something is interrupting the I2C bus on the Raspberry Pi every once in while and locking the bus. We are going to need to dig into the drivers and probably rewrite some of the existing ones to mimic what we do on the ESP8266 which absolutely fixed the similar reliability problem. More on that as we dig into the problem. First of all, regarding the power locking (we saw the same thing on the ESP8266 until we fixed it), there is a solid work around. We have something called the Grove PowerSave which allows you to power off and power on specific I2C devices under software control. We used it extensively in Project Curacao2 to increase reliability and reduce power. Check this out: store.switchdoc.com/grovepowersave-control-grove-device-power-with-your-computer-perfect-for-solar-power/Using this allows you to power off and on under GPIO control. Now that isn't the same as fixing the reliability problem, but we will be doing that too. This will work, however. Hope this is helpful. When John returns next week, he is going to take this over and figure it out. He's really good at stuff like this. Not so good at drinking beer. BP |
|
|
|
Post by caughron on Sept 22, 2017 21:46:37 GMT -8
I think you're on to something here. My system is currently in the failed mode. Checking the log I see the following being reported:
Local WeatherRack Weather Sensors Sampling
Rain Total= 0.00 in
Rain Last 60 Minutes= 0.00 in
Wind Speed= 0.00 MPH
MPH wind_gust= 0.00 MPH
Wind Direction= 0.00 Degrees
Wind Direction Voltage= 2.114 V
DS3231 Real Time Clock
BMP280 Barometer
Temperature = 67.42 F
Pressure = 99.82 KPa
Altitude = 126.40 m
Sealevel Pressure = 101.24 KPa
Sunlight Vi/IR/UV Sensor Not Present
HDC1080 Temp/Hum Not Present
HTU21DF Temp/Hum
AS3935 Lightning Detector Not Present
AM2315 Temperature/Humidity Sensor Not Present
Local WeatherRack Weather Sensors Sampling
Rain Total= 0.00 in
Rain Last 60 Minutes= 0.00 in
Wind Speed= 0.00 MPH
MPH wind_gust= 0.00 MPH
Wind Direction= 0.00 Degrees
Wind Direction Voltage= 2.124 V
DS3231 Real Time Clock
BMP280 Barometer
Temperature = 67.44 F
Pressure = 99.82 KPa
Altitude = 126.40 m
Sealevel Pressure = 101.23 KPa
Sunlight Vi/IR/UV Sensor Not Present
HDC1080 Temp/Hum Not Present
HTU21DF Temp/Hum
AS3935 Lightning Detector Not Present
AM2315 Temperature/Humidity Sensor Not Present
Local WeatherRack Weather Sensors Sampling
Rain Total= 0.00 in
Rain Last 60 Minutes= 0.00 in
Wind Speed= 0.00 MPH
MPH wind_gust= 0.00 MPH
Wind Direction= 0.00 Degrees
Wind Direction Voltage= 2.114 V
DS3231 Real Time Clock
BMP280 Barometer
Temperature = 67.42 F
Pressure = 99.81 KPa
Altitude = 126.48 m
Sealevel Pressure = 101.23 KPa
Sunlight Vi/IR/UV Sensor Not Present
HDC1080 Temp/Hum Not Present
HTU21DF Temp/Hum
AS3935 Lightning Detector Not Present
AM2315 Temperature/Humidity Sensor Not Present
AS3935, HDC1080 and HTU21DF are all installed and report present when system is running correctly. With all these failures it has to be the common factor of the I2C bus.
Thanks for the update, I will try the workaround tomorrow.
|
|
|
|
Post by SDL on Sept 23, 2017 12:01:06 GMT -8
Yes, the I2C bus looks locked. Out of curiosity, is the white LED always on on the Pi2Grover board when it locks?
BTW, the HDC1080 and HTU21DF both have 0x40 as the I2C address. They can't be on the same I2C segment. If they are, I wonder if they are contributing to this problem?
BP
|
|
|
|
Post by caughron on Sept 23, 2017 18:34:59 GMT -8
I'll check on that light in the morning. As far as HDC1080 and HTU21DF, not sure if the conflicting address is a result of the problem or like that prior to the failure. I followed the installation instructions exactly from the tutorial, so it shouldn't be a result of installation error.
My question is this. Could it simply be that the I2C board is going out? This system was running without this issue for several months. It just started with no changes to installation, configuration or anything else that I am aware of. Would it be more beneficial to replace that board than to purchase another component to compensate for a failing board?
Thanks for the help. I will reset the system after I check on the light and send an updated log with it running to check those addresses.
PS - i2cdetect readout with the system in fail mode:
pi@WeatherPi:~ $ i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: 40 -- -- -- -- -- -- -- 48 -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- 57 -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
70: -- -- -- 73 -- -- -- 77
|
|
|
|
Post by caughron on Sept 24, 2017 8:55:26 GMT -8
Good morning. I checked the system this morning while in fail mode. In response to the Pi2Grover white LED - no it is not on constant during failure. It flashes on/off as normal in conjunction with the 16-bit ADC. I did notice however, the white LED does not flash at all on the I2C Bus during failure mode, but does after reset and running normally. Here is a readout of i2cdetect after reset: pi@WeatherPi:~ $ i2cdetect -y 1 0 1 2 3 4 5 6 7 8 9 a b c d e f 00: -- -- -- -- -- -- -- -- -- -- -- -- -- 10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 40: 40 -- -- -- -- -- -- -- 48 -- -- -- -- -- -- -- 50: -- -- -- -- -- -- -- 57 -- -- -- -- 5c -- -- -- 60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- -- 70: -- -- -- 73 -- -- -- 77 Here is a readout of GroveWeatherPi.log after reset: I2C Mux - TCA9545: Present BMP280: Present DS3231: Present HDC1080: Not Present HTU21DF: Present AM2315: Present ADS1015: Present ADS1115: Not Present AS3935: Present OLED: Not Present SunAirPlus: Not Present Sunlight Sensor: Not Present WXLink: Not Present ****Of interest is the reporting for ADS1015 & ADS1115 - I have 1115 installed. Not sure why system is reporting 1015 present and not 1115. Also, HDC1080 is reported not present. I believe that to be a chip on one of the boards? Readout of sampling after reset: Local WeatherRack Weather Sensors Sampling Rain Total= 0.00 in Rain Last 60 Minutes= 0.00 in Wind Speed= 2.99 MPH MPH wind_gust= 4.48 MPH Wind Direction= 247.50 Degrees Wind Direction Voltage= 2.816 V DS3231 Real Time Clock BMP280 Barometer Temperature = 66.45 F Pressure = 100.20 KPa Altitude = 94.59 m Sealevel Pressure = 101.62 KPa Sunlight Vi/IR/UV Sensor Not Present HDC1080 Temp/Hum Not Present HTU21DF Temp/Hum AS3935 Lightning Detector ----No Lightning detected--- AM2315 Temperature/Humidity Sensor outsideTemperature: 64.2 F outsideHumidity: 40.6 % crc: 1 Strange indeed! Attachments:
|
|
|
|
Post by SDL on Sept 25, 2017 18:15:39 GMT -8
Very strange indeed.
Remove the HDC1080 (I don't see it in your photo - nice job by the way) and see what changes.
It is possible that the ADS1115 can be misdirected as an ADS1015 and the software adjusts to it. Curse of the detection mechanism.
It looks like you don't have the I2C power jumpers installed on the I2C Mux. If you don't, then all sorts of odd things can happen.
BP
|
|
|
|
Post by caughron on Sept 25, 2017 22:17:42 GMT -8
Yes, the jumpers are installed. Just hard to pick out in the photo. Not sure what is meant by remove HDC1080? From code or hardware? I don't think I have it in hardware.
As for 1015 and 1115, I've only had that mis-detection once that I've seen. I rebooted again after the logs I sent and it detected correctly. So, not sure what that was about.
Additionally, I did a fresh clone of the SDL software to eliminate any errors I may have introduced in code changes. Still getting the lockup with latest code running.
Still like to hear your thoughts on replacing the I2C Mux board as a possible cause of the failure. As I said, this has just recently begun occurring. Prior, it ran without the lockup error for several months.
Thanks for the ongoing help.
|
|
|
|
Post by SDL on Oct 1, 2017 13:19:35 GMT -8
OK. Let's try your I2C Mux theory.
Pull the Mux out of the I2C Interface and put the AM2315 on the main I2C bus.
Run the software (it should still work, just not find many devices) and see how long it takes to fail.
BP
|
|
|
|
Post by caughron on Oct 4, 2017 21:54:31 GMT -8
I should've thought of that! I will give it a try hopefully tomorrow. Thanks for the suggestion.
|
|
|
|
Post by caughron on Oct 12, 2017 17:47:41 GMT -8
Ok, sorry for the delay but have been out of town.
I tried two things. First, I removed the AM2315 from the I2C Mux board and plugged it into an I2C slot on the Weather Board. I removed the lightning detector from the I2C Mux and ran the system. Everything booted into working condition and ran for about a day before I got the lockup. As usual that did not respond to anything but recycling the power.
Next I removed the I2C completely and left the AM2315 plugged into the Weather Board. Additionally, I plugged the I2C Hub into an I2C slot on the Weather Board and ran that configuration. Once again, everything booted fine and ran for about 12 hours before failing and locking up again.
This has eliminated the I2C Mux and isolated the recurring problem to the AM2315 it seems. The last thing I can try is eliminating the I2C Hub from the system as well, but at this point I'm pretty confident the problem lies with the AM2315.
Any thoughts or suggestions are greatly appreciated.
|
|
|
|
Post by SDL on Oct 14, 2017 17:49:15 GMT -8
Your result eliminating the I2C mux is what I was expecting. The I2C hub isn't what is doing this IMHO.
I think we have a similar problem with the Raspberry Pi that we saw with the ESP8266 in rare occasions. We will build a test jig here in the near future and start experimenting.
As we talked about earlier, regarding the power locking (we saw the same thing on the ESP8266 until we fixed it), there is a solid work around. We have something called the Grove PowerSave which allows you to power off and power on specific I2C devices under software control. We used it extensively in Project Curacao2 to increase reliability and reduce power. Check this out:
store.switchdoc.com/grovepowersave-control-grove-device-power-with-your-computer-perfect-for-solar-power/
|
|
|
|
Post by caughron on Oct 18, 2017 17:04:36 GMT -8
So what would I control with the Powersave? I’ve removed the I2C mix from the system and still getting the error. Is it the Pi itself locking up? My take on the Powersave was it would allow a remote reset of the I2C mix, but if that isn’t the problem not sure what I would use it to control.
If it is the I2C mix, I’ve read the documentation that shows there is a reset pin on the mux that can be controlled by software to do a reset in the event of a lockup. Looks like a possible known error of getting stuck during the reading of a bus. Looks like a resistor pull-up between the reset pin and vdd. Do you know if that resistor is in place, or would I need to add that?
Thanks for the help.
|
|
|
|
Post by SDL on Oct 24, 2017 9:15:30 GMT -8
I **think** (given all your good data) that the AM2315 is locking up, but not locking the I2C bus (hence all the questions on the white LED - that is SCL on the I2C Bus).
So, you plug the Grove PowerSave between the AM2315 and the I2C bus. Then you can flip the power off and on to the AM2315, hence doing a power on on the AM2315 device.
Note: We saw something similar on the AM2315 on an ESP8266 system. Our plan is to look at what we did to fix that and apply the same thing to the Pi if possible.
Note further: Not everyone is seeing this with the AM2315. That is why it is so important for us to duplicate this.
Best regards,
BP
|
|
