Post by SDL on Feb 17, 2019 12:22:58 GMT -8
Folks,
We are rerouting all of the AM2315 discussion to this thread so everybody is in the loop. Dr. Shovic and I met yesterday on all of the data that you fine folks (and us) have been collecting regarding the fairly rare AM2315 problems. I want to emphasize that these are RARE problems, but for a long term outdoor installed unit like GWP, we need to deal with it.
Our summary:
The AM2315 is a good inexpensive outdoor temperature and humidity sensor. We have sold 1000's of these over the years and have had very few returns and problems with the sensor. The problems with the sensors are clustered on a small group of people (< 2.5% by our best guess) however, the problems, especially with the Raspberry Pi, have been significant enough that we have been working the issue over the past 6 months. We have improved the reliability of the sensor on the Pi by swapping out drivers, adding redundancy, and checking CRCs, and doing data validity checking.
Here are some facts gathered from past 6 months:
- This is a rare problem, but significant enough to deal with
- Replacing the AM2315 does not always fix the problem (this is a very significant finding!)
- SDL has run long term tests in a variety of combinations of hardware / software and have only seen one or two glitches (see below)
- SDL adding the new Python drivers has increased the reliability significantly
- 3.3V versus 5.0V does not seem to make any difference
- The version of Raspberry Pi does not seem to make any difference
- The version of the Raspberry Pi OS (Stretch, etc.) does not seem to make any difference
- We contacted the manufacturer, AOSONG, and their engineers have not heard of the problem. Since they don't sell into the Raspberry Pi market directly, John wasn't too surprised.
- Noisy I2C lines do make a difference, but either will never work at all, intermittently or is caught by the CRC check
- It is NOT the driver that is causing this problem. Or at least all the drivers we have tried (including C based ones) have still had issues rarely. Note that Adafruit and their complicated Circuit Python related, non-standard (but that doesn't mean they are bad) I2C drivers have had issues also that seem to be similar to ours. They just haven't collected all the data that we have at this point.
- Sopwith and his AM2315 page have been helpful, but we still think that his solution improves things but does not fix them.
- The AM2315 problems seem to fall in four categories:
1) Humidity or Temperature spiking or dropping for a single read (seems to be related to an on board the AM2315 bit flit - if you look at the errors in degrees C, it always seems to be related to the value of a single bit) - Note it still passes the CRC check, which means it happened on the unit and not in transmission.
2) Really bad data - this seems to be always caught by the CRC, so this problem is handled by the new drivers
3) The AM2315 stops responding, requiring a power cycle to recover. Not a reboot on the Pi, a power cycle of the AM2315 will fix this. Both times we found this in the lab (after a lot of testing) doing this cleared the problem.
4) Locking the I2C bus - we think that this really doesn't happen. It looks like #3 in a lot of cases the we can think of.
---------
The drivers
---------
Our latest drivers address #1 and #2 (although Peter has come up with a better way that we are implementing shortly) and we gather the statistics necessary for debugging (John is very, very, very fond of data and is really good at looking at data to figure out patterns).
Note that practically all the drivers out there will work well with the AM2315. Stay away from the drivers that do not use the CRC.
#3 is the real issue still facing us.
------
What we suggest
------
One of the problems we have with working on this problem is that it happens so rarely in all of our systems that we use to test it. Sometimes we have run the AM2315 for over 700,000 reads (no kidding - two months of every 5 seconds) with zero failures or rewrites.
This seems to not be an AM2315 problem by itself, but rather a combination of maybe a weak AM2315 in combination with line issues in combination with some spec being on the edge on the Raspberry Pi I2C (note: Use of our I2C mix does not seem to be an issue with this problem, which leads us to believe that the spec has something to do with the Raspberry Pi - again since it is in combination with the AM2315, it is VERY hard to catch and debug). Again, we are dealing a relatively small amount of data so take our conclusions with a grain of salt.
We have had excellent support from the folks out there that have experienced the issue. Thank you for putting up with all my questions and suggestions. That is how we can get some kind of a pattern out of all of this.
1) If you have a lot of noise (call this failed CRC checks) you probably have a noisy line and you need to either shorten the line or add stronger pull-ups (go to 3K maybe). 5V will theoretically improve performance on a noisy line, but we don't have any empirical data on this.
2) Use the latest high-reliability Python drivers from GitHub.com/switchdoclabs - At least then we know what we are getting.
3) What we are doing now (since the vast majority of AM2315 based projects seem to function well) is we are adding a Grove PowerSave to each kit.
shop.switchdoc.com/products/grovepowersave-control-grove-device-power-with-your-computer-perfect-for-solar-power
It works like this. When we determine in the driver that the AM2315 is misbehaving badly enough, we power cycle the AM2315 using the Grove Power Save, wait about 3 seconds and hit it again. From all verifiable and confirmed accounts, this fixes the problem (until it occurs again).
You hook up the connection from the computer I2C bus (or the I2C max) to the bottom the AM2315 to the top and then you connect a GPIO to the middle. The driver will be configurable as to whether you are using a Grove PowerSave or not.
-------------
Free Grove PowerSave for you folks
-------------
The next time you order from SwitchDoc Labs, mention you have had this problem and we will include a grove power save with two Grove cables free with your order.
--------
Still looking for data
--------
We know we are addressing the symptoms rather than the cause of this problem. If you have something different happen or confirm one of our scenarios please add to this thread. We are still looking at the problem for a long term fix.
--------
Note on ESP8266 and ESP32 Problems
--------
We have seen very few problems with these units, but even with a lot less data (OurWeather is more of a beginners kit so we don't get very detailed information back like we do from you folks) there may be a similar problem. We are planning on addressing it in the same way as time goes on.
Thank you for all your information and support on this nasty little problem.
Best regards
BP and John and everybody at SwitchDoc Labs
We are rerouting all of the AM2315 discussion to this thread so everybody is in the loop. Dr. Shovic and I met yesterday on all of the data that you fine folks (and us) have been collecting regarding the fairly rare AM2315 problems. I want to emphasize that these are RARE problems, but for a long term outdoor installed unit like GWP, we need to deal with it.
Our summary:
The AM2315 is a good inexpensive outdoor temperature and humidity sensor. We have sold 1000's of these over the years and have had very few returns and problems with the sensor. The problems with the sensors are clustered on a small group of people (< 2.5% by our best guess) however, the problems, especially with the Raspberry Pi, have been significant enough that we have been working the issue over the past 6 months. We have improved the reliability of the sensor on the Pi by swapping out drivers, adding redundancy, and checking CRCs, and doing data validity checking.
Here are some facts gathered from past 6 months:
- This is a rare problem, but significant enough to deal with
- Replacing the AM2315 does not always fix the problem (this is a very significant finding!)
- SDL has run long term tests in a variety of combinations of hardware / software and have only seen one or two glitches (see below)
- SDL adding the new Python drivers has increased the reliability significantly
- 3.3V versus 5.0V does not seem to make any difference
- The version of Raspberry Pi does not seem to make any difference
- The version of the Raspberry Pi OS (Stretch, etc.) does not seem to make any difference
- We contacted the manufacturer, AOSONG, and their engineers have not heard of the problem. Since they don't sell into the Raspberry Pi market directly, John wasn't too surprised.
- Noisy I2C lines do make a difference, but either will never work at all, intermittently or is caught by the CRC check
- It is NOT the driver that is causing this problem. Or at least all the drivers we have tried (including C based ones) have still had issues rarely. Note that Adafruit and their complicated Circuit Python related, non-standard (but that doesn't mean they are bad) I2C drivers have had issues also that seem to be similar to ours. They just haven't collected all the data that we have at this point.
- Sopwith and his AM2315 page have been helpful, but we still think that his solution improves things but does not fix them.
- The AM2315 problems seem to fall in four categories:
1) Humidity or Temperature spiking or dropping for a single read (seems to be related to an on board the AM2315 bit flit - if you look at the errors in degrees C, it always seems to be related to the value of a single bit) - Note it still passes the CRC check, which means it happened on the unit and not in transmission.
2) Really bad data - this seems to be always caught by the CRC, so this problem is handled by the new drivers
3) The AM2315 stops responding, requiring a power cycle to recover. Not a reboot on the Pi, a power cycle of the AM2315 will fix this. Both times we found this in the lab (after a lot of testing) doing this cleared the problem.
4) Locking the I2C bus - we think that this really doesn't happen. It looks like #3 in a lot of cases the we can think of.
---------
The drivers
---------
Our latest drivers address #1 and #2 (although Peter has come up with a better way that we are implementing shortly) and we gather the statistics necessary for debugging (John is very, very, very fond of data and is really good at looking at data to figure out patterns).
Note that practically all the drivers out there will work well with the AM2315. Stay away from the drivers that do not use the CRC.
#3 is the real issue still facing us.
------
What we suggest
------
One of the problems we have with working on this problem is that it happens so rarely in all of our systems that we use to test it. Sometimes we have run the AM2315 for over 700,000 reads (no kidding - two months of every 5 seconds) with zero failures or rewrites.
This seems to not be an AM2315 problem by itself, but rather a combination of maybe a weak AM2315 in combination with line issues in combination with some spec being on the edge on the Raspberry Pi I2C (note: Use of our I2C mix does not seem to be an issue with this problem, which leads us to believe that the spec has something to do with the Raspberry Pi - again since it is in combination with the AM2315, it is VERY hard to catch and debug). Again, we are dealing a relatively small amount of data so take our conclusions with a grain of salt.
We have had excellent support from the folks out there that have experienced the issue. Thank you for putting up with all my questions and suggestions. That is how we can get some kind of a pattern out of all of this.
1) If you have a lot of noise (call this failed CRC checks) you probably have a noisy line and you need to either shorten the line or add stronger pull-ups (go to 3K maybe). 5V will theoretically improve performance on a noisy line, but we don't have any empirical data on this.
2) Use the latest high-reliability Python drivers from GitHub.com/switchdoclabs - At least then we know what we are getting.
3) What we are doing now (since the vast majority of AM2315 based projects seem to function well) is we are adding a Grove PowerSave to each kit.
shop.switchdoc.com/products/grovepowersave-control-grove-device-power-with-your-computer-perfect-for-solar-power
It works like this. When we determine in the driver that the AM2315 is misbehaving badly enough, we power cycle the AM2315 using the Grove Power Save, wait about 3 seconds and hit it again. From all verifiable and confirmed accounts, this fixes the problem (until it occurs again).
You hook up the connection from the computer I2C bus (or the I2C max) to the bottom the AM2315 to the top and then you connect a GPIO to the middle. The driver will be configurable as to whether you are using a Grove PowerSave or not.
-------------
Free Grove PowerSave for you folks
-------------
The next time you order from SwitchDoc Labs, mention you have had this problem and we will include a grove power save with two Grove cables free with your order.
--------
Still looking for data
--------
We know we are addressing the symptoms rather than the cause of this problem. If you have something different happen or confirm one of our scenarios please add to this thread. We are still looking at the problem for a long term fix.
--------
Note on ESP8266 and ESP32 Problems
--------
We have seen very few problems with these units, but even with a lot less data (OurWeather is more of a beginners kit so we don't get very detailed information back like we do from you folks) there may be a similar problem. We are planning on addressing it in the same way as time goes on.
Thank you for all your information and support on this nasty little problem.
Best regards
BP and John and everybody at SwitchDoc Labs