dow4hurst
Full Member
SkyWeather2 Newbie
Posts: 117
|
Post by dow4hurst on Aug 5, 2021 12:33:01 GMT -8
I changed my SkyWeather2 to not do any uploading of images. It just runs now. Been running for over a month without a crash.
Behavior identified when crashing so far by everyone that provided log files 1. A kernel module is corrupted according to log files. 2. pi locks up requiring hard reboot if not already connected by ssh 3. SkyWeather2 scripts keep running, but pictures are no longer uploaded 4. The image upload keeps trying for a while and failing for new images 5. Eventually SkyWeather2 can't even try an upload of an image 6. If connected via ssh, the ssh connection will allow a magic SysRq reboot sequence.
The loss of ability to create a new ssh login connection after the failure fully happens indicates that the kernel is crashed but functioning. The linux kernel fails in a mode that keeps packets passing through the current ipfilter rules, but it can't spawn a new process. That is why network routers that are crashed will much of the time work a long time in that mode. Eventually people or the system does something that requires spawning a new process, which won't happen. Then people reboot their routers not realizing just how long it was functioning in a crashed state. Using a network sniffer to watch each SkyWeather2 image upload separately until a failure is captured would be really helpful.
Were there any other failure modes found? Dow
|
|
|
Post by SDL on Aug 6, 2021 10:27:00 GMT -8
Dow,
This all points to something deep in the software used to upload the picture. Interestingly, we don't have that problem in our test units. BUT, I did have an unexpected crash on my new Skyweather2 software (V027D) that we are waiting for replication. We were uploading.
BP
|
|
|
Post by pagerman on Aug 9, 2021 16:37:50 GMT -8
And I am now still crashing on a daily basis. No reasoning behind the time. Random.
|
|
|
Post by pagerman on Aug 10, 2021 6:16:24 GMT -8
Now it's crashing constantly.
|
|
|
Post by SDL on Aug 10, 2021 8:29:38 GMT -8
Running for two weeks here. I don't think sending a SkyCam Remote is going to fix this.
I'm correct in understanding that you aren't uploading anything to WeatherSTEM or WeatherUnderground?
I think you need to send me your hardware.
BP
|
|
|
Post by pagerman on Aug 11, 2021 3:53:28 GMT -8
I am still loading to WU data only. I'll stop that as well now and lets see how it does. I use the graph in WU to show me when this thing crashes, since there's no log to show me.
|
|
|
Post by pagerman on Aug 11, 2021 4:41:34 GMT -8
I do see that every time I have issues, SW2 has restarted for some reason. I have nothing asking for any type of restart. In the log below the restart is when it "crashed" and I had no data and I could not get to it vis VNC or SSH. Why is it restarting the software randomly? Attachments:
|
|
|
Post by pagerman on Aug 12, 2021 4:27:18 GMT -8
Running for two weeks here. I don't think sending a SkyCam Remote is going to fix this. I'm correct in understanding that you aren't uploading anything to WeatherSTEM or WeatherUnderground? I think you need to send me your hardware. BP When you say running for 2 weeks - what is running? Are you doing any uploading to anywhere? Or just looking at data on the dash_app? I've been running since yesterday at 7am only collecting data. Seems to be working. But, if I can't upload to anywhere this is pretty useless to me........
|
|
|
Post by pagerman on Aug 13, 2021 4:32:21 GMT -8
Still running. 2 days now.
|
|
|
Post by SDL on Aug 13, 2021 10:49:38 GMT -8
Pagerman,
Send me your hardware and I'll run a download test. I don't know what else to do as we can't duplicate the problem.
BP
|
|
|
Post by pagerman on Aug 20, 2021 4:34:48 GMT -8
Ok. Observations now: I am not uploading the data to anywhere. It keeps running fine. That's kind of useless to me as I got this to upload data and webcam images. I have now further eliminated my network as an issue since I have recently built and deployed a FlightRadar24 ADS-B receiver (it uses a SDR dongle as well) and it's uploading 24/7 with no issues. See image attached. Tell me where to send my hardware and also what hardware you need. I have the SkyWeather2, 2 indoor sensors, lightning detector and the SW rack. Attachments:
|
|
|
Post by SDL on Aug 20, 2021 13:45:21 GMT -8
All I need you to send to me is your Raspberry Pi, power supply, SDR and the SD Card. The rest of it can't be the problem.
Include your Camera too.
Send it to (with directions on how to make it fail - set it up on your side before you ship it so IT DOES FAIL) and we will figure it out. I want to be able to plug it in, change the WiFi and run it until it fails.
BP
SwitchDoc Labs RMA#082021-01 ATTN: BP 909 S Shelley Lake Lane Spokane Valley, WA 99037
|
|
|
Post by pagerman on Aug 21, 2021 7:10:37 GMT -8
Will do. Thank you for looking at it. I'll set it up this week to do the uploads so it crashes a few times. Do you want me to turn on debugging or anything else that may help you?
|
|
|
Post by SDL on Aug 21, 2021 17:15:25 GMT -8
Turn on all the debugging. The key to all of this is to get a system that fails.
BP
|
|
|
Post by pagerman on Aug 23, 2021 4:33:05 GMT -8
Turn on all the debugging. The key to all of this is to get a system that fails. BP Ran for 7 days doing no uploads flawlessly. Turned everything on and it crashed within 5 minutes. I'll pack it up later this week and get it off to you. Thanks again.
|
|