Has anyone out there experienced multiple, simultaneous BAS panel failures and coldstarts with a Siemens Apogeee BAS? It looks like the problem may be caused by problems with the ethernet network connecting the servers and panels.
No, but I'd like to learn more about it
Are these Bacnet panels by chance?
I haven't seen that happen with an ethernet network but I have seen it on rs485 BLNs more than once. Usually what turns out to cause it has been some sort of a network device failure. The first time I saw it happen was due to a bad Trunk Interface. The second time was due to a shorted network cable (although that time I am not sure the cabinet coldstarts were due to the cable short). The third time was due to someone connecting a new FLN network to the BLN. One of the common symptoms with all three incidences was that after disconnecting all of the cabinets from the network some of the cabinet node tables were corrupted.
I hope this helps.....
Multiple panel failures and returns from failure on ethernet, yes.
Haven't seen multiple simultaneous coldstarts yet though.
Welcome to Hell. Here's your copy of Metasys.
After the power was lost/cycled,,, bad things happened
I've not heard of this on ethernet panels either, like Shiff said only on RS-485 networks and then usually when a device on the network fails.
Originally Posted by crnelson
What led you to the conclusions regarding the network causing the problems?
What's the battery status? Don't just take the field panel's status at it's word, go out and check it with a VOM. It is a AA battery or a Li battery?
You sure the power didn't flash several times before dropping out completely? I've had that mess up panels on both types of network.
Enough guessing on my part.
Could you provide more detials on yours, please?
Had another thought, did you add those panels to the Insight and then they did the coldstart or was the connection already established?
I agree, more details are needed. Obviously, these panels exist on a ethernet, so we know that.
1) Is the problem replicable?
2) Are you SURE the batteries are good.
3) What type of panels
4) Firmware revision would be nice
It takes a large event to coldstart multiple panels on a RS-485 and I can't remember ever having this issue with an Ethernet set of panels. (other than when adding them to the BLN, like Cheesey-dude said)
Originally Posted by chesehd
The problem is random. Twice it has coincided with a "network storm". It seems like the panels automatically disconnect from the network when the network is having a problem. The panels also automatically cold-start which deletes all trend data.
I have not checked the batteries other than by running a panel configuration report. Just to be sure, I'll check & replace as needed. However, a battery failure in one panel should not cause the other 36 panels to coldstart at the same time. As proof of this, a prior battery failure did not cause any problems. All of the panels are on E-power and have a local UPS with clean power. The reserve time is approximately 20 minutes.
The panels are mostly MBC-42 with a few MECs. The firmware is a mix of v2.7 and v2.6. Most of the v2.6 has been upgraded to build 946, per Siemens recommendation. The upgrade has not corrected the problems.
I have talked to Siemens technicians who have experience with installations of large numbers of MBCs on ethernet networks. My site is the only one that has had this problem.
None of these panels use BACnet. They use the standard Siemens' protocols.
Thanks for the feedback.
I am dealing with a sight with similar network failures, minus all of the coldstarts. I have all MECs v 2.6 build 941 and 2.7 build 1005. I was told to flash the 2.6 to the 946. One panel locked up and that is why we have the v2.7 with the 1005 build. The 1005 is supposed to have all of the fixes that the v2.6 946 build, and then some. The problem continued with 2.7 and field support is really scratching their heads, me too. I believe we have gone over all of the basics (cable condition and connectons, power quality, etc.) I will be back to run NDPT, again, and Sniffer this week in hopes of narrowing our search on the network. I will check back if I come up with any possible solutions that may help you. Good luck.
Last week I had an Insight workstation that would reboot everytime that Insight was opened up. It turned out that 2 Microsoft updates caused this, after too many hours of the customer looking over my shoulder asking if I figured it out and wanting to know when his system will work again.
This is interesting, I'll be watching to see if you come with anything definitive.
Originally Posted by ttstrutt
Those are good details.
Since your panels and firmware are up to date, I don't have an easy answer. The next step as I see it would be to try to re-create the issue (the "network storm") while watching the network with a sniffer. I'd be watching for whatever packets were coming in faster than others and if any of the packets were corrupted.
Sorry, I don't have any really large ethernet BLN's, so I'm not much help. It does seem that the panels are getting either too many of a particular packet or getting corrupted packets. Nothing else I can think of that would cause what you are seeing.
Originally Posted by crnelson