Widespread JACE Failure

[theGAXman]

My company is having widespread JACE failure across multiple distribution channels (we primarily have Distech, but have seen at least one case of JCI and Honeywell JACEs going down as well), starting around 2pm today. They are all on Niagara AX 3.7.106 or 3.8.38.

In most cases, the system boots up, gets the station started, then has this pair of errors:

java.lang.OutOfMemoryError: Java heap space
ENGINE WATCHDOG TIMEOUT STACK DUMP @ Tue Jun 14 16:52:29 CDT 2016

Station usually remains inoperable during this time, but in a few cases the station continues to work while the platform attempts to stop it.

Anyone else experiencing the same? Any fixes? We're about to start pushing normal backups, then clean.dist if that fails.

[ddcctrl]

We have been experiencing the same thing on a couple of our sites. The only common thread between the JACE's are Weather Service. We have disabled the weather service and are monitoring to see if situation improves.

[timj772]

Situation has improved with disabling the weather service. Tridium has been contacted and they are investigating the issue.

[theGAXman]

Disabling the weather service solved our issue on every JACE we had platform access to. The only trouble we ran into are JACEs that are too frozen to restart, in which case they also needed a power cycle. Otherwise, worked great.
Thanks for the tip, guys!!

[lwarren]

Wasn't that fun. Spent all day yesterday and part of this morning fixing I don't know how many jaces.You would think Tridium would have made provisions for this since it happened before in 3.4 I believe.

[MDaly]

Wasn't that fun.

You sir need a new hobby.

I only had a few thank god, and only one that I had to go to site for. Hard to call a customer and say Hey I need a PO to come out and fix your system that stopped working just because.

[brockers]

We have several dozen remote systems we needed to disabled and finally just build a Niagara service module that we could put on a supervisor to remove the Weather Service from any attached Jace's. If anyone else is interested your welcome to use it. It is on the products of the Automation Integrated website.

Needless to say yesterday was a long day. Until we got that program installed we were hurting. It got us down to a half dozen that had gone into a hard lock and were forced to have someone unplug the box.

[lwarren]

You sir need a new hobby.

I only had a few thank god, and only one that I had to go to site for. Hard to call a customer and say Hey I need a PO to come out and fix your system that stopped working just because.

Yeah we just bit the bullet and didn't charge any of our customers. We did have one jace that died in the reboot process, all the lights went out and stayed out. Haven't tried serial shell yet to see if it is recoverable.

[MDaly]

Yeah we just bit the bullet and didn't charge any of our customers. We did have one jace that died in the reboot process, all the lights went out and stayed out. Haven't tried serial shell yet to see if it is recoverable.

That's how I felt about it. However I was told to charge them. :/

[klrogers]

Yeah we just bit the bullet and didn't charge any of our customers.

That's how we handled it as well, not worth the loss of customer goodwill to charge them. Just hope there is not a repeat to soon.

[brockers]

Yeah we just bit the bullet and didn't charge any of our customers.

Thankfully we were able to write the program using money from a existing service contract. Gave the fix to the rest of the customers.

[davem]

Wow, brockers, you guys work fast - write a cool piece of code, and make a nice-looking webpage with a link...good stuff.

[brockers]

Thanks davem, hope it helps somebody.

[ascj]

It's not the weather service causes the issues but the advisories. You don't need to disable the service. Here is a video from Jeff from Honeywell.

https://drive.google.com/open?id=0By...WlkWmczZFREN0E

[klrogers]

A question was asked on the Niagara community

Are there any other parts of the weather service that could potentially cause station crash if the NWS decides to change their site? I only disabled the advisories for now, but I'm wondering if it would be prudent to disable the entire weather service...


reply from kevin (Tridium)

Yes, if the forecast and/or the conditions server were to redirect to HTTPS the Niagara weather service will react the same. If your station doesn't require the weather service for control I would recommend disabling the entire service until Tridium supplies a patch.

[mechmike2]

Had 1 site where advisories were already disables and it crashes.

[klrogers]

Does it have a ndio board attached?

Sent from my mobile device

[orion242]

Are there any other parts of the weather service that could potentially cause station crash

More fluff, more bugs. Exactly what I don't need in my I/O controller.