Niagara Service Stopping

[numbawunfela]

I have an N4 Server on Windows Server 2016 and Niagara 4.7.110.32

I have had 2 instances where the web UI was unavailable - people screamed that their hair was on fire (Site with 25 JACEs) - and I look at the server and cannot get in remotely with my own workbench - so I RDP into the VM and the Niagara service is stopped. I reboot the server and the service starts right up like it should. I checked Spy -> Consol for some backups and I got nuthin since about 9/5/19. All I have is a small backup of the console from today that came from between the first reboot of the station and the second. That is odd....

What I do have from 9/5 is a bunch of this:
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

SEVERE [15:00:26 06-Sep-19 PDT][web.jetty]
java.lang.OutOfMemoryError: GC overhead limit exceeded

That WAS an issue - but I bothered the contractor building the server and told them to request more memory from the IT department who provided the VM and it has not been since. It has 4GB now and runs about 30% CPU all day - I would like more memory but that is not my call just yet - that is a long story. In short, the College put the server upgrade out to bid - a contractor that the College HATES violently got the contract, and I get to watch them screw that up and then spend the next few years billing for my time by the hour to make it work... they sure saved money on that whole thing...

Anyhow the system has been running since September with not many issues until these 2 blips in the last 2 weeks, but does not somehow leave a trail I can Audit.

All I have in the Windows Event Viewer is notices that the service is starting - no error indicating why it stopped.

My suspicious mind looks for sabotage - the other contractor still has access to the system - so I turned on all the Windows Auditing features so I see what is being done.

If there are any suggestions I am open to hearing what you all might think.

[orion242]

More RAM to the VM doesn't mean the javavm running Niagara will use more.

You need to edit a config file in your Niagara install. That many Jaces, the stock max may not be enough. Don't have my notes handy but off memory its a file called nre.properties.

Search niag central for nre.properties and I would assume you will find the details on what to adjust.

[numbawunfela]

ahhh yes - I think i remember that

Thanks for pointing me in the right direction.

http://kb.ct-supply.com/kb/a168/modi...sor-in-n4.aspx

Something like this I believe - good to have a link that odes not require a login for ease of access for all here.

[numbawunfela]

*

[pmoo]

And keep in mind every time you do an upgrade you need to do this again. Believe it or not I've walked away multiple times and forgotten to change this...

[numbawunfela]

So the default is:

################################################## ######
#
# File: nre.properties
#
# This file contains properties which used
# the NRE launch executable
#
# Revision History
# 17 Feb 04 Brian Original
#
################################################## ######


# The list of options separated by spaces to pass thru to the VM
station.java.options=-Dfile.encoding=UTF-8 -Xss512K -Xmx256M
wb.java.options=-Dfile.encoding=UTF-8 -Xss512K -Xmx1024M

The station I am working with says:

# The list of options separated by spaces to pass thru to the VM
station.java.options=-Dfile.encoding=UTF-8 -Xss512K -Xmx1024M
wb.java.options=-Dfile.encoding=UTF-8 -Xss512K -Xmx2048M


So this has been done already apparently.

In Niagara Central I see conversations about nre.properties that are basically identical to the links in this thread. The only interesting thing was that a Tridium guy said:

Please take a look at the Tech Tip titled, "Modifying Max Heap for JVM on Niagara Supervisor in N4" here:

https://www.niagara-community.com/ar...pervisor-in-N4

It's best not to go far beyond what you need on heap. Garbage collection is a CPU intensive process that runs at a high priority, starving lower priority threads. If you allow room for a lot of "garbage" to pile up, then the periodic GC process will take a long time. If you don't allow enough room, then it runs too frequently, and less efficiently. There's a spy page, util/gc that will tell you how much heap your application is using when the gc process finishes. Typically, that number should be 50% to 75% of the max.

So there are limits to what is useful in this regard - more of a 'just right' setting than a 'as high as possible' one.
I may post on there, as my util/gc says my used heap is only about 40% of the total after Garbage Collection. I do not think that would stop my Niagara service, but it is best to ask the Niagara Central people to be sure.

[orion242]

If its already pushed up, you probably have an Easter egg hunt on your hands.