Out of Memory - LINUX

[BigB]

Hi all,
I was hoping one of the Linux gurus could clarify something for me..

Sometime starting over the weekend, some OTM actions.. such as change password and enter shipment events started throwing an out of memory error.
(specific error below)

OTM was ok otherwise and the memory issue seemed to occur only doing the above.

OAS and Tomcat console.log did not show the error (or any errors) and the memory usage on each, before GC, was low.

Before GC was around 500 MB on both.

The out of memory only showed in glog.app.log and on the UI screen.

Web and APP (OAS) are on their own servers.
APP/OAS is a 4x2 CPU with 8 GB ram.

During the issue yesterday, TOP showed: (APP)

Tasks: 180 total, 1 running, 179 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5% us, 0.2% sy, 0.0% ni, 99.2% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 8309152k total, 8281672k used, 27480k free, 227524k buffers
Swap: 9727348k total, 0k used, 9727348k free, 6392856k cached

CURRENT top on APP - no issues today:

Tasks: 174 total, 1 running, 173 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5% us, 0.4% sy, 0.0% ni, 98.9% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 8309152k total, 2556968k used, 5752184k free, 165652k buffers
Swap: 9727348k total, 0k used, 9727348k free, 1266168k cached

free showed a huge cached memory amount, which accounted for the very low free memory.

There was zero swapping.
OAS was using 1.1 GB and nothing else appeared to be using much memory.

My understanding is that cached memory is normal and LINUX will re-use any of the needed cached pages.
I just have never seen an extra 4 GB of cached pages before.

We normally see TOP showing as above in the 'CURRENT' snapshot.

This is a hosted server (IBM).
There are other things on the box, such as Tivoli running.

We asked IBM engineers to look at the box to see if they noticed something we didn't.
We received a very vague response from IBM saying the cached memory was used up and the physical server had to be restarted.
They were given the green light before anyone was consulted.. and the physical server was restarted.

The out of memory and high cached memory issue did not come back, and that was the only time we have seen both occur.

Question: Is this normal for LINUX to cache so much physical memory ?

Does anyone have any thoughts on the specific out of memory error ?

Any comments in general on this issue ?

Thanks a lot !
Brian

2008-04-21 10:14:55.730 0 Error Exception java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:111)
at java.util.jar.JarFile.(JarFile.java:127)
at java.util.jar.JarFile.(JarFile.java:92)
at oracle.classloader.SharedJar.doOpen(SharedJar.java :20
at oracle.classloader.SharedCodeSource.open(SharedCod eSource.java:1124)
at oracle.classloader.SharedCodeSource.ensureOpen(Sha redCodeSource.java:94
at oracle.classloader.SharedCodeSource.getResourceByt es(SharedCodeSource.java:967)
at oracle.classloader.CodeSourceSearchPolicy.findReso urceBytes(CodeSourceSearchPolicy.java:57)
at oracle.classloader.PolicyClassLoader.findLocalClas s(PolicyClassLoader.java:1450)
at oracle.classloader.SearchPolicy$FindLocal.getClass (SearchPolicy.java:167)
at oracle.classloader.SearchSequence.getClass(SearchS equence.java:119)
at oracle.classloader.PolicyClassLoader.internalLoadC lass(PolicyClassLoader.java:1660)
at oracle.classloader.PolicyClassLoader.loadClass(Pol icyClassLoader.java:1621)
at oracle.classloader.PolicyClassLoader.loadClass(Pol icyClassLoader.java:1606)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:302)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:141)
at oracle.jdbc.ttc7.TTC7Protocol.safelyGetClassForNam e(TTC7Protocol.java:3635)
at oracle.jdbc.driver.OracleConnection.safelyGetClass ForName(OracleConnection.java:5125)
at oracle.jdbc.driver.OracleConnection.addClassMapEnt ry(OracleConnection.java:2881)
at oracle.jdbc.driver.OracleConnection.addDefaultClas sMapEntriesTo(OracleConnection.java:2872)
at oracle.jdbc.driver.OracleConnection.initializeClas sMap(OracleConnection.java:255
at oracle.jdbc.driver.OracleConnection.ensureClassMap Exists(OracleConnection.java:2552)
at oracle.jdbc.driver.OracleConnection.getTypeMap(Ora cleConnection.java:285
at glog.util.jdbc.noserver.T2SharedConnection.setType MapUser(T2SharedConnection.java:202)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:170)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:176)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:156)
at glog.util.jdbc.noserver.T2SharedConnection.setUser (T2SharedConnection.java:227)
at glog.util.datasource.ConnectionEstablisher.setVpdU ser(ConnectionEstablisher.java:2
at glog.util.datasource.GC3ConnectionEstablisher.conn ect(GC3ConnectionEstablisher.java:45)
at glog.util.datasource.GC3ConnectionPoolInstance.con nect(GC3ConnectionPoolInstance.java:212)
at glog.util.datasource.GC3ConnectionPoolInstance.res erveConnections(GC3ConnectionPoolInstance.java:205 )
at glog.util.datasource.GC3ConnectionPoolInstance.can ReserveConnection(GC3ConnectionPoolInstance.java:1 84)
at glog.util.datasource.GC3ConnectionPoolInstance.res erve(GC3ConnectionPoolInstance.java:46)
at glog.util.datasource.GC3ConnectionPool.reserve(GC3 ConnectionPool.java:36)
at glog.util.datasource.DataSource.reserve(DataSource .java:143)
at glog.util.datasource.DataSource.reserve(DataSource .java:137)
at glog.util.jdbc.noserver.T2SharedConnection$Referen cedConnection.addRef(T2SharedConnection.java:296)
at glog.util.jdbc.noserver.T2SharedConnection.open(T2 SharedConnection.java:53)
at glog.server.eventgroup.EventGroup.loadEventGroups( EventGroup.java:59)
at glog.server.eventgroup.EventGroup.get(EventGroup.j ava:17)
at glog.server.eventgroup.EventGroupClassification.lo ad(EventGroupClassification.java:82)
at glog.server.eventgroup.EventGroupClassification.ge tClassification(EventGroupClassification.java:49)
at glog.server.eventgroup.EventGroupClassification.cl assify(EventGroupClassification.java:19)
at glog.business.action.ShipmentTrackingExecutor.clas sifySsEventGroups(ShipmentTrackingExecutor.java:15 6)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addSSHistoryAndPublish(Sh ipmentStatusSessionBean.java:1054)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addShipmentStatusInternal (ShipmentStatusSessionBean.java:1013)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addShipmentStatus(Shipmen tStatusSessionBean.java:725)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionServerSideEJBWrapper.addShipme ntStatus(ShipmentStatusSessionServerSideEJBWrapper .java:220)

[chrisplough]

Re: Out of Memory - LINUX

Brian,

I'm not sure that I agree with the IBM tech. You're right that Linux (and most modern OS's) will utilize any free memory in order to cache filesystem activity. In fact, I've seen this as high as 12GB on a 16GB server. The point is, as soon as this memory is needed for another process, the OS will immediate release it. So if you're looking at memory utilization graphs, it's important to exclude the system cache, in order to get a true view.

Now - based on the error, it seems that someone tried to load up a large zip file into OTM (see below - highlighted). The out of memory issue is likely code related, rather than JVM heap memory, which is why the error showed up in the glog.app.log and not the console.log file. So - some component of OTM was trying to unzip this file and was unable to deal with the expanded size.

2008-04-21 10:14:55.730 0 Error Exception java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)

I believe there is a limit in how large uploaded files can be in OTM, but don't remember the exact limit (5, 10 or 20MB - something like that). There is another servlet that allows larger uploads, but it doesn't kick of Agents and other workflow actions.

I would find out who was doing the upload and then try it again with a smaller file.

--Chris

[BigB]

Re: Out of Memory - LINUX

Hey Chris,
Thanks for the reply and confirming my understanding of the file system cache.

I also don't think the IBM tech knew what he was talking about.. and I haven't been shy about expressing my opinion for the past 2 days..

My first thought was an upload also, but we were able to reproduce the error just trying to add a shipment event as well as a new user trying to change their password on their first login attempt.

The issue has only occurred 1 time, so perhaps there was a fluke or such..

Thanks for the input !

-Brian