Hi all,
I was hoping one of the Linux gurus could clarify something for me..
Sometime starting over the weekend, some OTM actions.. such as change password and enter shipment events started throwing an out of memory error.
(specific error below)
OTM was ok otherwise and the memory issue seemed to occur only doing the above.
OAS and Tomcat console.log did not show the error (or any errors) and the memory usage on each, before GC, was low.
Before GC was around 500 MB on both.
The out of memory only showed in glog.app.log and on the UI screen.
Web and APP (OAS) are on their own servers.
APP/OAS is a 4x2 CPU with 8 GB ram.
During the issue yesterday, TOP showed: (APP)
Tasks: 180 total, 1 running, 179 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5% us, 0.2% sy, 0.0% ni, 99.2% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 8309152k total, 8281672k used, 27480k free, 227524k buffers
Swap: 9727348k total, 0k used, 9727348k free, 6392856k cached
CURRENT top on APP - no issues today:
Tasks: 174 total, 1 running, 173 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5% us, 0.4% sy, 0.0% ni, 98.9% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 8309152k total, 2556968k used, 5752184k free, 165652k buffers
Swap: 9727348k total, 0k used, 9727348k free, 1266168k cached
free showed a huge cached memory amount, which accounted for the very low free memory.
There was zero swapping.
OAS was using 1.1 GB and nothing else appeared to be using much memory.
My understanding is that cached memory is normal and LINUX will re-use any of the needed cached pages.
I just have never seen an extra 4 GB of cached pages before.
We normally see TOP showing as above in the 'CURRENT' snapshot.
This is a hosted server (IBM).
There are other things on the box, such as Tivoli running.
We asked IBM engineers to look at the box to see if they noticed something we didn't.
We received a very vague response from IBM saying the cached memory was used up and the physical server had to be restarted.
They were given the green light before anyone was consulted.. and the physical server was restarted.
The out of memory and high cached memory issue did not come back, and that was the only time we have seen both occur.
Question: Is this normal for LINUX to cache so much physical memory ?
Does anyone have any thoughts on the specific out of memory error ?
Any comments in general on this issue ?
Thanks a lot !
Brian
2008-04-21 10:14:55.730 0 Error Exception java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:111)
at java.util.jar.JarFile.(JarFile.java:127)
at java.util.jar.JarFile.(JarFile.java:92)
at oracle.classloader.SharedJar.doOpen(SharedJar.java :20
at oracle.classloader.SharedCodeSource.open(SharedCod eSource.java:1124)
at oracle.classloader.SharedCodeSource.ensureOpen(Sha redCodeSource.java:94
at oracle.classloader.SharedCodeSource.getResourceByt es(SharedCodeSource.java:967)
at oracle.classloader.CodeSourceSearchPolicy.findReso urceBytes(CodeSourceSearchPolicy.java:57)
at oracle.classloader.PolicyClassLoader.findLocalClas s(PolicyClassLoader.java:1450)
at oracle.classloader.SearchPolicy$FindLocal.getClass (SearchPolicy.java:167)
at oracle.classloader.SearchSequence.getClass(SearchS equence.java:119)
at oracle.classloader.PolicyClassLoader.internalLoadC lass(PolicyClassLoader.java:1660)
at oracle.classloader.PolicyClassLoader.loadClass(Pol icyClassLoader.java:1621)
at oracle.classloader.PolicyClassLoader.loadClass(Pol icyClassLoader.java:1606)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:302)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:141)
at oracle.jdbc.ttc7.TTC7Protocol.safelyGetClassForNam e(TTC7Protocol.java:3635)
at oracle.jdbc.driver.OracleConnection.safelyGetClass ForName(OracleConnection.java:5125)
at oracle.jdbc.driver.OracleConnection.addClassMapEnt ry(OracleConnection.java:2881)
at oracle.jdbc.driver.OracleConnection.addDefaultClas sMapEntriesTo(OracleConnection.java:2872)
at oracle.jdbc.driver.OracleConnection.initializeClas sMap(OracleConnection.java:255
at oracle.jdbc.driver.OracleConnection.ensureClassMap Exists(OracleConnection.java:2552)
at oracle.jdbc.driver.OracleConnection.getTypeMap(Ora cleConnection.java:285
at glog.util.jdbc.noserver.T2SharedConnection.setType MapUser(T2SharedConnection.java:202)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:170)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:176)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:156)
at glog.util.jdbc.noserver.T2SharedConnection.setUser (T2SharedConnection.java:227)
at glog.util.datasource.ConnectionEstablisher.setVpdU ser(ConnectionEstablisher.java:2
at glog.util.datasource.GC3ConnectionEstablisher.conn ect(GC3ConnectionEstablisher.java:45)
at glog.util.datasource.GC3ConnectionPoolInstance.con nect(GC3ConnectionPoolInstance.java:212)
at glog.util.datasource.GC3ConnectionPoolInstance.res erveConnections(GC3ConnectionPoolInstance.java:205 )
at glog.util.datasource.GC3ConnectionPoolInstance.can ReserveConnection(GC3ConnectionPoolInstance.java:1 84)
at glog.util.datasource.GC3ConnectionPoolInstance.res erve(GC3ConnectionPoolInstance.java:46)
at glog.util.datasource.GC3ConnectionPool.reserve(GC3 ConnectionPool.java:36)
at glog.util.datasource.DataSource.reserve(DataSource .java:143)
at glog.util.datasource.DataSource.reserve(DataSource .java:137)
at glog.util.jdbc.noserver.T2SharedConnection$Referen cedConnection.addRef(T2SharedConnection.java:296)
at glog.util.jdbc.noserver.T2SharedConnection.open(T2 SharedConnection.java:53)
at glog.server.eventgroup.EventGroup.loadEventGroups( EventGroup.java:59)
at glog.server.eventgroup.EventGroup.get(EventGroup.j ava:17)
at glog.server.eventgroup.EventGroupClassification.lo ad(EventGroupClassification.java:82)
at glog.server.eventgroup.EventGroupClassification.ge tClassification(EventGroupClassification.java:49)
at glog.server.eventgroup.EventGroupClassification.cl assify(EventGroupClassification.java:19)
at glog.business.action.ShipmentTrackingExecutor.clas sifySsEventGroups(ShipmentTrackingExecutor.java:15 6)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addSSHistoryAndPublish(Sh ipmentStatusSessionBean.java:1054)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addShipmentStatusInternal (ShipmentStatusSessionBean.java:1013)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addShipmentStatus(Shipmen tStatusSessionBean.java:725)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionServerSideEJBWrapper.addShipme ntStatus(ShipmentStatusSessionServerSideEJBWrapper .java:220)
I was hoping one of the Linux gurus could clarify something for me..
Sometime starting over the weekend, some OTM actions.. such as change password and enter shipment events started throwing an out of memory error.
(specific error below)
OTM was ok otherwise and the memory issue seemed to occur only doing the above.
OAS and Tomcat console.log did not show the error (or any errors) and the memory usage on each, before GC, was low.
Before GC was around 500 MB on both.
The out of memory only showed in glog.app.log and on the UI screen.
Web and APP (OAS) are on their own servers.
APP/OAS is a 4x2 CPU with 8 GB ram.
During the issue yesterday, TOP showed: (APP)
Tasks: 180 total, 1 running, 179 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5% us, 0.2% sy, 0.0% ni, 99.2% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 8309152k total, 8281672k used, 27480k free, 227524k buffers
Swap: 9727348k total, 0k used, 9727348k free, 6392856k cached
CURRENT top on APP - no issues today:
Tasks: 174 total, 1 running, 173 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5% us, 0.4% sy, 0.0% ni, 98.9% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 8309152k total, 2556968k used, 5752184k free, 165652k buffers
Swap: 9727348k total, 0k used, 9727348k free, 1266168k cached
free showed a huge cached memory amount, which accounted for the very low free memory.
There was zero swapping.
OAS was using 1.1 GB and nothing else appeared to be using much memory.
My understanding is that cached memory is normal and LINUX will re-use any of the needed cached pages.
I just have never seen an extra 4 GB of cached pages before.
We normally see TOP showing as above in the 'CURRENT' snapshot.
This is a hosted server (IBM).
There are other things on the box, such as Tivoli running.
We asked IBM engineers to look at the box to see if they noticed something we didn't.
We received a very vague response from IBM saying the cached memory was used up and the physical server had to be restarted.
They were given the green light before anyone was consulted.. and the physical server was restarted.
The out of memory and high cached memory issue did not come back, and that was the only time we have seen both occur.
Question: Is this normal for LINUX to cache so much physical memory ?
Does anyone have any thoughts on the specific out of memory error ?
Any comments in general on this issue ?
Thanks a lot !
Brian
2008-04-21 10:14:55.730 0 Error Exception java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.
at java.util.jar.JarFile.
at java.util.jar.JarFile.
at oracle.classloader.SharedJar.doOpen(SharedJar.java :20

at oracle.classloader.SharedCodeSource.open(SharedCod eSource.java:1124)
at oracle.classloader.SharedCodeSource.ensureOpen(Sha redCodeSource.java:94

at oracle.classloader.SharedCodeSource.getResourceByt es(SharedCodeSource.java:967)
at oracle.classloader.CodeSourceSearchPolicy.findReso urceBytes(CodeSourceSearchPolicy.java:57)
at oracle.classloader.PolicyClassLoader.findLocalClas s(PolicyClassLoader.java:1450)
at oracle.classloader.SearchPolicy$FindLocal.getClass (SearchPolicy.java:167)
at oracle.classloader.SearchSequence.getClass(SearchS equence.java:119)
at oracle.classloader.PolicyClassLoader.internalLoadC lass(PolicyClassLoader.java:1660)
at oracle.classloader.PolicyClassLoader.loadClass(Pol icyClassLoader.java:1621)
at oracle.classloader.PolicyClassLoader.loadClass(Pol icyClassLoader.java:1606)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:302)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:141)
at oracle.jdbc.ttc7.TTC7Protocol.safelyGetClassForNam e(TTC7Protocol.java:3635)
at oracle.jdbc.driver.OracleConnection.safelyGetClass ForName(OracleConnection.java:5125)
at oracle.jdbc.driver.OracleConnection.addClassMapEnt ry(OracleConnection.java:2881)
at oracle.jdbc.driver.OracleConnection.addDefaultClas sMapEntriesTo(OracleConnection.java:2872)
at oracle.jdbc.driver.OracleConnection.initializeClas sMap(OracleConnection.java:255

at oracle.jdbc.driver.OracleConnection.ensureClassMap Exists(OracleConnection.java:2552)
at oracle.jdbc.driver.OracleConnection.getTypeMap(Ora cleConnection.java:285

at glog.util.jdbc.noserver.T2SharedConnection.setType MapUser(T2SharedConnection.java:202)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:170)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:176)
at glog.util.jdbc.noserver.T2SharedConnection.setCach edUser(T2SharedConnection.java:156)
at glog.util.jdbc.noserver.T2SharedConnection.setUser (T2SharedConnection.java:227)
at glog.util.datasource.ConnectionEstablisher.setVpdU ser(ConnectionEstablisher.java:2

at glog.util.datasource.GC3ConnectionEstablisher.conn ect(GC3ConnectionEstablisher.java:45)
at glog.util.datasource.GC3ConnectionPoolInstance.con nect(GC3ConnectionPoolInstance.java:212)
at glog.util.datasource.GC3ConnectionPoolInstance.res erveConnections(GC3ConnectionPoolInstance.java:205 )
at glog.util.datasource.GC3ConnectionPoolInstance.can ReserveConnection(GC3ConnectionPoolInstance.java:1 84)
at glog.util.datasource.GC3ConnectionPoolInstance.res erve(GC3ConnectionPoolInstance.java:46)
at glog.util.datasource.GC3ConnectionPool.reserve(GC3 ConnectionPool.java:36)
at glog.util.datasource.DataSource.reserve(DataSource .java:143)
at glog.util.datasource.DataSource.reserve(DataSource .java:137)
at glog.util.jdbc.noserver.T2SharedConnection$Referen cedConnection.addRef(T2SharedConnection.java:296)
at glog.util.jdbc.noserver.T2SharedConnection.open(T2 SharedConnection.java:53)
at glog.server.eventgroup.EventGroup.loadEventGroups( EventGroup.java:59)
at glog.server.eventgroup.EventGroup.get(EventGroup.j ava:17)
at glog.server.eventgroup.EventGroupClassification.lo ad(EventGroupClassification.java:82)
at glog.server.eventgroup.EventGroupClassification.ge tClassification(EventGroupClassification.java:49)
at glog.server.eventgroup.EventGroupClassification.cl assify(EventGroupClassification.java:19)
at glog.business.action.ShipmentTrackingExecutor.clas sifySsEventGroups(ShipmentTrackingExecutor.java:15 6)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addSSHistoryAndPublish(Sh ipmentStatusSessionBean.java:1054)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addShipmentStatusInternal (ShipmentStatusSessionBean.java:1013)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionBean.addShipmentStatus(Shipmen tStatusSessionBean.java:725)
at glog.server.executionapi.servprov.shipmentstatus.S hipmentStatusSessionServerSideEJBWrapper.addShipme ntStatus(ShipmentStatusSessionServerSideEJBWrapper .java:220)
Comment