Announcement

Collapse
No announcement yet.

Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 2883334

Collapse
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 2883334

    Since beginning of this week UPM changed their platform from HP-UX to Linux in Production, but on the webservers we are facing following issue :
    Has anybody see this message before because we were not facing this message during our testing :

    in console.log :

    INFO | jvm 1 | 2007/12/05 16:32:09 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 2883334

    On UI :

    java.lang.OutOfMemoryError: allocLargeArray - Object size: 5766688, Num elements: 2883334
    at java.lang.StringCoding$CharsetSD.decode([BII)[C(Unknown Source)
    at java.lang.StringCoding.decode(Ljava.lang.String;[BII)[C(Unknown Source)
    at java.lang.String.([BIILjava.lang.StringV(Unknown Source)
    at java.io.ByteArrayOutputStream.toString(ByteArrayOu tputStream.java:174)
    at glog.server.producer.ProducerXML.toXML(ProducerXML .java:73)
    at glog.webserver.util.Producer.createMasterElementRe mote(Producer.java:159)
    at glog.webserver.util.Producer.loadDocument(Producer .java:59)
    at glog.webserver.util.AbstractManagementServlet.getD ocument(AbstractManagementServlet.java:32
    at glog.webserver.util.AbstractServletProducer.proces s(AbstractServletProducer.java:80)
    at glog.webserver.util.BaseServlet.service(BaseServle t.java:591)
    at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
    at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
    at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
    at glog.webserver.screenlayout.ClientSessionTracker.d oFilter(ClientSessionTracker.java:54)
    at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:202)
    at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
    at glog.webserver.util.SetCharacterEncodingFilter.doF ilter(SetCharacterEncodingFilter.java:44)
    at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:202)
    at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
    at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:17
    at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
    at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
    at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
    at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:14
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyo teHandler.java:199)
    at org.apache.jk.common.HandlerRequest.invoke(Handler Request.java:282)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelS ocket.java:754)
    at org.apache.jk.common.ChannelSocket.processConnecti on(ChannelSocket.java:684)
    at org.apache.jk.common.ChannelSocket$SocketConnectio n.runIt(ChannelSocket.java:876)
    at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
    at java.lang.Thread.run()V(Unknown Source)

    java version "1.4.2_11"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_11-b06)
    BEA JRockit(R) (build R26.4.0-63-63688-1.4.2_11-20060626-2259-linux-ia32, )


    # Java Additional Parameters
    wrapper.java.additional.1=-jrockit
    wrapper.java.additional.2=-Xms2800m
    wrapper.java.additional.3=-Xmx2800m
    wrapper.java.additional.4=-Xgcprioausetime
    wrapper.java.additional.5=-Xverbose:memory
    wrapper.java.additional.6=-Xnoclassgc
    wrapper.java.additional.7=-DGC3EncodedPassword=Q0hBTkdFTUU=
    wrapper.java.additional.8=-Dglog.properties=glog.tomcat.properties
    wrapper.java.additional.9=-Dglog.home=%GLOG_HOME%/glog/config
    wrapper.java.additional.10=-Duser.home=%GLOG_HOME%/glog/config
    wrapper.java.additional.11=-Djava.io.tmpdir=%GLOG_HOME%/temp
    wrapper.java.additional.12=-Djava.awt.headless=true
    wrapper.java.additional.13=-Djava.security.auth.login.config=%GLOG_HOME%/glog/config/gc3_jaas.config
    wrapper.java.additional.14=-Dweblogic.ThreadPoolSize=15
    wrapper.java.additional.15=-Djava.endorsed.dirs=%GLOG_HOME%/tomcat/common/endorsed
    wrapper.java.additional.16=-Dcatalina.base=%GLOG_HOME%/tomcat
    wrapper.java.additional.17=-Dcatalina.home=%GLOG_HOME%/tomcat
    wrapper.java.additional.18=-Xmanagement
    Tags: alloclargearray, elements, num, object, otmfaq.com, outofmemory, size, throwing
  • #2
    Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

    Try changing them from 2800 to 2400 and see if the system improves. Usually for tomcat you shouldn't need to increase the JVM past 1536.
    If my post was helpful please click on the Thanks! button

    MavenWire Hosting Admin
    15 years of OTM experience

    Comment

    • #3
      Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

      I definitely agree with Nick. I had this issue with a large client in APAC. Setting the heap to 2800MB didn't leave enough extra memory in the 32-bit memory space to handle the rest of JRockit's memory requirements, particularly around thread stacks.

      Let us know if 2400MB works!

      --Chris
      Chris Plough
      twitter.com/chrisplough
      MavenWire

      Comment

      • #4
        Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

        It is definitely Tomcat. We took over this 2800Mb setting from HP-UX when we changed platforms from HP-UX to Linux...But anyway, I decreased memory now back to 2400 Mb.

        Will let you know the results if the OOM is gone.

        Thx,

        Kristof

        Comment

        • #5
          Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

          Today we were facing the same issue again after reducing memory from 2800 Mb to 2400 Mb on Tomcat. We'll update memory back to 2Gb.

          INFO | jvm 1 | 2007/12/05 16:20:47 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7458224, Num elements: 3729102
          INFO | jvm 1 | 2007/12/05 16:32:09 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 4194320, Num elements: 2097150
          INFO | jvm 1 | 2007/12/05 17:07:25 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/05 17:07:45 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/05 17:07:58 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/05 17:08:15 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 14704656, Num elements: 14704640
          INFO | jvm 1 | 2007/12/05 17:09:43 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7434256, Num elements: 7434240
          INFO | jvm 1 | 2007/12/05 17:10:00 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/05 17:10:09 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/05 17:10:22 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/05 17:10:38 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/05 17:11:06 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7352336, Num elements: 7352320
          INFO | jvm 1 | 2007/12/06 16:09:28 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 17527952, Num elements: 8763966
          INFO | jvm 1 | 2007/12/06 16:24:46 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 17527952, Num elements: 8763966
          INFO | jvm 1 | 2007/12/06 16:25:03 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7331856, Num elements: 7331840
          INFO | jvm 1 | 2007/12/06 16:25:13 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7331856, Num elements: 7331840
          INFO | jvm 1 | 2007/12/06 16:25:24 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7331856, Num elements: 7331840
          INFO | jvm 1 | 2007/12/06 16:25:37 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 17527952, Num elements: 8763966
          INFO | jvm 1 | 2007/12/06 16:26:58 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7799168, Num elements: 3899576
          INFO | jvm 1 | 2007/12/07 13:00:53 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7487504, Num elements: 7487488
          INFO | jvm 1 | 2007/12/07 13:01:28 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7487504, Num elements: 7487488
          INFO | jvm 1 | 2007/12/07 13:03:07 | [memory ] Throwing OutOfMemory: allocLargeArray - Object size: 7446544, Num elements: 7446528

          Comment

          • #6
            Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

            If you are still getting the out of memory errors, then something else is causing the problem. Have you performed thread dumps to see what is actively running? Please post those and I'll see what I can find. Usually this is caused by someone using SQL backdoor and putting a large query into it, or a CSV export.
            If my post was helpful please click on the Thanks! button

            MavenWire Hosting Admin
            15 years of OTM experience

            Comment

            • #7
              Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

              Sometimes this message appears on the UI screen when a user is opening a shipment.
              Never seen it in SQL Backdoor or during CSV export/import. Then it's already too late to make a threaddump on the webserver.
              We are facing this issue since we changed our Prod platform from HP-UX to Red Hat Linux. Don't you think it will help if we reduce memory back from 2400Mb to 2000Mb?

              Kristof

              Comment

              • #8
                Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

                Typically, with out of memory errors, the root cause of the problem come in lines 6 & 7 in the error. You should try to correlate those errors with a business user who is generating the error. In other words, reproduce the problem. Once you do, setup a web conf with support to show them what is causing it.

                Some of problems are caused by the application attempting to allocate more memory than it needs. (IOW an OTM BUG ) You can see in the memory errors that it is doubling at some point, which is not a good sign.

                You have plenty of memory to run the app.

                lines 6 & 7:

                at glog.server.producer.ProducerXML.toXML(ProducerXML .java:73)
                at glog.webserver.util.Producer.createMasterElementRe mote(Producer.java:159)

                Comment

                • #9
                  Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

                  Originally posted by Kristof Stevens View Post
                  Sometimes this message appears on the UI screen when a user is opening a shipment.
                  Never seen it in SQL Backdoor or during CSV export/import. Then it's already too late to make a threaddump on the webserver.
                  We are facing this issue since we changed our Prod platform from HP-UX to Red Hat Linux. Don't you think it will help if we reduce memory back from 2400Mb to 2000Mb?

                  Kristof

                  Kristof,

                  I wasn't referring to where the issue would appear but what the issue would have been caused by. Getting the error in the UI is only a symptom that the system is out of memory. What caused it is totally unrelated. The only true way to know what happened is with the thread dumps. You need to have at least 3 every 30 seconds but I would recommend having at least 6 to get a true representation of where the issue is.

                  Nick
                  If my post was helpful please click on the Thanks! button

                  MavenWire Hosting Admin
                  15 years of OTM experience

                  Comment

                  • #10
                    Re: Throwing OutOfMemory: allocLargeArray - Object size: 5766688, Num elements: 28833

                    Hi Kristof,

                    Recently after migration from 4.5 to 5.5, our client too had a similar scenario where the App Server (Instead of Web Server) was going out of memory, and we had a situation to restart the Prod environment 2-3 times in day.
                    Well after a weeks effort of constant analysis, we could find that one of our agent which gets triggered when Invoice gets created through integration had a Direct Sql Update (DSU) insert statement. As per Oracle Development team and support, the recommendation was to uncheck the Refresh Cache checkbox for all Insert statements (Which by default enabled in 5.5 for all DSUs). And luckily after this change in the agent, its now two weeks now, we didn't face this situation again of Out of Memory. OOM reminded that we also codenamed this issue as OOM

                    I would also recommend to check by enabling Agent, AgentUtility and SQL enabled to check if there is pattern when OOM occurs, do you see that at a specific time this issue appears during a day, also removing of threaddumps as suggested by Nick would be of great help for Oracle Support... well I agree that the configuration on workflow may be different on your setup compared to ours, but still I thought to share my experience.

                    Please do checkout some more loggings in all the enabled logs present in OTM...

                    Regards,
                    Suresh

                    Comment

                    Previous template Next
                    Working...
                    X