I never did figure this one out. My hunch is a threading issue in log4j i.e. “upgrade to log4j2”? Any other ideas out there?
“bindingRegistry-pool-5-thread-1” #43 daemon prio=5 os_prio=0 tid=0x00007f79409f1000 nid=0x2e71 waiting for monitor entry [0x00007f794416b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.ResourceBundle$CacheKey.clone(ResourceBundle.java:657)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1462)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1419)
at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)
at java.util.ResourceBundle.getBundle(ResourceBundle.java:890)
at sun.util.resources.LocaleData$1.run(LocaleData.java:167)
at sun.util.resources.LocaleData$1.run(LocaleData.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at sun.util.resources.LocaleData.getBundle(LocaleData.java:163)
at sun.util.resources.LocaleData.getDateFormatData(LocaleData.java:127)
at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:710)
at java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:145)
at sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
at java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:364)
at java.text.DateFormatSymbols.getInstance(DateFormatSymbols.java:340)
at java.util.Calendar.getDisplayName(Calendar.java:2110)
at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1125)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:966)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936)
at java.text.DateFormat.format(DateFormat.java:345)
at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:443)
at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
– locked <0x00000006c142ccb0> (a org.apache.log4j.RollingFileAppender)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
– locked <0x00000006c13fbe78> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:444)
“Camel (camel-5) thread #14 – JmsConsumer[eat.jobs.film]” #198736 daemon prio=5 os_prio=0 tid=0x0000000003410800 nid=0x6026 runnable [0x00007f793e298000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
at java.io.File.lastModified(File.java:943)
at org.apache.logging.log4j.core.config.FileConfigurationMonitor.checkConfiguration(FileConfigurationMonitor.java:81)
at org.apache.logging.log4j.core.Logger$PrivateConfig.filter(Logger.java:328)
at org.apache.logging.log4j.core.Logger.isEnabled(Logger.java:132)
at org.apache.logging.log4j.spi.AbstractLogger.isTraceEnabled(AbstractLogger.java:591)
at com.singularity.ee.agent.util.log4j.x.isTraceEnabled(x.java:134)
at com.singularity.ee.agent.util.log4j.h.isTraceEnabled(h.java:186)
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.H.<init>(
“bindingRegistry-pool-5-thread-1” #43 daemon prio=5 os_prio=0 tid=0x00007f79409f1000 nid=0x2e71 waiting for monitor entry [0x00007f794416d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)
at java.lang.StringBuilder.<init>(StringBuilder.java:89)
at java.lang.StackTraceElement.toString(StackTraceElement.java:172)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at java.lang.Throwable.printStackTrace(Throwable.java:658)
– locked <0x0000000766e638a0> (a java.io.PrintWriter)
at java.lang.Throwable.printStackTrace(Throwable.java:721)
at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
– locked <0x0000000766e48cb8> (a org.apache.log4j.spi.ThrowableInformation)
at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
– locked <0x00000006c142ccb0> (a org.apache.log4j.RollingFileAppender)
“pool-16-thread-1” #198750 prio=5 os_prio=0 tid=0x00007f7940bc8000 nid=0x6038 runnable [0x00007f79170ef000]
java.lang.Thread.State: RUNNABLE
at org.postgresql.core.v3.QueryExecutorImpl.sendSync(QueryExecutorImpl.java:1334)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:286)
– locked <0x00000006c4d355d8> (a org.postgresql.core.v3.QueryExecutorImpl)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:262)
GC explain the 6 cores?
Log4j bug?
Remove console appender?
We know log4j seems to be eating one core
Hold onto more logs
Visualvm???
She’s unresponsive, Jim!
Fix it so it’s fast UI
Fix it so it run film validation threaded
“See” if we can just start film validation…continue it…
Seems to have run out of RAM
From a mystery cause!
Logging?