• Bug
  • Status: Open
  • 2 Major
  • Resolution:
  • ehcache-terracotta
  • serverteam
  • Reporter: gadams00
  • April 08, 2013
  • 0
  • Watchers: 10
  • July 02, 2014

Attachments

Description

Redeploying (or stopping and stopping) a war deployment on tomcat7 that instantiates (and properly shuts down) a CacheManager where the ehcache.xml includes terracotta distributed caches causes tomcat to issue its severe warnings about several threads that ehcache starts not being stopped:

Apr 8, 2013 10:49:49 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/ehcache-war-leak-0.0.1-SNAPSHOT] appears to have started a thread named [JarManager idle thread] but has failed to stop it. This is very likely to create a memory leak.
Apr 8, 2013 10:49:49 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/ehcache-war-leak-0.0.1-SNAPSHOT] appears to have started a thread named [non stop reaper thread] but has failed to stop it. This is very likely to create a memory leak.

If you use VisualVM to monitor permgen heap usage for the tomcat process, you can see that the WebappClassLoader is unable to GC, which makes it so that you can only redeploy a handful of times until permgen is exhausted and you get OutOfMemory exceptions.

I’ve attached a simple maven project that packages a war that just contains a ServletContextListener that calls CacheManager.create() on contextInitialized and CacheManager.getInstance().shutdown() on contextDestroyed.

To reproduce the issue,

  1. download and extract terracotta open source 3.7.4 and start the terracotta server with default configuration.
  2. Download and install tomcat 7 (latest version) and start it
  3. extract the attached ehcache-war-leak zip and run mvn package in the root
  4. Deploy target/ehcache-war-leak-0.0.1-SNAPSHOT.war to tomcat
  5. Via the tomcat manager, stop and start the ehcache-war-leak webapp and monitor the catalina log (stdout) for the severe warnings described above.
  6. use visual vm to monitor permgen size and see that tomcat is unable to GC the WebappClassLoader because of the threads not being stopped in time

I’ve noticed that if you get rid of the nonstop configuration in ehcache.xml that the nonstop reaper thread goes away from the threads that aren’t stopped by the application. Jar manager idle thread remains whenever terracotta caches are used. I also noticed that if you attach a debugger and stop and step over the CacheManager.getInstance().shutdown() call, the ehcache threads (JarManager idle thread and nonstop reaper thread) will eventually terminate, but this is done in a background thread, so perhaps this is a race condition where tomcat expects the threads to be terminated, but they’re not quite done yet. A simple fix might be to not background the termination process for those threads.

Comments

Tim Eck 2013-04-09

I’m not sure if the “non stop reaper thread” is indicative of a problem but the other thread listed there “jar manager idle thread” should pass away naturually although toolkit shutdown perhaps is missing a wait for it to die. One thing of note about the jar manager thread is that it no longer exists in 4.0.0.

Perhaps platform team should take the first look at this one

Eugene Shelestovich 2013-04-09

Such permgen leaks in Tomcat are usually caused by non-daemon threads with ThreadLocal variables. A ThreadLocal holds a reference to an object and prevents whole ClassLoader root to be GCed across Tomcat restarts. The possible solution would be to make the thread daemon, so it will not prevent jvm from exiting. Or locate a problematic ThreadLocal variable and write a proper cleaning up procedure.

Tim Eck 2013-04-09

for sure threadlocals can be problematic. With regards to the “non stop reaper thread” can you confirm whether it is probably stopped in 3.7.x?

Greg Adams 2013-04-09

I’ve tested using

  1. terracotta opensource 3.7.4 and ehcache 2.6.5
  2. terracotta opensource 3.7.5 and ehcache 2.6.6

the problem reported is evident in both versions.

I’ve also tried moving ehcache-terracotta, ehcache-core, terracotta-toolkit-1.6-runtime-5.5.0 jars to tomcat/lib (out of the war) to see if this problem went away when the jars were loaded using the tomcat server classloader, and it made no difference.

Tim Wu 2013-04-18

Seems like this is a 3.7.x thing. Our express client implementation has changed a bit since then, so this probably isn’t applicable to 4.0.x

Greg Adams 2013-04-23

Actually, when using BigMemory Max 4.0.0 tomcat doesn’t issue the unstopped threads message, but there is still a permgen leak.

Mark Cade 2013-12-18

I have a prospect that is using BigMemory MAX 4.0.5 experiencing the same symptoms. Do you have an idea which version this will be resolved in?

Danilo Fernandez 2014-07-02

Hi, is there any news regarding this problem?, I’ using

  • terracotta opensource 3.7.4 and ehcache 2.6.5 thank you