• Bug
  • Status: Closed
  • 2 Major
  • Resolution: Cannot Reproduce
  • Sessions
  • interfaces
  • Reporter: eric.sun
  • July 30, 2009
  • 0
  • Watchers: 2
  • December 16, 2011
  • October 12, 2009

Attachments

  • eric.sun (113.00 k) application/octet-stream session.war

Description

The issue was arisen from our production and I reproduced it in our QA server, the following is the environment of the QA JDK 1.6 2 Apache instances 3 Tomcat6.0.18 instances Terracotta3.0.0 active + Terracotta3.0.0 passive

Keep sending ssl login requests and plain http user profile request to our webapp for 30-40 minutes, 20-30 requests per second. The requests sent by a small app writen with Apache HttpClient 3.1.

I am expecting HttpSessionListener.sessionDestroyed() get called back when the sessions associated with those requests expire, which is in 30 minutes.But some of calls got delayed and some of them missed the calls. So I suspect there’s leak or lock holding the session from invalidting.

I did more testing on my dev environment( 2 tomcats, 1 Apache Http Server without ssl enabled, 1 terracotta, same client), i can only reproduce the delayed invalidating but no leak. The attachment is a simplified web app, i did not reproduce the issue with it on my dev pc, but it can give you an idea how our web looks like.

Thanks

Comments

Tim Eck 2009-07-30

I’ll have to give the sample a look, but one thing that is often confusing is that he created() and destroyed() events might not happen on the same VM (in the event that you have more than one tomcat node). It is more or less random where the destroy() callback will happen (it has no relation to where the session was last accessed in particular)

eric sun 2009-07-31

Thanks Tim That is very true. But I did grep stuck sessionid across logs of all tomcats, they were nowhere. Plus,we have a ONLINEUSER List and a USERSESSION<User, SessionID> Map, which are both synchronized by Terracotta, they had not been cleaned up for weeks. I saw those sessionid were in those collections via Terracotta Dev console. According to, we suspecte there might be an issue in terracotta. Correct me if my guess doesn't sound reasonable.

Tim Eck 2009-07-31

It sounds like your methodology and the resulting are reasonable and do point towards a bug in TC. That said we do have a test that is almost exactly like this. It creates 4 nodes, creates thousands of sessions and then makes sure a listener sees all of the created and destroyed events.

One thing that can prevent a sessionDestroyed() from getting called is there are more than one listener and one of them throws an exception. In this case we do not run all the listeners. We should be logging this in the terracotta client log file though.

This code is not super easy to read since parts of it deal with our web testing framework, but if you’re curious the test code I’m referring to his here: http://svn.terracotta.org/svn/forge/projects/tim-session/trunk/tim-session-system-tests/src/test/java/com/tctest/server/appserver/load/MultiNodeInvalidatorTest.java http://svn.terracotta.org/svn/forge/projects/tim-session/trunk/tim-session-system-tests/src/test/java/com/tctest/webapp/listeners/MultiNodeInvalidatorListener.java

All this said, I’d love to find (and fix) a bug here. Just need to reproduce it first :-)

eric sun 2009-07-31

Thanks Tim

We only have one HTTP Session listener on each instance and I did another round test, did not find exception thrown from Destory method.

Disabling ssl doesn’t help solve it, now we are doing the test with only 1 tomcat + 1 terracotta + 1 apache.

Do you think this has anything to do with the delayed invalidating?

Fiona OShea 2009-08-12

Does anyone have any ideas on this one?

Tim Eck 2009-10-12

It is possible (though unknown) that this linked bug is the culprit here