EHC ❯ ConcurrentModificationException while replication with JGroups

• Type: Bug
• Status: Closed
• Priority: 2 Major
• Resolution: Not a Bug
• Component/s: ehcache-core
• Labels:

• Assignee: cdennis
• Reporter: kehren
• Created: December 02, 2011
• Votes: 0
• Watchers: 2
• Updated:

  July 27, 2012
• Resolved:

  December 06, 2011

Description

we’re using ehcache 2.3.2 with jgroups replication 1.4. Sometimes a ConcurrentModificationException occurs in our logs:

2011-12-01 19:30:41,226 ERROR [net.sf.ehcache.distribution.jgroups.JGroupsCachePeer] (Async Replication Thread) Error serializing data, it will not be sent: …. java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) at java.util.HashMap$KeyIterator.next(HashMap.java:828) at java.util.HashSet.writeObject(HashSet.java:267) at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1469) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:422) at net.sf.ehcache.Element.writeObject(Element.java:787) at sun.reflect.GeneratedMethodAccessor1688.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1469) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at net.sf.ehcache.distribution.EventMessage.writeObject(EventMessage.java:134) at sun.reflect.GeneratedMethodAccessor1685.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1469) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at java.util.ArrayList.writeObject(ArrayList.java:570) at sun.reflect.GeneratedMethodAccessor2027.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1469) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at org.jgroups.util.Util.objectToByteBuffer(Util.java:341) at net.sf.ehcache.distribution.jgroups.JGroupsCachePeer.sendData(JGroupsCachePeer.java:229) at net.sf.ehcache.distribution.jgroups.JGroupsCachePeer.flushQueue(JGroupsCachePeer.java:278) at net.sf.ehcache.distribution.jgroups.JGroupsCachePeer.access$200(JGroupsCachePeer.java:53) at net.sf.ehcache.distribution.jgroups.JGroupsCachePeer$AsyncTimerTask.run(JGroupsCachePeer.java:299) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

Comments

Tim Eck 2011-12-02

I can’t flatly assert this is what is going on, but ConcurrentModificationException like this is usually the result of the application modifying object(s) reachable from an element already placed in the cache.

Chris Dennis 2011-12-02

The stack trace indicates that while the JGroups code is trying to serialize one of the keys or values in order to send it over the wire while something else in your application is concurrently mutating it. The type being mutated is a HashSet - do you have a HashSet referenced from one of your key or value types that could be being concurrently mutated like this?

Peter Kehren 2011-12-03

yes, that’s right. The cached elements contain HashSets, which are potential changed in the time of replication.