• Bug
  • Status: Resolved
  • 2 Major
  • Resolution: Not a Bug
  • drb
  • Reporter: omkreddy
  • May 07, 2013
  • 0
  • Watchers: 4
  • May 28, 2013
  • May 28, 2013

Attachments

Description

Version: BigMemory GO 3.7.2 O/S : IBM Java

Ehcache.xml:

Scenario: Multiple threads are reading and writing to the cache without any external synchronization.

Stack trace of hung thread:

3XMJAVALTHREAD (java/lang/Thread getId:0x4E865, isDaemon:false) 3XMTHREADINFO1 (native thread ID:0x1170193, native priority:0x5, native policy:UNKNOWN) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at org/apache/log4j/Category.callAppenders(Category.java:204(Compiled Code)) 4XESTACKTRACE at org/apache/log4j/Category.forcedLog(Category.java:391(Compiled Code)) 4XESTACKTRACE at org/apache/log4j/Category.log(Category.java:856(Compiled Code)) 4XESTACKTRACE at org/slf4j/impl/Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:305(Compiled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/disk/storage/FileBackedStorageEngine.readFromChannel(FileBackedStorageEngine.java:388(Compile d Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/disk/storage/FileBackedStorageEngine.readIntFromChannel(FileBackedStorageEngine.java:334(Comp iled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/disk/storage/FileBackedStorageEngine.access$300(FileBackedStorageEngine.java:43(Compiled Code )) 4XESTACKTRACE at com/terracottatech/offheapstore/disk/storage/FileBackedStorageEngine$FileChunk.readKeyBuffer(FileBackedStorageEngine.java:452 (Compiled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/disk/storage/FileBackedStorageEngine.readKeyBuffer(FileBackedStorageEngine.java:257(Compiled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/storage/PortabilityBasedStorageEngine.equalsKey(PortabilityBasedStorageEngine.java:108(Compil ed Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/OffHeapHashMap.keyEquals(OffHeapHashMap.java:914(Compiled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/OffHeapHashMap.removeNoReturn(OffHeapHashMap.java:788(Compiled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/AbstractLockedOffHeapHashMap.removeNoReturn(AbstractLockedOffHeapHashMap.java:154(Compiled Co de)) 4XESTACKTRACE at com/terracottatech/offheapstore/concurrent/AbstractConcurrentOffHeapMap.removeNoReturn(AbstractConcurrentOffHeapMap.java:210( Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/store/offheap/disk/OffHeapDiskStore.removeNoReturn(OffHeapDiskStore.java:176(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/store/FrontEndCacheTier.remove(FrontEndCacheTier.java:316(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/Cache.removeInternal(Cache.java:2340(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/Cache.remove(Cache.java:2253(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/Cache.remove(Cache.java:2178(Compiled Code)) 4XESTACKTRACE at in/co/nmsworks/cache/Cache.remove(Cache.java:71(Compiled Code))

Ehcache logs: below log is coming continuously

FileBackedStorageEngine: readFromChannel(): Creating new read-channel for Cache.data as previous one was closed (likely due to interrupt)

Stack trace of waiting threads:

XMTHREADINFO3 Java callstack: 4XESTACKTRACE at sun/misc/Unsafe.park(Native Method) 4XESTACKTRACE at java/util/concurrent/locks/LockSupport.park(LockSupport.java:182(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822(Compiled Code )) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:952(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1272(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/locks/ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:698(Compiled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/AbstractLockedOffHeapHashMap.get(AbstractLockedOffHeapHashMap.java:86(Compiled Code)) 4XESTACKTRACE at com/terracottatech/offheapstore/concurrent/AbstractConcurrentOffHeapMap.get(AbstractConcurrentOffHeapMap.java:183(Compiled Co de)) 4XESTACKTRACE at net/sf/ehcache/store/offheap/disk/OffHeapDiskStore.get(OffHeapDiskStore.java:151(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/store/offheap/disk/OffHeapDiskStore.getQuiet(OffHeapDiskStore.java:155(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/store/FrontEndCacheTier.getQuiet(FrontEndCacheTier.java:230(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/Cache.searchInStoreWithoutStats(Cache.java:2074(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/Cache.getQuiet(Cache.java:1864(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/Cache.putIfAbsent(Cache.java:3845(Compiled Code)) 4XESTACKTRACE at net/sf/ehcache/Cache.putIfAbsent(Cache.java:3825(Compiled Code))

Comments

Tim Eck 2013-05-13

Chris – I can’t really tell from this, but usually when I see things stuck in a logging call like that I think either a bad disk, or the stupid problem on windows where if you select and text in the console window all I/O calls block until you release the text selection. Can’t see that this is (or is not) windows.

Chris Dennis 2013-05-13

The thread blocked in the logging is trying to take a lock within log4j. I strongly suspect there is another thread stuck within the logging system that is holding this one up. Given the nature of logging this could be a thread stuck doing some kind of IO process (file and/or network). If you posted the full thread dump it should be possible to track down which thread is causing the hold-up.

Kumar Reddy 2013-05-13

As i mentioned in the description the below log is printing continuously in the logs.

“FileBackedStorageEngine: readFromChannel(): Creating new read-channel for Cache.data as previous one was closed (likely due to interrupt)”

This means no problem with logging thread.

I attached the thread dump for your reference.

Kumar Reddy 2013-05-28

This is not an issue. This problem is due to improper handling on thread interrupted status in our code. Hence please close the issue. Thanks for your support.