CDV ❯ Server shutdown due a deadlock in BerkelyDB between flush thread and GC thread.

• Type: Bug
• Status: Closed
• Priority: 2 Major
• Resolution: Fixed
• Component/s:
• Labels:

• Assignee: hhuynh
• Reporter: ssubbiah
• Created: November 05, 2007
• Votes: 0
• Watchers: 0
• Updated:

  March 31, 2008
• Resolved:

  November 09, 2007

Description

This is from the forum post http://forums.terracotta.org/forums/posts/list/588.page

From the logs :

2007-11-04 19:42:24,540 [WorkerThread(managed_object_flush_stage,3)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(managed_object_flush_stage,3),5,TC Thread Group] got an uncaught exception. About to sleep then exit. com.tc.objectserver.persistence.sleepycat.TCDatabaseException: com.sleepycat.je.DeadlockException: (JE 3.2.23) Lock expired. Locker 30970876_WorkerThread(managed_object_flush_stage,3)_Txn: waited for lock on database=objects node=3409750 type=WRITE grant=WAIT_NEW timeoutMillis=180000 startTime=1194233786667 endTime=1194234144540 Owners: [] Waiters: [] Transaction 30970875_GC_Txn waits for node 1464631 Transaction 30970876_WorkerThread(managed_object_flush_stage,3)_Txn owns 1464631 Transaction 30970876_WorkerThread(managed_object_flush_stage,3)_Txn waits for node 3409750

at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.basicSaveObject(ManagedObjectPersistorImpl.java:280) at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.saveAllObjects(ManagedObjectPersistorImpl.java:315) at com.tc.objectserver.impl.PersistentManagedObjectStore.commitAllObjects(PersistentManagedObjectStore.java:81) at com.tc.objectserver.impl.ObjectManagerImpl.flushAll(ObjectManagerImpl.java:614) at com.tc.objectserver.impl.ObjectManagerImpl.flushAndEvict(ObjectManagerImpl.java:801) at com.tc.objectserver.handler.ManagedObjectFlushHandler.handleEvent(ManagedObjectFlushHandler.java:22) at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:140)

Caused by: com.sleepycat.je.DeadlockException: (JE 3.2.23) Lock expired. Locker 30970876_WorkerThread(managed_object_flush_stage,3)_Txn: waited for lock on database=objects node=3409750 type=WRITE grant=WAIT_NEW timeoutMillis=180000 startTime=1194233786667 endTime=1194234144540 Owners: [] Waiters: [] Transaction 30970875_GC_Txn waits for node 1464631 Transaction 30970876_WorkerThread(managed_object_flush_stage,3)_Txn owns 1464631 Transaction 30970876_WorkerThread(managed_object_flush_stage,3)_Txn waits for node 3409750

at com.sleepycat.je.txn.LockManager.lock(LockManager.java:266) at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:295) at com.sleepycat.je.txn.Locker.lock(Locker.java:257) at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2349) at com.sleepycat.je.tree.Tree.insert(Tree.java:2567) at com.sleepycat.je.dbi.CursorImpl.putLN(CursorImpl.java:988) at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:1011) at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:861) at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:798) at com.sleepycat.je.Cursor.putInternal(Cursor.java:752) at com.sleepycat.je.Database.putInternal(Database.java:662) at com.sleepycat.je.Database.put(Database.java:608) at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.basicSaveObject(ManagedObjectPersistorImpl.java:270) … 6 more

2007-11-04 19:42:24,712 [GC] ERROR com.tc.server.TCServerMain - Thread:Thread[GC,5,TC Thread Group] got an uncaught exception. About to sleep then exit. com.sleepycat.je.DeadlockException: (JE 3.2.23) Lock expired. Locker 30970875_GC_Txn: waited for lock on database=objects node=1464631 type=WRITE grant=WAIT_NEW timeoutMillis=180000 startTime=1194233786839 endTime=1194234144712 Owners: [] Waiters: []

at com.sleepycat.je.txn.LockManager.lock(LockManager.java:266) at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:295) at com.sleepycat.je.txn.Locker.lock(Locker.java:257) at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2349) at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2271) at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:1963) at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1178) at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1148) at com.sleepycat.je.Cursor.search(Cursor.java:1014) at com.sleepycat.je.Database.deleteInternal(Database.java:473) at com.sleepycat.je.Database.delete(Database.java:405) at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.deleteObjectByID(ManagedObjectPersistorImpl.java:358) at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.deleteAllObjectsByID(ManagedObjectPersistorImpl.java:372) at com.tc.objectserver.impl.PersistentManagedObjectStore.removeAllObjectsByIDNow(PersistentManagedObjectStore.java:86) at com.tc.objectserver.impl.ObjectManagerImpl.removeFromStore(ObjectManagerImpl.java:600) at com.tc.objectserver.impl.ObjectManagerImpl.notifyGCComplete(ObjectManagerImpl.java:585) at com.tc.objectserver.core.impl.MarkAndSweepGarbageCollector.gc(MarkAndSweepGarbageCollector.java:176) at com.tc.objectserver.impl.ObjectManagerImpl$1.run(ObjectManagerImpl.java:686)

2007-11-04 19:42:27,556 [CommonShutDownHook - com.tc.object.msg.RequestManagedObjectResponseMessage$1@1142e30] INFO com.tc.net.protocol.TCNetworkMessage - No of times Buffers wasted = 0 Buffers wasted count = 0 2007-11-04 19:42:31,712 [Timer-4] INFO com.terracottatech.console - Server exiting… 2007-11-04 19:42:31,712 [Timer-4] INFO com.terracottatech.console - Server exiting…

Comments

Saravanan Subbiah 2007-11-09

fixed in 2.4, 2.5 and trunk