Disk space filling leads to hard server crash

Type: Bug
Status: Open
Priority: 2 Major
Resolution:
Component/s: DSO:L2
Labels:

Assignee: prodmgmt
Reporter: KirkWylie
Created: March 19, 2008
Votes: 0
Watchers: 1
Updated:
March 19, 2010
Resolved:

Description

If the disk you’re using for persistence fills up completely, the entire terracotta server will crash.

I would expect that it would handle this somewhat more gracefully.

Start of stack traces: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException: (JE 3.2.70) IOE during write at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1036) at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:219) at com.sleepycat.je.log.LogBufferPool.getWriteBuffer(LogBufferPool.java:166) at com.sleepycat.je.log.LogManager.logInternal(LogManager.java:434) at com.sleepycat.je.log.SyncedLogManager.logItem(SyncedLogManager.java:52) at com.sleepycat.je.log.LogManager.log(LogManager.java:257) at com.sleepycat.je.log.LogManager.log(LogManager.java:186) at com.sleepycat.je.tree.LN.log(LN.java:554) at com.sleepycat.je.tree.LN.optionalLog(LN.java:429) at com.sleepycat.je.tree.LN.delete(LN.java:240) at com.sleepycat.je.dbi.CursorImpl.delete(CursorImpl.java:822) at com.sleepycat.je.Cursor.deleteNoNotify(Cursor.java:709) at com.sleepycat.je.Database.deleteInternal(Database.java:504) at com.sleepycat.je.Database.delete(Database.java:415) at com.tc.objectserver.persistence.sleepycat.TransactionPersistorImpl.deleteAllGlobalTransactionDescriptors(TransactionPersistorImpl.java:95) at com.tc.objectserver.persistence.impl.TransactionStoreImpl.removeGlobalTransacionDescs(TransactionStoreImpl.java:145) at com.tc.objectserver.persistence.impl.TransactionStoreImpl.clearCommitedTransactionsBelowLowWaterMark(TransactionStoreImpl.java:127) at com.tc.objectserver.gtx.ServerGlobalTransactionManagerImpl.clearCommitedTransactionsBelowLowWaterMark(ServerGlobalTransactionManagerImpl.java:58) at com.tc.objectserver.handler.TransactionLowWaterMarkHandler.handleEvent(TransactionLowWaterMarkHandler.java:24) at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142) Caused by: java.io.IOException: Disc quota exceeded at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:456) at com.sleepycat.je.log.FileManager.writeToFile(FileManager.java:1142) at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1022) … 19 more com.tc.objectserver.persistence.sleepycat.TCDatabaseException: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException: (JE 3.2.70) IOE during write at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.basicSaveCollection(ManagedObjectPersistorImpl.java:318) at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.basicSaveObject(ManagedObjectPersistorImpl.java:296) at com.tc.objectserver.persistence.sleepycat.ManagedObjectPersistorImpl.saveAllObjects(ManagedObjectPersistorImpl.java:340) at com.tc.objectserver.impl.PersistentManagedObjectStore.commitAllObjects(PersistentManagedObjectStore.java:81) at com.tc.objectserver.impl.ObjectManagerImpl.flushAll(ObjectManagerImpl.java:633) at com.tc.objectserver.impl.ObjectManagerImpl.releaseAll(ObjectManagerImpl.java:493) at com.tc.objectserver.tx.ServerTransactionManagerImpl.release(ServerTransactionManagerImpl.java:315) at com.tc.objectserver.tx.ServerTransactionManagerImpl.commit(ServerTransactionManagerImpl.java:307)

…

Comments

Hung Huynh 2008-05-05

this also happened in 2.6-stable4 (as expected)

Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException: (JE 3.2.74) IOE during write at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1036) at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:219) at com.sleepycat.je.log.LogBufferPool.writeCompleted(LogBufferPool.java:275) at com.sleepycat.je.log.LogManager.logInternal(LogManager.java:505) at com.sleepycat.je.log.SyncedLogManager.logItem(SyncedLogManager.java:52) at com.sleepycat.je.log.LogManager.log(LogManager.java:257) at com.sleepycat.je.log.LogManager.logForceFlush(LogManager.java:129) at com.sleepycat.je.txn.Txn.commit(Txn.java:441) at com.sleepycat.je.txn.Txn.commit(Txn.java:365) at com.sleepycat.je.Transaction.commit(Transaction.java:80) at com.tc.objectserver.persistence.sleepycat.TransactionWrapper.commit(TransactionWrapper.java:28) at com.tc.objectserver.impl.ObjectManagerImpl.flushAllAndCommit(ObjectManagerImpl.java:712) at com.tc.objectserver.impl.ObjectManagerImpl.releaseAll(ObjectManagerImpl.java:550) at com.tc.objectserver.tx.ServerTransactionManagerImpl.commit(ServerTransactionManagerImpl.java:361) at com.tc.objectserver.handler.CommitTransactionChangeHandler.handleEvent(CommitTransactionChangeHandler.java:30) at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142) Caused by: java.io.IOException: No space left on device at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:466) at com.sleepycat.je.log.FileManager.writeToFile(FileManager.java:1142) at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1022) … 15 more Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException: (JE 3.2.74) IOE during write at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1036) at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:219) at com.sleepycat.je.log.LogBufferPool.writeCompleted(LogBufferPool.java:275) at com.sleepycat.je.log.LogManager.logInternal(LogManager.java:505) at com.sleepycat.je.log.SyncedLogManager.logItem(SyncedLogManager.java:52) at com.sleepycat.je.log.LogManager.log(LogManager.java:257) at com.sleepycat.je.log.LogManager.logForceFlush(LogManager.java:129) at com.sleepycat.je.txn.Txn.commit(Txn.java:441) at com.sleepycat.je.txn.Txn.commit(Txn.java:365) at com.sleepycat.je.Transaction.commit(Transaction.java:80) at com.tc.objectserver.persistence.sleepycat.TransactionWrapper.commit(TransactionWrapper.java:28) at com.tc.objectserver.impl.ObjectManagerImpl.flushAllAndCommit(ObjectManagerImpl.java:712) at com.tc.objectserver.impl.ObjectManagerImpl.releaseAll(ObjectManagerImpl.java:550) at com.tc.objectserver.tx.ServerTransactionManagerImpl.commit(ServerTransactionManagerImpl.java:361) at com.tc.objectserver.handler.CommitTransactionChangeHandler.handleEvent(CommitTransactionChangeHandler.java:30) at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142) Caused by: java.io.IOException: No space left on device at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:466) at com.sleepycat.je.log.FileManager.writeToFile(FileManager.java:1142) at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1022) … 15 more Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException: (JE 3.2.74) IOE during write at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1036) at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:219) at com.sleepycat.j