CDV ❯ Exception in thread "WorkerThread(batch_transaction_lookup_stage,0)" running pojo grid

• Type: Bug
• Status: Closed
• Priority: 2 Major
• Resolution: Fixed
• Component/s: DSO:L2
• Labels:

• Assignee: teck
• Reporter: grose
• Created: January 04, 2007
• Votes: 0
• Watchers: 0
• Updated:

  July 27, 2012
• Resolved:

  February 13, 2007

Attachments

• grose (215.00 k) application/x-gzip-compressed pojogrid.tar.gz
• grose (249.00 k) application/x-zip-compressed pojogrid.zip
• grose (321.0) text/xml tc-config.xml
• grose (1.00 k) text/xml tc-config-spider-shasta.xml
• grose (30.00 k) text/plain terracotta-server.log

Description

Following L2 error running pojo-grid developed by Jonas. Code and log file attached.

2007-01-04 15:24:07,312 INFO - Terracotta, version 2.2 as of 20061126-221107. 2007-01-04 15:24:08,390 INFO - Configuration loaded from the file at ‘C:\tc-dso\ terracotta-2.2\dso\samples\tc-config.xml’. 2007-01-04 15:24:08,984 INFO - Log file: ‘C:\Documents and Settings\Gordon\terra cotta\demo-server\server-logs\terracotta-server.log’. 2007-01-04 15:24:09,031 INFO - Terracotta license: Open Source - No License Need ed 2007-01-04 15:24:10,359 INFO - JMX Server started. Available at URL[service:jmx: jmxmp://localhost:9520] 2007-01-04 15:24:11,546 INFO - DSO Server started on port 9510. 2007-01-04 15:24:11,593 INFO - Terracotta Server has started up successfully, an d is now ready for work. Exception in thread “WorkerThread(batch_transaction_lookup_stage,0)” java.lang.A ssertionError: SequenceID Validator Failed : current = SequenceID=[13606] but th is = SequenceID=[13608] TransactionQueue = TransactionQueue[ChannelID=[10] ] = { State[ PROCESSING ] cu rrent Txn = ServerTransaction[SequenceID=[13608] , TransactionID=[18374],Channel ID=[10],NORMAL] = { changes = 2, notifies = 0, newRoots = 0} batchedContext = Ba tchedTransactionProcessingContext : { Txn count = 145 ObjectIDs = [ObjectID=[612 804], ObjectID=[612742], ObjectID=[612790], ObjectID=[612814], ObjectID=[612746] , ObjectID=[612727], ObjectID=[612846], ObjectID=[612844], ObjectID=[612802], Ob jectID=[612792], ObjectID=[612728], ObjectID=[612809], ObjectID=[612718], Object ID=[612773], ObjectID=[612791], ObjectID=[612842], ObjectID=[612797], ObjectID=[ 544352], ObjectID=[612722], ObjectID=[612803], ObjectID=[612786], ObjectID=[6127 52], ObjectID=[612835], ObjectID=[544346], ObjectID=[612744], ObjectID=[612785], ObjectID=[612841], ObjectID=[612777], ObjectID=[612741], ObjectID=[612721], Obj ectID=[612784], ObjectID=[612848], ObjectID=[612826], ObjectID=[612798], ObjectI D=[612775], ObjectID=[612743], ObjectID=[612711], ObjectID=[612736], ObjectID=[6 12837], ObjectID=[612751], ObjectID=[612815], ObjectID=[612745], ObjectID=[61273 8], ObjectID=[612735], ObjectID=[612748], ObjectID=[612825], ObjectID=[612750], ObjectID=[612774], ObjectID=[612771], ObjectID=[612739], ObjectID=[612731], Obje ctID=[612816], ObjectID=[612749], ObjectID=[612787], ObjectID=[612813], ObjectID =[612845], ObjectID=[612730], ObjectID=[612838], ObjectID=[612827], ObjectID=[61 2729], ObjectID=[612769], ObjectID=[612776], ObjectID=[612828], ObjectID=[612732 ], ObjectID=[612847], ObjectID=[612823], ObjectID=[612788], ObjectID=[612734], O bjectID=[612740], ObjectID=[612762], ObjectID=[612818], ObjectID=[544358], Objec tID=[612714], ObjectID=[612761], ObjectID=[612737], ObjectID=[612817], ObjectID= [612824], ObjectID=[612760], ObjectID=[612770], ObjectID=[544356], ObjectID=[612 780], ObjectID=[612819], ObjectID=[612757], ObjectID=[612716], ObjectID=[612822] , ObjectID=[612779], ObjectID=[612747], ObjectID=[612707], ObjectID=[612733], Ob jectID=[612772], ObjectID=[612758], ObjectID=[612821], ObjectID=[612723], Object ID=[612831], ObjectID=[612712], ObjectID=[612759], ObjectID=[612795], ObjectID=[ 612805], ObjectID=[612724], ObjectID=[612764], ObjectID=[612713], ObjectID=[6128 20], ObjectID=[612783], ObjectID=[544364], ObjectID=[612796], ObjectID=[612840], ObjectID=[612793], ObjectID=[612839], ObjectID=[612726], ObjectID=[612754], Obj ectID=[612807], ObjectID=[612720], ObjectID=[612812], ObjectID=[612806], ObjectI D=[612778], ObjectID=[612799], ObjectID=[612834], ObjectID=[612782], ObjectID=[6 12709], ObjectID=[612768], ObjectID=[612800], ObjectID=[612832], ObjectID=[61280 1], ObjectID=[612706], ObjectID=[612753], ObjectID=[612725], ObjectID=[612763], ObjectID=[612810], ObjectID=[612829], ObjectID=[612717], ObjectID=[612836], Obje ctID=[612794], ObjectID=[612766], ObjectID=[612830], ObjectID=[612715], ObjectID =[612755], ObjectID=[612811], ObjectID=[612781], ObjectID=[612843], ObjectID=[61 2765], ObjectID=[612789], ObjectID=[612708], ObjectID=[612756], ObjectID=[612705 ], ObjectID=[612719], ObjectID=[544370], ObjectID=[612833], ObjectID=[612808], O bjectID=[612710], ObjectID=[612767]] } } at com.tc.objectserver.tx.BatchedTransactionProcessorImpl$TransactionQue ue.processTransactions(BatchedTransactionProcessorImpl.java:167) at com.tc.objectserver.tx.BatchedTransactionProcessorImpl.processTransac tions(BatchedTransactionProcessorImpl.java:66) at com.tc.objectserver.handler.BatchTransactionLookupHandler.handleEvent (BatchTransactionLookupHandler.java:19) at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:141) java.lang.AssertionError: SequenceID Validator Failed : current = SequenceID=[13 606] but this = SequenceID=[13608] TransactionQueue = TransactionQueue[ChannelID=[10] ] = { State[ PROCESSING ] cu rrent Txn = ServerTransaction[SequenceID=[13608] , TransactionID=[18374],Channel ID=[10],NORMAL] = { changes = 2, notifies = 0, newRoots = 0} batchedContext = Ba tchedTransactionProcessingContext : { Txn count = 145 ObjectIDs = [ObjectID=[612 804], ObjectID=[612742], ObjectID=[612790], ObjectID=[612814], ObjectID=[612746] , ObjectID=[612727], ObjectID=[612846], ObjectID=[612844], ObjectID=[612802], Ob jectID=[612792], ObjectID=[612728], ObjectID=[612809], ObjectID=[612718], Object ID=[612773], ObjectID=[612791], ObjectID=[612842], ObjectID=[612797], ObjectID=[ 544352], ObjectID=[612722], ObjectID=[612803], ObjectID=[612786], ObjectID=[6127 52], ObjectID=[612835], ObjectID=[544346], ObjectID=[612744], ObjectID=[612785], ObjectID=[612841], ObjectID=[612777], ObjectID=[612741], ObjectID=[612721], Obj ectID=[612784], ObjectID=[612848], ObjectID=[612826], ObjectID=[612798], ObjectI D=[612775], ObjectID=[612743], ObjectID=[612711], ObjectID=[612736], ObjectID=[6 12837], ObjectID=[612751], ObjectID=[612815], ObjectID=[612745], ObjectID=[61273 8], ObjectID=[612735], ObjectID=[612748], ObjectID=[612825], ObjectID=[612750], ObjectID=[612774], ObjectID=[612771], ObjectID=[612739], ObjectID=[612731], Obje ctID=[612816], ObjectID=[612749], ObjectID=[612787], ObjectID=[612813], ObjectID =[612845], ObjectID=[612730], ObjectID=[612838], ObjectID=[612827], ObjectID=[61 2729], ObjectID=[612769], ObjectID=[612776], ObjectID=[612828], ObjectID=[612732 ], ObjectID=[612847], ObjectID=[612823], ObjectID=[612788], ObjectID=[612734], O bjectID=[612740], ObjectID=[612762], ObjectID=[612818], ObjectID=[544358], Objec tID=[612714], ObjectID=[612761], ObjectID=[612737], ObjectID=[612817], ObjectID= [612824], ObjectID=[612760], ObjectID=[612770], ObjectID=[544356], ObjectID=[612 780], ObjectID=[612819], ObjectID=[612757], ObjectID=[612716], ObjectID=[612822] , ObjectID=[612779], ObjectID=[612747], ObjectID=[612707], ObjectID=[612733], Ob jectID=[612772], ObjectID=[612758], ObjectID=[612821], ObjectID=[612723], Object ID=[612831], ObjectID=[612712], ObjectID=[612759], ObjectID=[612795], ObjectID=[ 612805], ObjectID=[612724], ObjectID=[612764], ObjectID=[612713], ObjectID=[6128 20], ObjectID=[612783], ObjectID=[544364], ObjectID=[612796], ObjectID=[612840], ObjectID=[612793], ObjectID=[612839], ObjectID=[612726], ObjectID=[612754], Obj ectID=[612807], ObjectID=[612720], ObjectID=[612812], ObjectID=[612806], ObjectI D=[612778], ObjectID=[612799], ObjectID=[612834], ObjectID=[612782], ObjectID=[6 12709], ObjectID=[612768], ObjectID=[612800], ObjectID=[612832], ObjectID=[61280 1], ObjectID=[612706], ObjectID=[612753], ObjectID=[612725], ObjectID=[612763], ObjectID=[612810], ObjectID=[612829], ObjectID=[612717], ObjectID=[612836], Obje ctID=[612794], ObjectID=[612766], ObjectID=[612830], ObjectID=[612715], ObjectID =[612755], ObjectID=[612811], ObjectID=[612781], ObjectID=[612843], ObjectID=[61 2765], ObjectID=[612789], ObjectID=[612708], ObjectID=[612756], ObjectID=[612705 ], ObjectID=[612719], ObjectID=[544370], ObjectID=[612833], ObjectID=[612808], O bjectID=[612710], ObjectID=[612767]] } } at com.tc.objectserver.tx.BatchedTransactionProcessorImpl$TransactionQue ue.processTransactions(BatchedTransactionProcessorImpl.java:167) at com.tc.objectserver.tx.BatchedTransactionProcessorImpl.processTransac tions(BatchedTransactionProcessorImpl.java:66) at com.tc.objectserver.handler.BatchTransactionLookupHandler.handleEvent (BatchTransactionLookupHandler.java:19) at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:141)

C:\tc-dso\terracotta-2.2\dso\samples>

Comments

Gordon Rose 2007-01-04

Used an URL of http://www.terracotta.org and a depth of 10 and two workers to run. Workers were started close to one another in time.

Gordon Rose 2007-01-04

This is the tc-config.xml for the clients.

Gordon Rose 2007-01-04

This is the tc-config.xml used by the server.

Saravanan Subbiah 2007-01-04

It seems like when this happens one of the clients throws an OOME. It is possible that if the OOME is thrown during a commit, then that transaction might not be sent to the server.

Also this code is entirely changed and this particular assertion is removed from 2.2.1 code base.

Gordon, can you please

1) check if there was an OOME in the client when this happened in the server ? 2) Increase the memory of L1 such that the OOME doesnt happen and check to see if the assertion still happens in the server 3) Try it with 2.2.1 to see of the problem happens ?

thanks, Saravanan

Steve Harris 2007-01-24

If it is the oome then the other bug I assigned to you should hopefully protect against it. Can you verify this after you do the protect on exception stuff I asked for :-)

Fiona OShea 2007-02-07

Updating Due date to last day of Moraga dev iteration

Tim Eck 2007-02-13

the only problem we found with this was OOMEs in the client. Although nothing we did will prevent an OOME, we did make it such that transactions out of sequence will not make it to the server and crash it anymore