2013-04-23 08:54:13,099 [main] INFO com.terracottatech.general - New logging session started. 2013-04-23 08:54:13,123 [main] INFO com.terracottatech.console - Terracotta Enterprise 4.0.0, as of 20130301-064135 (Revision 25020-22359 by cruise@rh5vmo135.terracotta.lan from 4.0.0) 2013-04-23 08:54:13,125 [main] INFO com.tc.logging.TCLogging - PID is 5868 2013-04-23 08:54:13,169 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - Attempting to load configuration from the file at '/etc/bigmemory/tc-config.xml'... 2013-04-23 08:54:13,682 [main] INFO com.terracottatech.console - Successfully loaded base configuration from file at '/etc/bigmemory/tc-config.xml'. 2013-04-23 08:54:13,694 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - The configuration specified by 'base configuration from file at '/etc/bigmemory/tc-config.xml'': /var/bigmemory/tc1/logs /opt/extraspace/bigmemory/tc1/data /opt/extraspace/bigmemory/tc1/data-backup /opt/extraspace/bigmemory/tc1/index true 6g 2013-04-23 08:54:13,695 [main] INFO com.tc.properties.TCProperties - Loading default properties from tc.properties 2013-04-23 08:54:13,695 [main] WARN com.tc.properties.TCProperties - The property "l2.transactionmanager.logging.enabled" has been removed/renamed in the latest release. Please update the tc.properties file or some of your settings might not work 2013-04-23 08:54:13,695 [main] WARN com.tc.properties.TCProperties - The property "l2.transactionmanager.logging.verbose" has been removed/renamed in the latest release. Please update the tc.properties file or some of your settings might not work 2013-04-23 08:54:13,695 [main] WARN com.tc.properties.TCProperties - The property "l2.transactionmanager.logging.printStats" has been removed/renamed in the latest release. Please update the tc.properties file or some of your settings might not work 2013-04-23 08:54:13,695 [main] WARN com.tc.properties.TCProperties - The property "l2.transactionmanager.logging.printCommits" has been removed/renamed in the latest release. Please update the tc.properties file or some of your settings might not work 2013-04-23 08:54:13,695 [main] INFO com.tc.properties.TCProperties - The property "l2.dump.on.exception.timeout" was read before initialization completed. "l2.dump.on.exception.timeout" = 30 2013-04-23 08:54:13,695 [main] INFO com.tc.properties.TCProperties - The property "tc.config.total.timeout" was read before initialization completed. "tc.config.total.timeout" = 300000 2013-04-23 08:54:13,695 [main] INFO com.tc.properties.TCProperties - The property "tc.config.getFromSource.timeout" was read before initialization completed. "tc.config.getFromSource.timeout" = 30000 2013-04-23 08:54:13,695 [main] INFO com.tc.properties.TCProperties - tc-config doesn't have any tc-property. No tc-property will be overridden 2013-04-23 08:54:13,698 [main] INFO com.tc.properties.TCProperties - Loaded TCProperties : TCProperties = { appgroups.debug = false, dev.console.max.operator.events = 5000, ehcache.clusterAllCacheManagers = true, ehcache.concurrency = 128, ehcache.evictor.logging.enabled = false, ehcache.evictor.pool.size = 1, ehcache.global.eviction.enable = true, ehcache.global.eviction.frequency = 10, ehcache.global.eviction.rest.timeMillis = 10, ehcache.global.eviction.segments = 2, ehcache.lock.readLevel = READ, ehcache.lock.writeLevel = WRITE, ehcache.logging.enabled = false, ehcache.storageStrategy.dcv2.evictUnexpiredEntries.enabled = true, ehcache.storageStrategy.dcv2.eviction.overshoot = 15, ehcache.storageStrategy.dcv2.localcache.enabled = true, ehcache.storageStrategy.dcv2.perElementTTITTL.enabled = false, ehcache.storageStrategy.dcv2.periodicEviction.enabled = true, http.defaultservlet.attribute.aliases = false;, http.defaultservlet.attribute.dirallowed = false;, http.defaultservlet.enabled = false;, l1.cachemanager.criticalObjectThreshold = -1, l1.cachemanager.criticalThreshold = 70, l1.cachemanager.enabled = true, l1.cachemanager.leastCount = 2, l1.cachemanager.logging.enabled = false, l1.cachemanager.monitorOldGenOnly = true, l1.cachemanager.percentageToEvict = 10, l1.cachemanager.sleepInterval = 3000, l1.cachemanager.threshold = 50, l1.clusterevents.outofbandnotifier.jointime.millis = 100, l1.connect.versionMatchCheck.enabled = true, l1.healthcheck.l2.bindAddress = 0.0.0.0, l1.healthcheck.l2.bindPort = 0, l1.healthcheck.l2.ping.enabled = true, l1.healthcheck.l2.ping.idletime = 5000, l1.healthcheck.l2.ping.interval = 1000, l1.healthcheck.l2.ping.probes = 3, l1.healthcheck.l2.socketConnect = true, l1.healthcheck.l2.socketConnectCount = 13, l1.healthcheck.l2.socketConnectTimeout = 5, l1.jvm.check.compatibility = true, l1.l2.config.validation.enabled = true, l1.lock.statistics.gatherInterval = 1, l1.lock.statistics.traceDepth = 0, l1.lockmanager.pinning.enabled = true, l1.lockmanager.striped.count = 128, l1.lockmanager.timeout.interval = 60000, l1.max.connect.retries = -1, l1.objectmanager.fault.count = 500, l1.objectmanager.fault.logging.enabled = false, l1.objectmanager.flush.logging.enabled = false, l1.objectmanager.objectid.request.size = 50000, l1.objectmanager.remote.batchLookupTimePeriod = 1, l1.objectmanager.remote.logging.enabled = false, l1.objectmanager.remote.maxDNALRUSize = 60, l1.objectmanager.remote.maxRequestSentImmediately = 4, l1.seda.stage.sink.capacity = -1, l1.serverarray.objectCreationStrategy = round-robin, l1.serverarray.objectCreationStrategy.groupAffinity.groupName = mirror-group1, l1.serverarray.objectCreationStrategy.roundRobin.startIndex = sequential, l1.servermapmanager.remote.batchLookupTimePeriod = 1, l1.servermapmanager.remote.maxRequestSentImmediately = 4, l1.shutdown.force.finalization = true, l1.shutdown.threadgroup.gracetime = 30000, l1.socket.connect.timeout = 10000, l1.socket.reconnect.waitInterval = 1000, l1.transactionmanager.completedAckFlushTimeout = 5000, l1.transactionmanager.folding.debug = false, l1.transactionmanager.folding.enabled = true, l1.transactionmanager.folding.lock.limit = 0, l1.transactionmanager.folding.object.limit = 0, l1.transactionmanager.logging.enabled = false, l1.transactionmanager.maxBatchSizeInKiloBytes = 128, l1.transactionmanager.maxOutstandingBatchSize = 4, l1.transactionmanager.maxPendingBatches = 88, l1.transactionmanager.maxSleepTimeBeforeHalt = 1024, l1.transactionmanager.strings.compress.enabled = true, l1.transactionmanager.strings.compress.logging.enabled = false, l1.transactionmanager.strings.compress.minSize = 512, l1.transactionmanager.timeoutForAckOnExit = 300, l2.beanshell.enabled = false, l2.beanshell.port = 9929, l2.dump.on.exception.timeout = 30, l2.enable.legacy.production.mode = false, l2.frs.compactor.lsnGap.maxLoad = 0.80, l2.frs.compactor.lsnGap.minLoad = 0.40, l2.healthcheck.l1.ping.enabled = true, l2.healthcheck.l1.ping.idletime = 5000, l2.healthcheck.l1.ping.interval = 1000, l2.healthcheck.l1.ping.probes = 3, l2.healthcheck.l1.socketConnect = true, l2.healthcheck.l1.socketConnectCount = 10, l2.healthcheck.l1.socketConnectTimeout = 5, l2.healthcheck.l2.ping.enabled = true, l2.healthcheck.l2.ping.idletime = 5000, l2.healthcheck.l2.ping.interval = 1000, l2.healthcheck.l2.ping.probes = 3, l2.healthcheck.l2.socketConnect = true, l2.healthcheck.l2.socketConnectCount = 10, l2.healthcheck.l2.socketConnectTimeout = 5, l2.l1reconnect.enabled = true, l2.l1reconnect.maxDelayedAcks = 16, l2.l1reconnect.sendWindow = 32, l2.l1reconnect.sendqueue.cap = 5000, l2.l1reconnect.timeout.millis = 5000, l2.lockmanager.greedy.lease.enabled = true, l2.lockmanager.greedy.lease.leaseTimeInMillis = 50, l2.lockmanager.greedy.locks.enabled = true, l2.logs.store = 1500, l2.nha.autoRestart = true, l2.nha.dirtydb.autoDelete = true, l2.nha.dirtydb.backup.enabled = true, l2.nha.dirtydb.rolling = 0, l2.nha.send.timeout.millis = 16000, l2.nha.tcgroupcomm.discovery.interval = 1000, l2.nha.tcgroupcomm.handshake.timeout = 5000, l2.nha.tcgroupcomm.reconnect.enabled = true, l2.nha.tcgroupcomm.reconnect.maxDelayedAcks = 16, l2.nha.tcgroupcomm.reconnect.sendWindow = 32, l2.nha.tcgroupcomm.reconnect.sendqueue.cap = 5000, l2.nha.tcgroupcomm.reconnect.timeout = 5000, l2.objectmanager.dgc.inline.cleanup.delaySeconds = 0, l2.objectmanager.dgc.inline.intervalInSeconds = 10, l2.objectmanager.dgc.inline.maxObjects = 10000, l2.objectmanager.dgc.throttle.requestsPerThrottle = 1000, l2.objectmanager.dgc.throttle.timeInMillis = 0, l2.objectmanager.maxTxnsInTxnObjectGrouping = 10, l2.objectmanager.objectrequest.logging.enabled = false, l2.objectmanager.objectrequest.split.size = 500, l2.objectmanager.passive.sync.batch.size = 500, l2.objectmanager.passive.sync.message.maxSizeInMegaBytes = 10, l2.objectmanager.passive.sync.throttle.maxPendingMessages = 10, l2.objectmanager.passive.sync.throttle.timeInMillis = 0, l2.objectmanager.persistor.logging.enabled = false, l2.objectmanager.request.logging.enabled = false, l2.offHeap.allocation.critical = 15000, l2.offHeap.allocation.critical.halt = true, l2.offHeap.allocation.slow = 1500, l2.offHeap.map.concurrency = 1, l2.offHeap.map.tableSize = 128, l2.operator.events.store = 1500, l2.remotejmx.idletime = 5, l2.remotejmx.maxthreads = 50, l2.seda.apply.stage.threads = 8, l2.seda.evictionprocessorstage.sink.capacity = 1000, l2.seda.local.cache.invalidations.sink.capacity = 5000, l2.seda.local.cache.transaction.complete.sink.capacity = 5000, l2.seda.local.cache.transaction.complete.threads = 8, l2.seda.query.threads = 4, l2.seda.search.threads = 16, l2.seda.stage.sink.capacity = -1, l2.serverarray.2pc.enabled = true, l2.servermap.eviction.broadcast.maxkeys = 10000, l2.servermap.eviction.clientObjectReferences.refresh.interval = 60000, l2.startuplock.retries.enabled = false, l2.transactionmanager.logging.enabled = false, l2.transactionmanager.logging.printBroadcastStats = false, l2.transactionmanager.logging.printCommits = false, l2.transactionmanager.logging.printStats = true, l2.transactionmanager.logging.verbose = false, l2.transactionmanager.passive.throttle.enabled = true, l2.transactionmanager.passive.throttle.maxSleepSeconds = 5, l2.transactionmanager.passive.throttle.threshold = 20000, lock.statistics.enabled = false, logging.longgc.threshold = 8000, logging.maxBackups = 20, logging.maxLogFileSize = 512, management.rest.enabled = true, memory.monitor.forcebasic = false, net.core.keepalive = false, net.core.tcpnodelay = true, offHeapFileSystem.file.maxDataPageSize = 262144, offHeapFilesystem.chm.segments = 4, offHeapFilesystem.file.blockSize = 8192, productkey.path = /opt/bigmemory-max-4.0.0/server/bin/../../terracotta-license.key, search.lucene.indexes.per.cache = 4, search.lucene.max.boolean.clauses = 1024, search.lucene.max.buffer = 16.0, search.lucene.use.offHeap.directory = false, search.lucene.use.ram.directory = false, search.passive.max.chunk = 2097152, search.passive.max.pending = 8, session.debug.hops = false, session.debug.hops.interval = 100, session.debug.invalidate = false, session.debug.sessions = false, session.invalidator.bench.enabled = true, session.invalidator.sleep = 300, session.request.bench.enabled = false, session.request.tracking = false, session.request.tracking.dump = false, session.request.tracking.interval = 2500, session.request.tracking.threshold = 15000, session.statistics.enabled = false, session.verify.set.attribute = false, session.vhosts.excluded = , stats.printer.intervalInMillis = 5000, tc.bytebuffer.common.pool.maxcount = 3000, tc.bytebuffer.pooling.enabled = true, tc.bytebuffer.threadlocal.pool.maxcount = 2000, tc.config.getFromSource.timeout = 30000, tc.config.total.timeout = 300000, tc.management.mbeans.enabled = true, tc.management.test.mbeans.enabled = false, tc.messages.grouping.enabled = true, tc.messages.grouping.maxSizeKiloBytes = 1024, tc.messages.packup.enabled = true, tc.stage.monitor.delay = 5000, tc.stage.monitor.enabled = false, tc.time.sync.threshold = 30, tc.transport.handshake.timeout = 10000, tcm.monitor.delay = 5, tcm.monitor.enabled = false } 2013-04-23 08:54:13,731 [main] INFO com.terracottatech.console - Log file: '/var/bigmemory/tc1/logs/terracotta-server.log'. 2013-04-23 08:54:13,732 [main] INFO com.tc.logging.TCLogging - All Java System Properties for this Terracotta instance: ======================================================================== JVM arguments: [-XX:MaxDirectMemorySize=9223372036854775807, -Xms4g, -Xmx4g, -XX:+HeapDumpOnOutOfMemoryError, -Dcom.sun.management.jmxremote, -Dtc.install-root=/opt/bigmemory-max-4.0.0/server/bin/.., -Dsun.rmi.dgc.server.gcInterval=31536000000, -Dcom.tc.productkey.path=/opt/bigmemory-max-4.0.0/server/bin/../../terracotta-license.key, -Dnet.sf.ehcache.offheap.categoryTree.config.idealMaxSegmentSize=30M] awt.toolkit : sun.awt.X11.XToolkit com.sun.management.jmxremote : com.tc.productkey.path : /opt/bigmemory-max-4.0.0/server/bin/../../terracotta-license.key file.encoding : UTF-8 file.encoding.pkg : sun.io file.separator : / java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment java.awt.printerjob : sun.print.PSPrinterJob java.class.path : /opt/bigmemory-max-4.0.0/server/bin/../lib/tc.jar java.class.version : 51.0 java.endorsed.dirs : /usr/java/jdk1.7.0_21/jre/lib/endorsed java.ext.dirs : /usr/java/jdk1.7.0_21/jre/lib/ext:/usr/java/packages/lib/ext java.home : /usr/java/jdk1.7.0_21/jre java.io.tmpdir : /tmp java.library.path : /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.rmi.server.randomIDs : true java.runtime.name : Java(TM) SE Runtime Environment java.runtime.version : 1.7.0_21-b11 java.specification.name : Java Platform API Specification java.specification.vendor : Oracle Corporation java.specification.version : 1.7 java.vendor : Oracle Corporation java.vendor.url : http://java.oracle.com/ java.vendor.url.bug : http://bugreport.sun.com/bugreport/ java.version : 1.7.0_21 java.vm.info : mixed mode java.vm.name : Java HotSpot(TM) 64-Bit Server VM java.vm.specification.name : Java Virtual Machine Specification java.vm.specification.vendor : Oracle Corporation java.vm.specification.version : 1.7 java.vm.vendor : Oracle Corporation java.vm.version : 23.21-b01 line.separator : net.sf.ehcache.offheap.categoryTree.config.idealMaxSegmentSize: 30M os.arch : amd64 os.name : Linux os.version : 2.6.32-220.7.1.el6.x86_64 path.separator : : sun.arch.data.model : 64 sun.boot.class.path : /usr/java/jdk1.7.0_21/jre/lib/resources.jar:/usr/java/jdk1.7.0_21/jre/lib/rt.jar:/usr/java/jdk1.7.0_21/jre/lib/sunrsasign.jar:/usr/java/jdk1.7.0_21/jre/lib/jsse.jar:/usr/java/jdk1.7.0_21/jre/lib/jce.jar:/usr/java/jdk1.7.0_21/jre/lib/charsets.jar:/usr/java/jdk1.7.0_21/jre/lib/jfr.jar:/usr/java/jdk1.7.0_21/jre/classes sun.boot.library.path : /usr/java/jdk1.7.0_21/jre/lib/amd64 sun.cpu.endian : little sun.cpu.isalist : sun.io.unicode.encoding : UnicodeLittle sun.java.command : com.tc.server.TCServerMain -f /etc/bigmemory/tc-config.xml -name tc1 sun.java.launcher : SUN_STANDARD sun.jnu.encoding : UTF-8 sun.management.compiler : HotSpot 64-Bit Tiered Compilers sun.os.patch.level : unknown sun.rmi.dgc.server.gcInterval : 31536000000 tc.install-root : /opt/bigmemory-max-4.0.0/server/bin/.. user.country : US user.dir : /home/terracotta user.home : /home/terracotta user.language : en user.name : terracotta user.timezone : America/Los_Angeles ======================================================================== 2013-04-23 08:54:13,781 [main] INFO com.terracottatech.console - Terracotta license loaded from /opt/bigmemory-max-4.0.0/server/bin/../../terracotta-license.key Capabilities: ehcache, search, TMC, server array offheap Date of Issue: 2013-04-17 Edition: Free Email: greg.adams@build.com License Number: 1004106 License Type: Free Production Licensee: Greg Adams, Build.com Max Client Count: 4 Product: BigMemory Max terracotta.serverArray.maxOffHeap: 8G 2013-04-23 08:54:13,821 [main] INFO com.terracottatech.console - Available Max Runtime Memory: 3925MB 2013-04-23 08:54:14,035 [main] INFO com.terracottatech.dso - Standard DSO Server created 2013-04-23 08:54:14,050 [main] INFO com.tc.objectserver.storage.cache.offheap.OffHeapCacheConfig - OffHeapCacheConfig [ Enabled:true; MaxDataSize: 6g; MapsDBConcurrency: 1; mapsTableSize: 128; mapMaxPageSize, mapMinPageSize: 512k,4k; ObjInitialDataSize: 1m; ObjTableSize: 1048576; ObjConcurrency: 96 ], Vm.MaxDirectMemorySize [ 8589934592g ] 2013-04-23 08:54:14,053 [main] INFO com.terracottatech.dso - Creating server nodeID: NodeID[devbox5.build.internal:9510] 2013-04-23 08:54:14,074 [main] INFO com.tc.objectserver.managedobject.ConcurrentDistributedServerMapManagedObjectState - Eviction overshoot threshold is 15.0 2013-04-23 08:54:14,137 [main] INFO com.terracottatech.console - JMX Server started. Available at URL[service:jmx:jmxmp://0.0.0.0:9520] 2013-04-23 08:54:14,722 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Starting recovery for /opt/extraspace/bigmemory/tc1/data/objectdb/config 2013-04-23 08:54:14,724 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 0% 2013-04-23 08:54:14,859 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 10% 2013-04-23 08:54:14,995 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 20% 2013-04-23 08:54:15,089 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 30% 2013-04-23 08:54:15,198 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 40% 2013-04-23 08:54:15,301 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 50% 2013-04-23 08:54:15,417 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 60% 2013-04-23 08:54:15,509 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 70% 2013-04-23 08:54:15,587 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 80% 2013-04-23 08:54:15,657 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 90% 2013-04-23 08:54:15,736 [main] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 100% 2013-04-23 08:54:15,810 [Thread-35] INFO com.terracottatech.offheapstore.paging.UpfrontAllocatingPageSource - Allocating 6.0GB in chunks 2013-04-23 08:54:19,290 [Thread-35] INFO com.terracottatech.offheapstore.paging.UpfrontAllocatingPageSource - Allocation 40% complete 2013-04-23 08:54:21,048 [Thread-35] INFO com.terracottatech.offheapstore.paging.UpfrontAllocatingPageSource - Allocation 100% complete 2013-04-23 08:54:21,049 [Thread-35] INFO com.terracottatech.offheapstore.paging.UpfrontAllocatingPageSource - Allocated 6.0GB in 1GB chunks. 2013-04-23 08:54:21,049 [Thread-35] INFO com.terracottatech.offheapstore.paging.UpfrontAllocatingPageSource - Took 5239 ms to create CacheManager off-heap storage of 6.0GB. 2013-04-23 08:54:32,820 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Starting recovery for /opt/extraspace/bigmemory/tc1/data/objectdb/data 2013-04-23 08:54:32,821 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 0% 2013-04-23 08:54:32,888 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 10% 2013-04-23 08:54:32,923 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 20% 2013-04-23 08:54:32,970 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 30% 2013-04-23 08:54:33,017 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 40% 2013-04-23 08:54:33,056 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 50% 2013-04-23 08:54:33,115 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 60% 2013-04-23 08:54:33,158 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 70% 2013-04-23 08:54:33,197 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 80% 2013-04-23 08:54:33,208 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 90% 2013-04-23 08:54:33,306 [Thread-35] INFO com.terracottatech.frs.recovery.RecoveryManager - Recovery progress 100% 2013-04-23 08:54:34,442 [main] INFO com.tc.net.core.TCWorkerCommManager - Creating 4 worker comm threads for L2_L1 2013-04-23 08:54:34,471 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Server - HealthChecker Started 2013-04-23 08:54:34,471 [main] INFO com.tc.net.protocol.tcm.CommunicationsManager - HealthCheck CallbackPort Listener not requested 2013-04-23 08:54:34,482 [main] INFO com.tc.objectserver.l1.impl.ClientObjectReferenceSet - ServerMapEviction Client Object References refresh interval 60 seconds. 2013-04-23 08:54:34,506 [main] INFO com.tc.runtime.TCMemoryManagerImpl - GarbageCollector: PS Scavenge 2013-04-23 08:54:34,506 [main] INFO com.tc.runtime.TCMemoryManagerImpl - GarbageCollector: PS MarkSweep 2013-04-23 08:54:34,747 [main] INFO com.tc.objectserver.impl.ProgressiveEvictionManager - Using threshold EvictionThreshold{name=8G, max=8589934592, used=773849088, reserved=134217728} for total size 6442450944 2013-04-23 08:54:34,748 [main] INFO com.tc.objectserver.impl.ProgressiveEvictionManager - used:773849088,reserved:134217728 2013-04-23 08:54:34,769 [main] WARN com.terracottatech.dso - High Availability Not Configured Properly: L1L2HealthCheck should be less than L2-L2HealthCheck + ElectionTime + ClientReconnectWindow 2013-04-23 08:54:34,784 [main] INFO com.tc.net.core.TCWorkerCommManager - Creating 4 worker comm threads for L2_L2 2013-04-23 08:54:34,797 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: TCGroupManager - HealthChecker Started 2013-04-23 08:54:34,797 [main] INFO com.tc.net.protocol.tcm.CommunicationsManager - HealthCheck CallbackPort Listener not requested 2013-04-23 08:54:34,820 [main] INFO com.tc.objectserver.search.JournalIndexManager - Removing any files present in /opt/extraspace/bigmemory/tc1/index/journal 2013-04-23 08:54:34,840 [main] INFO com.terracottatech.search.LuceneIndexManager - Lucene version: 4.0.0.2 2013-04-23 08:54:35,755 [main] INFO com.tc.l2.L2DebugLogging - L2 debug logging: DISABLED 2013-04-23 08:54:35,799 [main] INFO com.terracottatech.dso - This L2 Node ID = NodeID[devbox5.build.internal:9510] 2013-04-23 08:54:35,801 [main] INFO com.tc.l2.state.ElectionManagerImpl - Election Started : Enrollment [ NodeID[devbox5.build.internal:9510], isNew = true, weights = -3911155255003808781,9152171096548158796 ] 2013-04-23 08:54:40,802 [main] INFO com.tc.l2.state.ElectionManagerImpl - Election Complete : [Enrollment [ NodeID[devbox5.build.internal:9510], isNew = true, weights = -3911155255003808781,9152171096548158796 ]] : State[ Election-Complete ] 2013-04-23 08:54:40,804 [main] INFO com.tc.l2.state.StateManagerImpl - Becoming State[ ACTIVE-COORDINATOR ] 2013-04-23 08:54:40,804 [main] INFO com.terracottatech.console - Becoming State[ ACTIVE-COORDINATOR ] 2013-04-23 08:54:40,811 [main] INFO tc.operator.event - NODE : tc1 Subsystem: CLUSTER_TOPOLOGY Message: Moved to ACTIVE-COORDINATOR 2013-04-23 08:54:40,811 [main] INFO com.tc.l2.state.ElectionManagerImpl - Declared as Winner: Winner is : Enrollment [ NodeID[devbox5.build.internal:9510], isNew = true, weights = -3911155255003808781,9152171096548158796 ] 2013-04-23 08:54:40,811 [main] WARN tc.operator.event - NODE : tc1 Subsystem: SYSTEM_SETUP Message: tc1 started as an ACTIVE with existing data 2013-04-23 08:54:40,813 [WorkerThread(l2_state_change_stage, 0)] INFO com.terracottatech.search.LuceneIndexManager - Initializing lucene index directory at /opt/extraspace/bigmemory/tc1/index/lucene offheap : false ram : false index/cache ratio : 4 2013-04-23 08:54:40,842 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ServerTransactionManager - Waiting for txns to complete 2013-04-23 08:54:40,844 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ServerTransactionManager - No more txns in the system. 2013-04-23 08:54:40,844 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ResentTransactionSequencer - Making callback com.tc.objectserver.gtx.GlobalTransactionIDLowWaterMarkProvider$2@7ad138dd pending since in State[ ADD_RESENT ] resent txns size : 0 2013-04-23 08:54:40,846 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Starting DSO services... 2013-04-23 08:54:40,847 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.TransactionStoreImpl - shutdownAllClientsExcept() : Removing txns from DB : 0 2013-04-23 08:54:40,847 [main] INFO com.tc.server.TCServer - deploying management REST services from archive /opt/bigmemory-max-4.0.0/server/bin/../lib/management-tsa-war-4.0.0.war 2013-04-23 08:54:40,850 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ObjectRequestManagerRestartImpl - Processing Pending Lookups = 0 2013-04-23 08:54:40,850 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ResentTransactionSequencer - Unregistering ResentTransactionSequencer since no more resent Transactions : 0 2013-04-23 08:54:40,851 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.tx.ResentTransactionSequencer - Adding Pending resent CallBacks to TxnMgr : com.tc.objectserver.gtx.GlobalTransactionIDLowWaterMarkProvider$2@7ad138dd 2013-04-23 08:54:40,851 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.gtx.GlobalTransactionIDLowWaterMarkProvider - Switching GlobalTransactionID Low Water mark provider since all resent transactions are applied 2013-04-23 08:54:40,851 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ServerMapEvictionEngine - ehcache.evictor.logging.enabled : false 2013-04-23 08:54:40,851 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.impl.ServerMapEvictionEngine - ehcache.storageStrategy.dcv2.periodicEviction.enabled : true 2013-04-23 08:54:40,853 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.l1.impl.InvalidateObjectManagerImpl - Restart txn processing complete : Adding validation of Objects for 0 Clients 2013-04-23 08:54:40,854 [WorkerThread(l2_state_change_stage, 0)] INFO com.terracottatech.console - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work. 2013-04-23 08:54:40,885 [WorkerThread(garbage_collect_stage, 0)] INFO tc.operator.event - NODE : tc1 Subsystem: DGC Message: Inline DGC [ 2 ] reference cleanup started. 2013-04-23 08:54:41,060 [Resource Monitor] INFO com.tc.objectserver.impl.ProgressiveEvictionManager - Percent usage:0 reserved:2 poll time:6113 msec. 2013-04-23 08:54:43,820 [L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)] INFO com.tc.net.core.TCConnection - Comms Message Batching enabled 2013-04-23 08:54:44,387 [WorkerThread(garbage_collect_stage, 0)] INFO com.tc.objectserver.dgc.impl.MarkAndSweepGarbageCollector - DGC[ 2 ] complete : GarbageCollectionInfo [ Iteration = 2 ] = type = INLINE_CLEANUP startTime = 1366732480868 begin object count = 8263 end object count = 8263 markStageTime = 3519 pauseStageTime = 0 elapsedTime = 3519 totalMarkCycleTime = 3519 candidate garbage count = 0 actual garbage count = 0 pre rescue count = 0 rescue1Time = 0 rescue 1 Count = 0 rescue2Time = 0 2013-04-23 08:54:44,388 [WorkerThread(garbage_collect_stage, 0)] INFO tc.operator.event - NODE : tc1 Subsystem: DGC Message: Inline DGC [ 2 ] reference cleanup finished. Begin Count : 8,263 Collected : 0 Time Taken : 3,519 ms Live Objects : 8,263 2013-04-23 08:54:45,718 [main] INFO com.tc.server.TCServer - Server started as tc1 2013-04-23 08:55:44,184 [Resource Monitor] INFO com.tc.objectserver.impl.ProgressiveEvictionManager - Percent usage:0 reserved:2 poll time:5730 msec. 2013-04-23 08:55:56,472 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(1.9c0b205fb1134e43879aae01ac652fb2.25652cff-f836-4906-bd63-938b2cefd5a3-13e379ac845)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@1292030534: connected: true, closed: false local=172.25.1.105:9510 remote=172.25.1.104:52087 connect=[Tue Apr 23 08:55:56 PDT 2013] idle=20ms [0 read, 0 write] 2013-04-23 08:55:56,490 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - Allocated connection license for jvm 25652cff-f836-4906-bd63-938b2cefd5a3-13e379ac845; ConnectionPolicy[maxConnections=4, connectedJvmCount=1] 2013-04-23 08:55:56,490 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - New connection [1] from jvm 25652cff-f836-4906-bd63-938b2cefd5a3-13e379ac845 2013-04-23 08:55:56,490 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerStackProvider - User null successfully authenticated 2013-04-23 08:55:56,490 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.TCWorkerCommManager - Selecting [L2_L1:TCWorkerComm # 0, FD, wt:0] from [[L2_L1:TCWorkerComm # 0, FD, wt:0], [L2_L1:TCWorkerComm # 1, FD, wt:0], [L2_L1:TCWorkerComm # 2, FD, wt:0], [L2_L1:TCWorkerComm # 3, FD, wt:0]] 2013-04-23 08:55:56,496 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.CoreNIOServices - selection key cancelled key@1950030121 2013-04-23 08:55:56,519 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - Health monitoring agent started for devbox4.build.internal:52087 2013-04-23 08:55:56,526 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.CoreNIOServices - IOException attempting to finish socket connection java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692) at com.tc.net.core.CoreNIOServices$CommThread.doConnect(CoreNIOServices.java:684) at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:624) at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:295) 2013-04-23 08:55:56,531 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.TCConnectionManager - error event on connection com.tc.net.core.TCConnectionImpl@1302272683: connected: false, closed: false connect=[no connect time] idle=10ms [0 read, 0 write]: No route to host 2013-04-23 08:55:56,534 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - Config Factor updated from 1 to 3 2013-04-23 08:55:56,535 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - HealthCheckCallbackPort verification FAILED for devbox4.build.internal:52087(callbackport: 36086) 2013-04-23 08:55:56,657 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Client connected ClientID[1] 2013-04-23 08:55:56,663 [WorkerThread(client_handshake_stage, 0)] INFO tc.operator.event - NODE : tc1 Subsystem: CLUSTER_TOPOLOGY Message: Node ClientID[1] joined the cluster 2013-04-23 08:55:56,701 [WorkerThread(jmxremote_tunnel_stage, 0)] INFO com.tc.management.remote.protocol.terracotta.ClientTunnelingEventHandler - L1[ChannelID=[1]] notified us that their JMX server is now available 2013-04-23 08:56:34,133 [L2_L2:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9530)] WARN com.tc.net.core.CoreNIOServices - Exception trying to process interest request: java.nio.channels.ClosedChannelException 2013-04-23 08:56:47,280 [Resource Monitor] INFO com.tc.objectserver.impl.ProgressiveEvictionManager - Percent usage:0 reserved:2 poll time:5731 msec. 2013-04-23 08:56:47,920 [L2_L1:TCWorkerComm # 0_R] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(1.9c0b205fb1134e43879aae01ac652fb2.25652cff-f836-4906-bd63-938b2cefd5a3-13e379ac845)[]: CLOSE EVENT : com.tc.net.core.TCConnectionImpl@1292030534: connected: false, closed: true local=172.25.1.105:9510 remote=172.25.1.104:52087 connect=[Tue Apr 23 08:55:56 PDT 2013] idle=3600ms [1059062 read, 1801891 write]. STATUS : ESTABLISHED 2013-04-23 08:56:47,921 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - Removed connection [1] from jvm 25652cff-f836-4906-bd63-938b2cefd5a3-13e379ac845 2013-04-23 08:56:47,921 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - De-allocated connection license for jvm 25652cff-f836-4906-bd63-938b2cefd5a3-13e379ac845; ConnectionPolicy[maxConnections=4, connectedJvmCount=0] 2013-04-23 08:56:47,921 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Server - Connection to [devbox4.build.internal:52087] DISCONNECTED. Health Monitoring for this node is now disabled. 2013-04-23 08:56:52,935 [OOO Connection Restore Timer] WARN tc.operator.event - NODE : tc1 Subsystem: CLUSTER_TOPOLOGY Message: Node ClientID[1] left the cluster 2013-04-23 08:56:52,935 [WorkerThread(channel_life_cycle_stage, 0)] INFO com.tc.objectserver.handler.ChannelLifeCycleHandler - : Received transport disconnect. Shutting down client ClientID[1] 2013-04-23 08:56:52,939 [WorkerThread(channel_life_cycle_stage, 0)] INFO com.tc.objectserver.impl.TransactionStoreImpl - shutdownClient() : Removing txns from DB : 0 2013-04-23 08:56:59,381 [Job_Executor6] INFO com.tc.management.L2Management - JMX shutdown request connectionId:jmxmp://localhost:53461 78759400 execute methodId: 14 params: org.terracotta.internal:type=Terracotta Server,name=Terracotta Server shutdown [B@4efe2553 [Ljava.lang.String;@f4494a9 2013-04-23 08:56:59,383 [RemoteJMXProcessor-13] WARN com.tc.management.beans.TCServerInfo - shutdown is invoked by MBean 2013-04-23 08:57:00,385 [TCServerInfo shutdown timer] INFO com.terracottatech.console - Server exiting... 2013-04-23 08:57:00,392 [CommonShutDownHook] INFO com.terracottatech.dso - L2 Exiting...