Orphaned Triggers/Jobs Stop firing when clustered with Terracotta

Type: Bug
Status: Closed
Priority: 2 Major
Resolution: Fixed
Component/s:
Labels:

Assignee: hsingh
Reporter: teck
Created: February 05, 2010
Votes: 0
Watchers: 0
Updated:
February 12, 2013
Resolved:
February 05, 2010

Attachments

teck (4.00 M) binary/octet-stream tcquartztest.war

Description

Source: http://forums.terracotta.org/forums/posts/list/3055.page

The attached sample application behaves unexpectedly (some jobs get “lost” and don’t ever run again) using tim-quartz-1.3.0 with TC 3.2.0

Comments

Tim Eck 2010-02-05

The local TreeSet instance used in org.terracotta.quartz.TriggerSet was getting corrupted as elements were mutated while they were still in the set.

The fix now rebuilds the local set on every mutation (which is far from optimal performance wise) but is now no longer getting corrupted

Himadri Singh 2010-02-23

Verified

w/ tc - 3.2.0 and quartz - 1.7.1

2010-02-22 23:29:58,645 INFO [org.quartz.core.SchedulerSignalerImpl] - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2010-02-22 23:29:58,662 INFO [org.quartz.core.QuartzScheduler] - Quartz Scheduler v.1.7.1 created. 2010-02-22 23:29:58,782 INFO [org.quartz.core.SchedulerSignalerImpl] - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2010-02-22 23:29:58,794 INFO [org.quartz.core.QuartzScheduler] - Quartz Scheduler v.1.7.1 created. 2010-02-22 23:29:59,481 INFO [org.terracotta.quartz.ClusteredJobStore] - ClusteredJobStore initialized. 2010-02-22 23:29:59,483 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler ‘Wildfire Scheduler’ initialized from an externally provided properties instance. 2010-02-22 23:29:59,483 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler version: 1.7.1 2010-02-22 23:29:59,488 INFO [org.quartz.core.QuartzScheduler] - JobFactory set to: org.springframework.scheduling.quartz.AdaptableJobFactory@6fbc6c87 2010-02-22 23:29:59,656 INFO [org.terracotta.quartz.ClusteredJobStore] - ClusteredJobStore initialized. 2010-02-22 23:29:59,670 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler ‘Wildfire Scheduler’ initialized from an externally provided properties instance. 2010-02-22 23:29:59,670 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler version: 1.7.1 2010-02-22 23:29:59,677 INFO [org.quartz.core.QuartzScheduler] - JobFactory set to: org.springframework.scheduling.quartz.AdaptableJobFactory@da27335 2010-02-22 23:29:59,940 INFO [org.quartz.core.QuartzScheduler] - Scheduler Wildfire Scheduler_$_NON_CLUSTERED started. 2010-02-22 23:30:00,569 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:30:00,654 INFO [org.springframework.web.servlet.DispatcherServlet] - FrameworkServlet ‘super’: initialization completed in 2760 ms 2010-02-22 23:30:00,668 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:30:00,778 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:30:00,801 ERROR [OneAtATimeTriggerListener] - Do Super Things Every Five Seconds job is already executing (running for -1 ms, started Mon Feb 22 23:30:00 PST 2010), so new job is being vetoed and will not run. If this happens regularly, it may indicate a problem. 2010-02-22 23:30:01,017 INFO [org.quartz.core.QuartzScheduler] - Scheduler Wildfire Scheduler_$_NON_CLUSTERED started. 2010-02-22 23:30:01,139 INFO [org.springframework.web.servlet.DispatcherServlet] - FrameworkServlet ‘super’: initialization completed in 3088 ms 2010-02-22 23:30:16,060 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:30:30,016 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:30:30,062 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:31:00,014 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:31:00,025 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:31:30,016 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:31:30,028 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:31:43,257 INFO [org.terracotta.quartz.ClusteredJobStore] - Evaluating orphaned trigger TriggerWrapper( ACQUIRED, lastTC=ClientID[1], TriggerKey[superJob2, DEFAULT], Trigger ‘DEFAULT.superJob2’: triggerClass: ‘org.springframework.scheduling.quartz.CronTriggerBean isVolatile: false calendar: ‘null’ misfireInstruction: 0 nextFireTime: Mon Feb 22 23:31:00 PST 2010) 2010-02-22 23:32:00,007 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:32:30,012 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:33:00,008 INFO [MySuperQuartzJob] - I’m running yeah: superJob1

w/ tc -3.2.1 and quartz-1.7.2

2010-02-22 23:23:15,815 INFO [org.quartz.core.SchedulerSignalerImpl] - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2010-02-22 23:23:15,828 INFO [org.quartz.core.QuartzScheduler] - Quartz Scheduler v.1.7.2 created. 2010-02-22 23:23:16,054 INFO [org.quartz.core.SchedulerSignalerImpl] - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2010-02-22 23:23:16,085 INFO [org.quartz.core.QuartzScheduler] - Quartz Scheduler v.1.7.2 created. 2010-02-22 23:23:16,116 INFO [org.terracotta.quartz.ClusteredJobStore] - Synchronous write locking is [false] 2010-02-22 23:23:16,616 INFO [org.terracotta.quartz.ClusteredJobStore] - ClusteredJobStore initialized. 2010-02-22 23:23:16,617 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler ‘Wildfire Scheduler’ initialized from an externally provided properties instance. 2010-02-22 23:23:16,617 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler version: 1.7.2 2010-02-22 23:23:16,622 INFO [org.quartz.core.QuartzScheduler] - JobFactory set to: org.springframework.scheduling.quartz.AdaptableJobFactory@5a977cac 2010-02-22 23:23:16,762 INFO [org.terracotta.quartz.ClusteredJobStore] - ClusteredJobStore initialized. 2010-02-22 23:23:16,790 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler ‘Wildfire Scheduler’ initialized from an externally provided properties instance. 2010-02-22 23:23:16,790 INFO [org.quartz.impl.StdSchedulerFactory] - Quartz scheduler version: 1.7.2 2010-02-22 23:23:16,796 INFO [org.quartz.core.QuartzScheduler] - JobFactory set to: org.springframework.scheduling.quartz.AdaptableJobFactory@61cc1457 2010-02-22 23:23:16,953 INFO [org.quartz.core.QuartzScheduler] - Scheduler Wildfire Scheduler_$_NON_CLUSTERED started. 2010-02-22 23:23:17,041 INFO [org.springframework.web.servlet.DispatcherServlet] - FrameworkServlet ‘super’: initialization completed in 2083 ms 2010-02-22 23:23:17,819 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:23:18,069 INFO [org.quartz.core.QuartzScheduler] - Scheduler Wildfire Scheduler_$_NON_CLUSTERED started. 2010-02-22 23:23:18,189 INFO [org.springframework.web.servlet.DispatcherServlet] - FrameworkServlet ‘super’: initialization completed in 2919 ms 2010-02-22 23:23:18,243 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:23:30,025 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:23:30,040 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:24:00,011 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:24:00,035 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:24:03,100 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:24:30,019 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:24:30,028 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:24:33,101 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:24:49,319 INFO [org.terracotta.quartz.ClusteredJobStore] - Evaluating orphaned trigger TriggerWrapper( WAITING, lastTC=ClientID[0], TriggerKey[progTrigger, prog], Trigger ‘prog.progTrigger’: triggerClass: ‘org.quartz.SimpleTrigger isVolatile: false calendar: ‘null’ misfireInstruction: 0 nextFireTime: Mon Feb 22 23:24:33 PST 2010) 2010-02-22 23:24:49,320 INFO [org.terracotta.quartz.ClusteredJobStore] - Evaluating orphaned trigger TriggerWrapper( ACQUIRED, lastTC=ClientID[0], TriggerKey[superJob2, DEFAULT], Trigger ‘DEFAULT.superJob2’: triggerClass: ‘org.springframework.scheduling.quartz.CronTriggerBean isVolatile: false calendar: ‘null’ misfireInstruction: 0 nextFireTime: Mon Feb 22 23:25:00 PST 2010) 2010-02-22 23:25:00,011 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:25:00,022 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:25:03,105 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:25:18,100 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:25:30,009 INFO [MySuperQuartzJob] - I’m running yeah: superJob1 2010-02-22 23:25:30,020 INFO [MySuperQuartzJob] - I’m running yeah: superJob2 2010-02-22 23:25:33,099 INFO [ProgJob] - I was programmatically scheduled. W00T! 2010-02-22 23:25:48,100 INFO [ProgJob] - I was programmatically scheduled. W00T!