Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upStop failed timer task from breaking timers. #423
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
mdaley commentedNov 23, 2019
•
edited
Very interested to see whether this PR is considered to be useful...
At my current client, we came across the problem that for some reason (that we don't understand yet) a micro-service with a connection to the broker via the activemq-client would get in a state where every 5 seconds it would report a failure to connect to the broker with the reason being: IllegalStateException: Timer already cancelled.
There are several examples on the internet of this problem occurring but I've not found an actual solution to the problem, or something that explains what the underlying cause is.
After much digging, I've come to think that the basic cause of the problem is in
java.util.Timer. Tasks controlled by a timer should never throw exceptions because, if they do, the timer then becomes unusable. What I think is happening is that for some reason we have a task that is throwing an exception. However, this exception gets masked by the behaviour of the Timer; and, then, the Timer just continues ad-infinitum with IllegalStateExceptions every five seconds.The change I've made here is simply to put try/catch into the SchedulerTimerTask so the exception from the failed underlying task can be reported; and so that the timer does not actually break (maybe the next invocation of the underlying task will work fine if it were, for example, to be due to a connection failure or something else intermittent).
In java.util.Timer there is this code:
Interesting comment! Any exception in the mainLoop() kills the Timer; note that the mainLoop() does nothing itself to catch exceptions.
This problem has been noted elsewhere, for example: https://bugs.eclipse.org/bugs/show_bug.cgi?id=394215