Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop failed timer task from breaking timers. #423

Open
wants to merge 1 commit into
base: master
from

Conversation

@mdaley
Copy link

mdaley commented Nov 23, 2019

Very interested to see whether this PR is considered to be useful...

At my current client, we came across the problem that for some reason (that we don't understand yet) a micro-service with a connection to the broker via the activemq-client would get in a state where every 5 seconds it would report a failure to connect to the broker with the reason being: IllegalStateException: Timer already cancelled.

There are several examples on the internet of this problem occurring but I've not found an actual solution to the problem, or something that explains what the underlying cause is.

After much digging, I've come to think that the basic cause of the problem is in java.util.Timer. Tasks controlled by a timer should never throw exceptions because, if they do, the timer then becomes unusable. What I think is happening is that for some reason we have a task that is throwing an exception. However, this exception gets masked by the behaviour of the Timer; and, then, the Timer just continues ad-infinitum with IllegalStateExceptions every five seconds.

The change I've made here is simply to put try/catch into the SchedulerTimerTask so the exception from the failed underlying task can be reported; and so that the timer does not actually break (maybe the next invocation of the underlying task will work fine if it were, for example, to be due to a connection failure or something else intermittent).

In java.util.Timer there is this code:

    public void run() {
        try {
            mainLoop();
        } finally {
            // Someone killed this Thread, behave as if Timer cancelled
            synchronized(queue) {
                newTasksMayBeScheduled = false;
                queue.clear();  // Eliminate obsolete references
            }
        }
    }

Interesting comment! Any exception in the mainLoop() kills the Timer; note that the mainLoop() does nothing itself to catch exceptions.

This problem has been noted elsewhere, for example: https://bugs.eclipse.org/bugs/show_bug.cgi?id=394215

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant
You can’t perform that action at this time.