Tukukkk

I have a ScheduledExecutorService to schedule a task to run every 12 hours in a tomcat application. The task calls a REST endpoint and performs some other calculations and returns a result. It works perfectly fine locally and in test server however, it runs only once and never again in the live server. All exceptions are handled and the task takes less than 5 seconds to complete. I have also checked all properties in the server to make sure the time properties are not overridden by other properties. To test locally, I have reduced the delay time to every 10 minutes and this still shows the same behaviour so I’ve ruled time out as the issue. I have looked at other similar questions but none seem to help with this issue.
ScheduledExecutorService only runs once,
JAVA ScheduledExecutorService only runs once when calling a Task<V>,
ScheduledExecutorService only loops once,
ScheduledExecutorService - Task stops running,
ScheduledExecutorService - Ignore already running runnable,
ScheduledExecutorService schedule issue,
Timer vs. ScheduledExecutorService scheduling

Below is my code:

@JsonIgnoreProperties(ignoreUnknown = true)

@JsonPropertyOrder({"name", "number"})

public class Company {



    @JsonProperty("name")

    private String name;

    @JsonProperty("number")

    private int number;



    public String getName() {

        return this.name;

    }



    public int getNumber() {

        return this.number;

    }



    @Override

    public String toString() {

        return new ToStringBuilder(Company.class)

            .append("name", this.name)

            .append("number", this.number).toString();

    }

}



public class CompanyDifference {



    private String name;

    private int difference;



    public CompanyDifference(String name, int difference) {

        this.name = name;

        this.difference = difference;

    }



    public String getName() {

        return this.name;

    }



    public int getDifference() {

        return this.difference;

    }



    @Override

    public String toString() {

        return new ToStringBuilder(CompanyDifference.class)

            .append("name", this.name)

            .append("difference", this.difference).toString();

    }

}



@Singleton

public class TaskRunner {

    public void doTask () {

        try {

            System.out.println("takes 2 sets of json data and returns the difference for each company");



            // takes 2 sets of json data and returns the difference for each company

            ObjectMapper mapper = new ObjectMapper();

            InputStream dataOne = Company.class.getResourceAsStream("/data.json");

            InputStream dataTwo = Company.class.getResourceAsStream("/data2.json");



            Company companyDataOne = mapper.readValue(dataOne, Company.class);

            Company companyDataTwo = mapper.readValue(dataTwo, Company.class);



            // Find the difference for each company and map company name to difference

            Map<String, Integer> mapDifferenceToCompany = new HashMap<>();



            for (int i = 0; i < companyDataOne.length; i++) {

                mapDifferenceToCompany.put(companyDataOne[i].getName(), Math.abs(companyDataOne[i].getNumber() - companyDataTwo[i].getNumber()));

            }



            mapDifferenceToCompany.forEach((key, value) -> System.out.println(String.valueOf(new CompanyDifference(key, value))));

        } catch (IOException e) {

            logger.info(String.format("Error: Failed to convert json to object with exception %s", e));

            throw new TaskSchedulerException("Failed to convert json to object with exception", e);

        } catch (Exception e) {

            logger.info(String.format("Error: Failed with exception %s", e));

            throw new TaskSchedulerException("Failed with exception", e);

        }

    }

}



@Singleton

public class TaskScheduler {



    private final Runnable runnable; 

    private final ScheduledExecutorService executorService;

    private static final Logger logger = LoggerFactory.getLogger(TaskScheduler.class);



    @Inject

    public TaskScheduler(TaskRunner taskRunner, int initialDelay, int period, String timeUnits) {

        this.executorService = Executors.newScheduledThreadPool(1);

        this.runnable = taskRunner::doTask;



        this.scheduledFuture = this.executorService.scheduleAtFixedRate(this.runnable, initialDelay, period,    

             TimeUnit.valueOf(timeUnits));

    }



    public static void main(String args) {

        TaskRunner taskRunner = new TaskRunner();

        new TaskScheduler(taskRunner, 1, 10, "MINUTES");

    }

}

The task runs once on start-up after the initial delay of 1 minute but doesn’t run the next scheduled task. It attempts to run the scheduled task but doesn’t seem to complete the task and after examining the ScheduledThreadPoolExecutor properties in the live server, I find that the queue size drops to 0 (when it should always be 1) meaning that no task has been scheduled.

This suggests that when it attempts to run the scheduled task after the initial task is completed, it either removes the scheduled task or it fails to schedule the next task because the current task isn’t completed. No errors or exceptions are thrown as all exceptions have been handled in the doTask method. The scheduler works as expected locally and in the test server which makes it difficult to replicate the scenario.

Want to find out if the implementation is missing something or what might be causing it to not complete the next scheduled task and what makes the queue size drop to 0. Could the java version have any impact on the behaviour of the scheduler or are there any reasons why this might be happening in a live environment?

I created a REST endpoint to monitor what was going on under the hood with the properties from ScheduledFuture and ScheduledThreadPoolExecutor

ScheduledThreadPoolExecutor executor = (ScheduledThreadPoolExecutor) schedulerService;

this.queueSize = executor.getQueue().size();

this.remainingCapacity = executor.getQueue().remainingCapacity();

this.terminated = schedulerService.isTerminated();

this.shutdown = schedulerService.isShutdown();

this.taskCount = executor.getTaskCount();

this.activeTaskCount = executor.getActiveCount();

this.completedTaskCount = executor.getCompletedTaskCount();

this.keepAliveTime = executor.getKeepAliveTime(TimeUnit.SECONDS);

this.coreThreadTimeOut = executor.allowsCoreThreadTimeOut();

this.cancelled = scheduledFuture.isCancelled();

this.delay = scheduledFuture.getDelay(TimeUnit.MINUTES);

Result of first run after initial delay: from this, we can see that it completed the first task and I do get the required result from the calculations. The taskCount (number of scheduled tasks since start-up) is 2 meaning that the 2nd task has been scheduled and the queue size is still 1 which is fine.

{

  "queueSize": 1,

  "remainingCapacity": 2147483647,

  "terminated": false,

  "shutdown": false,

  "taskCount": 2,

  "activeTaskCount": 0,

  "completedTaskCount": 1,

  "keepAliveTime": 0,

  "coreThreadTimeOut": false, 

  "periodic": true, 

  "cancelled": false

}

Result after it attempted 2nd run: this is where it gets stuck. The completedTaskCount is 2 but I don’t think it actually completes the task as I don’t get the result from the calculations or any logs to show that it has either started or completed the task. The taskCount should go up to 3 but it is stuck in 2 and the queue size is now 0.

{

  "queueSize": 0,

  "remainingCapacity": 2147483647,

  "terminated": false,

  "shutdown": false,

  "taskCount": 2,

  "activeTaskCount": 0,

  "completedTaskCount": 2,

  "keepAliveTime": 0,

  "coreThreadTimeOut": false, 

  "periodic": true, 

  "cancelled": false

}

When I check these on the local and test server, it works fine and the taskCount goes up as expected and the queue size is always 1 which is expected. From this, I can tell that for some reason, the task gets stuck on the 2nd run and does not complete so it doesn’t schedule the next task.

The javadoc says: "If any execution of this task takes longer than its period, then subsequent executions may start late, but will not concurrently execute." I'm assuming this is why the next task is not scheduled. It would be great if you can explain what might cause this to happen

Issue was not with the implementation, the implementation is fine. After much debugging, the VM box running the app seemed to have had a glitch. After restarting multiple times and redeploying app service, It's back to normal.
Please see question for steps taken when debugging app.