1. Overview
By default, a Spring batch job fails for any errors raised during its execution. However, at times, we may want to improve our application's resiliency to deal with intermittent failures.
In this quick tutorial, we'll explore how to configure retry logic in the Spring Batch framework.
2. An Example Use Case
Let's say we have a batch job that reads an input CSV file:
username, userid, transaction_date, transaction_amount
sammy, 1234, 31/10/2015, 10000
john, 9999, 3/12/2015, 12321
Then, it processes each record by hitting a REST endpoint to fetch the user's age and postCode attributes:
public class RetryItemProcessor implements ItemProcessor<Transaction, Transaction> {
@Override
public Transaction process(Transaction transaction) throws IOException {
log.info("RetryItemProcessor, attempting to process: {}", transaction);
HttpResponse response = fetchMoreUserDetails(transaction.getUserId());
//parse user's age and postCode from response and update transaction
...
return transaction;
}
...
}
And finally, it generates a consolidated output XML:
<transactionRecord>
<transactionRecord>
<amount>10000.0</amount>
<transactionDate>2015-10-31 00:00:00</transactionDate>
<userId>1234</userId>
<username>sammy</username>
<age>10</age>
<postCode>430222</postCode>
</transactionRecord>
...
</transactionRecord>
3. Adding Retries to ItemProcessor
Now, what if the connection to the REST endpoint times out due to some network slowness? If so, our batch job will fail.
In such cases, we'd prefer the failed item processing to be retried a couple of times. And so, let's configure our batch job to perform up to three retries in case of failures:
@Bean
public Step retryStep(
ItemProcessor<Transaction, Transaction> processor,
ItemWriter<Transaction> writer) throws ParseException {
return stepBuilderFactory
.get("retryStep")
.<Transaction, Transaction>chunk(10)
.reader(itemReader(inputCsv))
.processor(processor)
.writer(writer)
.faultTolerant()
.retryLimit(3)
.retry(ConnectTimeoutException.class)
.retry(DeadlockLoserDataAccessException.class)
.build();
}
Here, we have a call to faultTolerant() for enabling the retry functionality. Additionally, we use retry and retryLimit to define the exceptions that qualify for a retry and the maximum retry count for an item, respectively.
4. Testing the Retries
Let's have a test scenario where the REST endpoint returning age and postCode was down just for a while. In this test scenario, we'll get a ConnectTimeoutException only for the first two API calls, and the third call will succeed:
@Test
public void whenEndpointFailsTwicePasses3rdTime_thenSuccess() throws Exception {
FileSystemResource expectedResult = new FileSystemResource(EXPECTED_OUTPUT);
FileSystemResource actualResult = new FileSystemResource(TEST_OUTPUT);
when(httpResponse.getEntity())
.thenReturn(new StringEntity("{ \"age\":10, \"postCode\":\"430222\" }"));
//fails for first two calls and passes third time onwards
when(httpClient.execute(any()))
.thenThrow(new ConnectTimeoutException("Timeout count 1"))
.thenThrow(new ConnectTimeoutException("Timeout count 2"))
.thenReturn(httpResponse);
JobExecution jobExecution = jobLauncherTestUtils
.launchJob(defaultJobParameters());
JobInstance actualJobInstance = jobExecution.getJobInstance();
ExitStatus actualJobExitStatus = jobExecution.getExitStatus();
assertThat(actualJobInstance.getJobName(), is("retryBatchJob"));
assertThat(actualJobExitStatus.getExitCode(), is("COMPLETED"));
AssertFile.assertFileEquals(expectedResult, actualResult);
}
Here, our job completed successfully. Additionally, it's evident from the logs that the first record with id=1234 failed twice and finally succeeded on the third retry:
19:06:57.742 [main] INFO o.s.batch.core.job.SimpleStepHandler - Executing step: [retryStep]
19:06:57.758 [main] INFO o.b.batch.service.RetryItemProcessor - Attempting to process user with id=1234
19:06:57.758 [main] INFO o.b.batch.service.RetryItemProcessor - Attempting to process user with id=1234
19:06:57.758 [main] INFO o.b.batch.service.RetryItemProcessor - Attempting to process user with id=1234
19:06:57.758 [main] INFO o.b.batch.service.RetryItemProcessor - Attempting to process user with id=9999
19:06:57.773 [main] INFO o.s.batch.core.step.AbstractStep - Step: [retryStep] executed in 31ms
Similarly, let's have another test case to see what happens when all retries are exhausted:
@Test
public void whenEndpointAlwaysFail_thenJobFails() throws Exception {
when(httpClient.execute(any()))
.thenThrow(new ConnectTimeoutException("Endpoint is down"));
JobExecution jobExecution = jobLauncherTestUtils
.launchJob(defaultJobParameters());
JobInstance actualJobInstance = jobExecution.getJobInstance();
ExitStatus actualJobExitStatus = jobExecution.getExitStatus();
assertThat(actualJobInstance.getJobName(), is("retryBatchJob"));
assertThat(actualJobExitStatus.getExitCode(), is("FAILED"));
assertThat(actualJobExitStatus.getExitDescription(),
containsString("org.apache.http.conn.ConnectTimeoutException"));
}
In this case, three retries were attempted for the first record before the job finally failed due to a ConnectTimeoutException.
5. Configuring Retries Using XML
Finally, let's look at the XML equivalent of the above configurations:
<batch:job id="retryBatchJob">
<batch:step id="retryStep">
<batch:tasklet>
<batch:chunk reader="itemReader" writer="itemWriter"
processor="retryItemProcessor" commit-interval="10"
retry-limit="3">
<batch:retryable-exception-classes>
<batch:include class="org.apache.http.conn.ConnectTimeoutException"/>
<batch:include class="org.springframework.dao.DeadlockLoserDataAccessException"/>
</batch:retryable-exception-classes>
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
6. Conclusion
In this article, we learned how to configure retry logic in Spring Batch. We looked at both Java and XML configurations.
We also used a unit test to see how the retries worked in practice.
As always, the example code for this tutorial is available over on GitHub.