Week 20: Batch Processing with Spring Batch

1. Jobs, Steps & Chunk-Oriented Processing

A Spring Batch Job is the whole unit of work — "import today's orders file" — composed of one or more Steps, each an independent, restartable phase. Most steps use chunk-oriented processing: read one item, process it, and repeat until a configured chunk size is reached, then write the entire chunk in one transaction — rather than reading everything into memory at once, or committing after every single item.

the chunk loop, conceptually

// For a chunk size of 100:
// read 1 item -> process it -> repeat 100 times -> write all 100 -> COMMIT
// read 1 item -> process it -> repeat 100 times -> write all 100 -> COMMIT
// ...continues until the reader returns null (no more items)

This shape is deliberate: reading and processing happen item-by-item (so memory use stays flat regardless of file size), while writing happens in batches (so the database isn't hit with one round-trip per row). If step 50,000 of a 200,000-row job fails, only the current, uncommitted chunk rolls back — the 49,900 rows already committed in earlier chunks stay committed, which is exactly what makes Section 3's restart story possible.

Chunk size is a real tuning knob, not a fixed default

Too small (e.g. 1) means paying a database round-trip per item, similar to the unbatched inserts Week 15 warned about. Too large means a single failed item near the end of a huge chunk rolls back a lot of already-correct work. A few hundred to a few thousand, depending on row size and how expensive processing each item is, is a reasonable starting point — measured, not guessed.

2. A Real Reader/Processor/Writer Job

A chunk-oriented step is built from three pieces implementing well-defined interfaces: an ItemReader that produces items one at a time, an ItemProcessor that transforms or validates each one (returning null to filter an item out entirely), and an ItemWriter that persists a completed chunk.

FlatFileItemReader — reading a CSV of orders

@Bean
FlatFileItemReader<OrderCsvRow> orderItemReader() {
    return new FlatFileItemReaderBuilder<OrderCsvRow>()
        .name("orderItemReader")
        .resource(new FileSystemResource("import/orders.csv"))
        .delimited().delimiter(",")
        .names("customerId", "sku", "quantity", "orderedAt")
        .targetType(OrderCsvRow.class)
        .linesToSkip(1)   // header row
        .build();
}

ItemProcessor — validating & transforming each row

@Component
class OrderRowProcessor implements ItemProcessor<OrderCsvRow, Order> {

    @Override
    public Order process(OrderCsvRow row) {
        if (row.quantity() <= 0) {
            return null;   // filters this row out of the chunk entirely -- no exception needed
        }
        return new Order(row.customerId(), row.sku(), row.quantity(), row.orderedAt());
    }
}

JdbcBatchItemWriter & assembling the Step and Job

@Bean
JdbcBatchItemWriter<Order> orderItemWriter(DataSource dataSource) {
    return new JdbcBatchItemWriterBuilder<Order>()
        .dataSource(dataSource)
        .sql("""
            INSERT INTO orders (customer_id, sku, quantity, ordered_at)
            VALUES (:customerId, :sku, :quantity, :orderedAt)
            """)
        .beanMapped()
        .build();
}

@Bean
Step importOrdersStep(JobRepository jobRepository, PlatformTransactionManager txManager,
                       FlatFileItemReader<OrderCsvRow> reader, OrderRowProcessor processor,
                       JdbcBatchItemWriter<Order> writer) {
    return new StepBuilder("importOrdersStep", jobRepository)
        .<OrderCsvRow, Order>chunk(500, txManager)
        .reader(reader)
        .processor(processor)
        .writer(writer)
        .build();
}

@Bean
Job importOrdersJob(JobRepository jobRepository, Step importOrdersStep) {
    return new JobBuilder("importOrdersJob", jobRepository)
        .start(importOrdersStep)
        .build();
}

Notice the processor returning null for an invalid row — that's a first-class part of the contract, not a workaround; Spring Batch simply excludes that item from the chunk that gets written, with no exception, no special-casing in the writer. .chunk(500, txManager) ties directly back to Section 1: 500 rows read and processed individually, then written and committed together as one unit.

3. Restartability, Skip/Retry Policies & Scheduling

Every Job execution's progress is recorded in Spring Batch's own metadata tables (BATCH_JOB_EXECUTION, BATCH_STEP_EXECUTION, and related), which is what makes restart possible at all: rerunning a Job with the exact same parameters after a failure resumes from the last successfully committed chunk instead of starting over from row one.

launching, and re-launching after a failure

JobParameters params = new JobParametersBuilder()
    .addString("file", "import/orders.csv")
    .addLong("run.id", System.currentTimeMillis())   // see callout below
    .toJobParameters();

jobLauncher.run(importOrdersJob, params);

// if it fails at row 120,000 of 200,000 and you re-run with the SAME
// job parameters, Spring Batch resumes from the last committed chunk --
// it does not reprocess rows 1 through ~119,500 again

A batch job also needs a policy for individual bad rows that isn't "the whole Job fails." Skip and retry policies handle exactly that, configured per step:

tolerating a bounded number of bad rows

Step importOrdersStep = new StepBuilder("importOrdersStep", jobRepository)
    .<OrderCsvRow, Order>chunk(500, txManager)
    .reader(reader)
    .processor(processor)
    .writer(writer)
    .faultTolerant()
    .skip(FlatFileParseException.class)   // a malformed CSV line
    .skipLimit(50)                        // give up on the whole job past 50 bad rows
    .retry(TransientDataAccessException.class)
    .retryLimit(3)                        // e.g. a brief database connection blip
    .build();

skipLimit(50) is the key design decision: a handful of malformed rows in a 200,000-row file is expected and shouldn't fail the entire import, but a file that's 50+ rows malformed is very likely the wrong file entirely, or corrupted, and should stop the Job rather than silently skip half the data.

Finally, a nightly import needs to actually run on a schedule. @Scheduled, the same mechanism from Week 12's async work, triggers the Job:

scheduling the job nightly

@Component
class OrderImportScheduler {

    private final JobLauncher jobLauncher;
    private final Job importOrdersJob;

    @Scheduled(cron = "0 0 2 * * *")   // 02:00 every day
    void runNightlyImport() throws Exception {
        JobParameters params = new JobParametersBuilder()
            .addLong("run.id", System.currentTimeMillis())
            .toJobParameters();
        jobLauncher.run(importOrdersJob, params);
    }
}

Job parameters are the identity of a run — get this wrong and restart breaks

Spring Batch identifies a Job execution by its exact combination of parameters; the same parameters mean "the same run," which is what enables restart-from-failure. A scheduled job that's meant to run fresh every night needs a parameter that changes each time (like the timestamp above) — otherwise every nightly run after the first would be treated as a restart of an already-completed Job and refuse to run at all.

4. Hands-on Exercise

Hands-on

Build a restartable CSV import job with skip logic

Import a real CSV into your task/order database with a fault-tolerant Spring Batch job.

Requirements:

Build a chunk-oriented Job with a FlatFileItemReader, a validating ItemProcessor that filters out invalid rows, and a JdbcBatchItemWriter, importing a CSV of at least 10,000 rows into a real table.
Deliberately corrupt a row partway through the file, run the job, and confirm it fails at that point — then re-run it with the same job parameters and confirm it resumes rather than reprocessing already-committed rows.
Add faultTolerant() with a skip policy and a reasonable skip limit, seed the file with a handful of malformed rows below that limit, and confirm the job completes successfully while skipping only the bad rows.
Schedule the job to run automatically with @Scheduled, using a parameter that changes on every run so successive scheduled executions aren't treated as restarts of each other.

Hint

Query BATCH_STEP_EXECUTION directly after a failed and resumed run — the read_count, write_count, and commit_count columns show exactly how much work each execution actually did, which makes the restart behavior concrete instead of something you have to take on faith.

5. Knowledge Check

Three quick questions. Expand each to check your answer.

Why does chunk-oriented processing read and process items one at a time but write them in batches?

Reading and processing item-by-item keeps memory usage flat regardless of how large the input file is — nothing requires loading the entire dataset at once. Writing in batches avoids the cost of one database round-trip per row, the same reasoning behind Week 15's JDBC batch inserts. Combining both gives constant memory use and efficient database writes at the same time.

Why does restarting a failed Job with the same job parameters resume from the last committed chunk instead of reprocessing everything?

Spring Batch persists execution progress to its own metadata tables as each chunk commits, and identifies a specific run by its exact combination of job parameters. Launching the Job again with identical parameters is recognized as a restart of that same execution, so Spring Batch resumes from the last successfully committed chunk rather than treating it as a brand-new run starting from the first row.

Why does a scheduled nightly Job need a parameter that changes on every run, like a timestamp?

Because job parameters define a run's identity, a scheduled Job launched with the exact same parameters every night would be treated as a restart of the very first night's (already-completed) execution, and Spring Batch would refuse to run it again. A parameter that changes each time, like the current timestamp, ensures every night's invocation is recognized as a genuinely new execution rather than a restart of a previous one.