Transactional Outbox pattern with Spring Boot

Pattern: Transactional outbox

Background

suppose in a transaction, we have following actions:
db operation 1 – sending message – db operation 2

issues here is:

  • if db operation 2 failed, the transaction data will roll back, but the message is sent.

one solution here is:

  • db operation 1/2/.. – sending message

we put the sending message in the last step, within the transaction.

most of us may do in this way, it’s simple and reliable. only concern is it could be a long transaction as we interacting with exteral system.

do we have other ways?

  • here comes transaction outbox pattern.

Definition

The Transactional Outbox is a way to ensure that 2 systems are in sync without having to use a distributed transaction between those systems.

with this pattern, we can first store the fact (send email/message…) that should do some external action in database. Then, an asynchronous process can look at the database to know what still needs to happen, and can do that whenever there is time. If the external system is not available, the task can be retried later until it succeeds.

Implementation

option 1: Spring Integration.

put the msg to a jdbc-backed output with a polling handler.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
@Bean
JdbcChannelMessageStore jdbcChannelMessageStore(DataSource dataSource) {
    JdbcChannelMessageStore jdbcChannelMessageStore = new JdbcChannelMessageStore(dataSource);
    jdbcChannelMessageStore.setTablePrefix("_spring_integration_");
    jdbcChannelMessageStore.setChannelMessageStoreQueryProvider(new PostgresChannelMessageStoreQueryProvider());
    return jdbcChannelMessageStore;
}

@Bean
public IntegrationFlow consumerFlow(JdbcChannelMessageStore jdbcChannelMessageStore,
    ConsumerService consumerService) {
return IntegrationFlow.from(msgInputDirectChannel())
    .channel(msgOutboxQueueChannel(jdbcChannelMessageStore))
    .handle(message -> {
        consumerService.consume(message.getPayload());
    }, e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(1)).transactional()))
    .get();
}

option 2: Spring Modulith

Communication between modules can be done asynchronously by using the ApplicationEventPublisher from Spring core.

Spring Modulith has additional infrastructure to ensure no such event is ever lost by first storing it in the database. We can leverage this to build our outbox pattern.

1
2
3
4
5
6
7
8
9
10
11
12
13
// first transaction, ensure event are stored in db.
@Transactional
void register(User user) {
    User registeredUser = userRepository.save(user);
    applicationEventPublisher.publishEvent(new UserRegisteredEvent(registeredUser.getId()));
}

// second async thread transaction, ensure msg are deliverd to external system.
@ApplicationModuleListener
void onUserRegistered(UserRegisteredEvent userRegisteredEvent) {
    log.info("user registered. id:{}", userRegisteredEvent.id());
    // send email/message, etc.
}