Reactive Programming: What & Why? — Part IV: The eCommerce Integration Case

The eCommerce Integration Case

Let’s take a moment to contextualize all of the topics we briefly touched upon in the previous sections.

Integrating with an external system is, in most cases, a careful act of balancing internal and external complexity.

This is because we want to keep our system as simple as possible for maintainability and performance reasons, while at the same time fulfill all the requirements for a successful integration, while avoid tightly coupling our system with the one we want to integrate with due to the fact that we want to be ready to adapt to changes (for example in their APIs) quickly without having to rewrite major parts of our own code.

On the first half side of the external complexity aspect (the eCommerce provider), all of this is further exacerbated by the fact that when it comes to the eCommerce space where money is involved and it’s important not to disrupt the user flow (e.g. the checkout flow, the shipping flow and so on), additional requirements need to be imposed, such as near real-time event dispatch (for a fast experience), data persistence and replay capabilities (we need to be able to reprocess something if we missed it for some reason) and tolerance for sudden spikes in traffic volume (like a Black Friday event) where we want to prevent a complete loss of service if we get overwhelmed by the external system, to name a few.

On the other half  internal complexity side (our own customers that integrate through us to the eCommerce systems), the list of requirements does not get any shorter, because we interact with systems that are outside of our control and we need to be careful with them, for example by rate limiting our own output flow as not to overwhelm them, or being able to “pause” our own ongoing calls towards them for a while because they are unable to process what we are sending them but still be able to re-process this at a later date, just to make an example.

All of these additional requirements that we are considering end up adding other complexity to our own system that needs to be managed in the appropriate way to avoid creating unmaintainable system that we cannot operate and keep performant.

Ideally, we also want to insulate ourselves from the external system specific details as much as we can because we don’t want to mirror their own APIs, as that would make it very difficult for us to quickly adapt when things change and APIs get deprecated/removed.

Another point worth considering is the implicit complexity of integrating with different eCommerce systems that share some common parts that are similar enough that could be served by the same flow (think Authorization like OAuth 2) but not similar enough that a shared microservice would be useful.

To tackle all of this and many other requirements that we are not discussing in this first article, we decided that using the approach of Reactive Programming would be the right thing to do, after carefully evaluating its tradeoffs.

A quick look at these tradeoffs, in no particular order:

1. Higher and steeper learning curve if one doesn’t have a background in Functional Programming already
2. Lack of proper resources to learn how to treat a complex business logic and how to model it following the reactive methodology.
   Luckily there is an increasing interest in the field and we are seeing more and more material on this side, but this was not the case just a couple of years ago.
3. A standardized way of defining a pipeline that we can leverage to implement a common series of operations on a given reactive collection, so that the similarities between the various flows for the eCommerce systems are encoded in the operations performed, but flexible enough that we can plug-in our own code for the steps where it makes sense to do so, e.g. to customize some result handling.
4. A reusable approach of defining structures and series of operations that we can reuse in multiple places and even place into an internal library that reduces the amount of custom written code for shared tasks.
5. An integrated approach to flow control/backpressure backed into the framework itself, which means that we don’t have to do it ourselves and we are, out of the box, able to leverage it and make our system more resilient with little to no effort.
6. The possibility of testing each module in isolation by testing all the steps that are defined in the pipelines and verifying that they do what they promised.

We can easily do this because, ideally, each step is a specialized piece of code with just one responsibility and there is little to no coupling with other modules.

7. Debugging experience is a bit different from the classical model because code is executed asynchronously now, but there are a lot of helpers from the framework itself that help, so the debugging experience ends up being more or less the same.

Conclusion

If you made it this far, thank you and congratulations!

Now that you know the basics of the topic, some tools of the trade and you have been introduced to some real-world examples, how do you really apply this knowledge in practice?

What are the next steps?

How do you properly design a reactive application?

Stay tuned for the second part of this article called: Reactive Programming – How and When where we’ll go in depth on how this fits all together, discussing how adopting the Reactive Programming approach might the right idea and when it might be best to do so, considering the existing design of a previous system and discussing the most common pitfalls that you are likely to encounter when adopting it.

<< Part III: Project Reactor                                                                                                                         >> Reactive Programming: How & When?