When to use the Pipeline Architecture Style

Gray pipe on green grass photo.

What is it?

The pipeline architecture is one of the most common architecture styles used in designing software systems. Also known as pipes and filters, it consists of a series of discrete steps performed in a predictable sequence. This is different from the model-view-controller pattern in a layered architecture. In this article I’ll define what it is and when to use the pipeline architecture style.

A pipeline is commonly used in an event-driven architecture. A business event happens in one system or application, and other systems need to be updated accordingly.

Here is a simple example of a pipeline architecture:

Diagram of a pipeline architecture
The basic structure of a pipeline architecture.

A “filter” in this context is a discrete processing step, such as transform, lookup, enrich, or write to a database. A filter does one thing and one thing only (the Single-Responsibility Principle).

The pipes are one-way conduits from one filter to another. They take the output of one filter and pass it on to the next one. The payloads that the filters send through the pipes can be any data format, but typically they are JSON or XML.

When To Use the Pipeline Architecture Style

The most common use case for a pipeline architecture is system integration. Any time you need to move data from one application to another, a pipeline is a very suitable style. Because you define the integration solution as a series of discrete steps, it makes for a very modular design. The filters themselves are highly cohesive and loosely coupled, meaning they do one thing only, and a filter has no dependency on other filters. This makes unit tests much easier to write, and it leads to a cleaner design that you can modify with minimal fuss. Changes to a filter mean a trivial impact to the overall integration, provided there are no changes to its output contract.

System integration is a use case that has been around for several years. Many organizations need to solve the problem of moving data from one application to another in response to some business event. These applications are often very different in terms of the data structures they expose. Connecting these applications is what system integration is all about.

Over the years, a set of common design patterns have emerged. Gregor Hohpe and Bobby Woolf captured these patterns in their seminal book Enterprise Integration Patterns. The companion website does a great job of outlining the various patterns. If you design or develop integration solutions, this book should be in your library and the website should be in your browser bookmarks.

Designing a Pipeline Architecture

System integration typically consists of a combination of Validate, Extract, Transform and Operate (VETO) steps. Begin your design effort with the target system. Determine what it needs in terms of data structure, fields and values, then figure out what it takes to get the source data to there.

For example, suppose you have an insurance claim management system. Various claim intake systems accept a new claim and publish it as an XML message on a message broker. You need to accept an XML message from the message broker, and call a web service to save the claim in the claim management system. Along the way you need to convert the XML message to JSON, and enrich the message using a database lookup to add some additional information. Here is the architecture of our example:

Diagram of a sample claim integration architecture.
Her’s a simple example of integration claims intake system with a claims management system.

Spring Integration makes this simple. You use the framework to configure the construction of the pipeline, then all you need to do is implement the logic in the filters. The framework takes care of passing the message payloads from one filter to the next.

Advantages

A pipeline architecture is extensible. Should you need to add additional steps, such as logging or enhanced error handling, simply insert them in the pipeline.

The cost of implementing a pipeline architecture is quite attractive. It’s normally implemented as a monolith, so it’s relatively easy to understand and build. You don’t have the complex orchestration or choreography you normally see with distributed architectures like microservices.

When you use a framework like Spring Integration, the cost to develop is low since the framework takes care of the undifferentiated effort of moving messages from one filter to another. You just develop the filters. Moreover, Spring Integration builds on Spring Framework. If you are already using the latter as your dependency injection framework, developers can learn the former with minimal effort.

Spring Integration reads and processes each message in a separate thread. This gives some boost to performance while isolating any processing failures to the message in question.

Drawbacks

Since a pipeline is frequently implemented as a monolith, it means changes of any size need to go through a full regression test. It is also subject to the same ceremony of an all-or-nothing deployment, along with the attendant risks.

A pipeline architecture is not very fault tolerant. If one of the filters causes a heap space exhaustion situation, the entire integration comes crashing down. You can however mitigate this with a decent monitoring and alerting strategy.

Wrapping Up

If you have a need to integrate two or more systems together, give serious consideration to a pipeline architecture. Avail yourself of the examples in the Enterprise Integration Patterns book.

How to Migrate Your Web Application to the Cloud

photography of cloud and forest trees

So it’s time to move one of your legacy web applications from your on-premise data centre to a public cloud provider such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). This may be a decision you made, or one that your leadership team made. In this post I’ll explain what you need to think about when figuring out how to migrate your web application to the cloud.

Read more

Microservices: When to Use Them

Brown and white long coated small dog wearing eyeglasses on black laptop computer

Introduction

Microservices are all the rage these days. You’ve seen so many blog posts, technical articles, even job postings calling for microservices experience. So it must be the new way to architect applications, right?

Well, as with everything else in software architecture, it depends. It depends on the context you are dealing with. It depends whether the benefits of microservices outweigh their drawbacks for your situation.

In this post I’ll describe what microservices are, why they’re so good, and what their drawbacks are. I’ll finish off by giving you some guidelines to help you decide whether microservices are right for your situation.

Read more

Software Product Teams are Better Than Project Teams

People working at a table.

When it comes to software development, product teams are better than project teams. I’ll explain why.

Seen This Before?

Have you noticed a pattern that most organizations use to build software? They assign a project manager to prepare the project charter that broadly defines the scope, cost and schedule. Senior management approves the charter, and then they assemble a team. Developers, analysts, QA testers join the new team even though they may be winding up other projects. Management assigns an architect. They might also assign a DBA, network and middleware experts on a part-time basis.

Read more

Become a Great Software Architect by Being a Great Communicator

Man giving a presentation.

Software architects are responsible for the technical solution to ensure it achieves the desired business outcomes. To do so requires broad technical and business knowledge with a deeper understanding in a couple technical areas. It requires being able to see the big picture. It requires the wisdom to evaluate different solutions to the problem. But most of all it requires really good communication skills to convey the solution. You can become a great software architect by being a great communicator.

Read more

When to Use the The Layered Architecture Style

Layer cake

A layered architecture style is one where the presentation, business and persistence layers are bundled into one deployment unit. In this article I’ll discuss what it is, and point out the situations where you would choose to adopt it.

This is one of a series of articles where I am going to discuss each of these architecture styles: Layered, Microservices, Pipeline, Microkernel, Service-Based, Event-Driven, Space-Based, and Orchestration-Driven Service-Oriented Architecture (SOA). I’ll provide some insights based on my experiences, and I’ll mention some of the Java / JVM frameworks that support the style we’re talking about.

Read more

Leadership Techniques for the Software Architect

Man standing in front of people sitting beside table with laptop computers.

Why is Leadership Important?

The typical path to becoming a software architect begins with years spent in the trenches of hands-on software development. You’ve accumulated technical knowledge that is broad and deep. Along the way you’ve developed your communications skills. And I don’t just mean status reports. I mean by mentoring junior developers, by giving presentations to your peers on the knowledge you have gained in a specific tool, framework or language, and by interacting with systems and business analysts.

Now that you’ve become an architect, more people will look to you as an authority, as an expert. Now you’re starting to provide guidance on software design and the usage of tools and frameworks. You notice you have more influence. You are now responsible for the technical solution of the product your team is building. You are now in a position of leadership.

Read more

Write More Maintainable Software With a Hexagonal Architecture

Adopting the hexagonal architecture pattern produces software that is more maintainable. It enables you to respond to changes with less fuss than many other architectural patterns. In this article I’ll explain why and offer my thoughts on this pattern.

What is a Hexagonal Architecture?

Alistair Cockburn first coined the term Hexagonal Architecture on his blog in 2005. Also known as the Ports and Adapters pattern, it is a layered architecture. It is a way of separating the domain concerns while making unit tests easier to write and changes simpler to accommodate.

Read more

Are Microservices The Same Old SOA?

Are microservices the same old SOA (Service Oriented Architecture)? Isn’t it just a rehash of the SOA hype? Let’s discuss the similarities and differences.

While microservices and Service Oriented Architecture (SOA) are two architecture patterns that deal with services, they have some differences. In fact, despite all the attention you see being given to microservices there are a couple use cases where SOA is the better choice.

So What Is a Microservice?

A microservice a stand-alone process that communicates with its clients using a lightweight mechanism, usually HTTP. Its datastore is a database of some sort. The service is the only application that interacts with its datastore. So let’s say for example you have a CustomerService that reads from and writes to the CUSTOMER table and any related child tables. Only the CustomerService performs SQL actions on those tables; any other services that do something with Customer information must go through the CustomerService using its published interface.

Read more