When to use the Pipeline Architecture Style

Gray pipe on green grass photo.

What is it?

The pipeline architecture is one of the most common architecture styles used in designing software systems. Also known as pipes and filters, it consists of a series of discrete steps performed in a predictable sequence. This is different from the model-view-controller pattern in a layered architecture. In this article I’ll define what it is and when to use the pipeline architecture style.

A pipeline is commonly used in an event-driven architecture. A business event happens in one system or application, and other systems need to be updated accordingly.

Here is a simple example of a pipeline architecture:

Diagram of a pipeline architecture
The basic structure of a pipeline architecture.

A “filter” in this context is a discrete processing step, such as transform, lookup, enrich, or write to a database. A filter does one thing and one thing only (the Single-Responsibility Principle).

The pipes are one-way conduits from one filter to another. They take the output of one filter and pass it on to the next one. The payloads that the filters send through the pipes can be any data format, but typically they are JSON or XML.

When To Use the Pipeline Architecture Style

The most common use case for a pipeline architecture is system integration. Any time you need to move data from one application to another, a pipeline is a very suitable style. Because you define the integration solution as a series of discrete steps, it makes for a very modular design. The filters themselves are highly cohesive and loosely coupled, meaning they do one thing only, and a filter has no dependency on other filters. This makes unit tests much easier to write, and it leads to a cleaner design that you can modify with minimal fuss. Changes to a filter mean a trivial impact to the overall integration, provided there are no changes to its output contract.

System integration is a use case that has been around for several years. Many organizations need to solve the problem of moving data from one application to another in response to some business event. These applications are often very different in terms of the data structures they expose. Connecting these applications is what system integration is all about.

Over the years, a set of common design patterns have emerged. Gregor Hohpe and Bobby Woolf captured these patterns in their seminal book Enterprise Integration Patterns. The companion website does a great job of outlining the various patterns. If you design or develop integration solutions, this book should be in your library and the website should be in your browser bookmarks.

Designing a Pipeline Architecture

System integration typically consists of a combination of Validate, Extract, Transform and Operate (VETO) steps. Begin your design effort with the target system. Determine what it needs in terms of data structure, fields and values, then figure out what it takes to get the source data to there.

For example, suppose you have an insurance claim management system. Various claim intake systems accept a new claim and publish it as an XML message on a message broker. You need to accept an XML message from the message broker, and call a web service to save the claim in the claim management system. Along the way you need to convert the XML message to JSON, and enrich the message using a database lookup to add some additional information. Here is the architecture of our example:

Diagram of a sample claim integration architecture.
Her’s a simple example of integration claims intake system with a claims management system.

Spring Integration makes this simple. You use the framework to configure the construction of the pipeline, then all you need to do is implement the logic in the filters. The framework takes care of passing the message payloads from one filter to the next.


A pipeline architecture is extensible. Should you need to add additional steps, such as logging or enhanced error handling, simply insert them in the pipeline.

The cost of implementing a pipeline architecture is quite attractive. It’s normally implemented as a monolith, so it’s relatively easy to understand and build. You don’t have the complex orchestration or choreography you normally see with distributed architectures like microservices.

When you use a framework like Spring Integration, the cost to develop is low since the framework takes care of the undifferentiated effort of moving messages from one filter to another. You just develop the filters. Moreover, Spring Integration builds on Spring Framework. If you are already using the latter as your dependency injection framework, developers can learn the former with minimal effort.

Spring Integration reads and processes each message in a separate thread. This gives some boost to performance while isolating any processing failures to the message in question.


Since a pipeline is frequently implemented as a monolith, it means changes of any size need to go through a full regression test. It is also subject to the same ceremony of an all-or-nothing deployment, along with the attendant risks.

A pipeline architecture is not very fault tolerant. If one of the filters causes a heap space exhaustion situation, the entire integration comes crashing down. You can however mitigate this with a decent monitoring and alerting strategy.

Wrapping Up

If you have a need to integrate two or more systems together, give serious consideration to a pipeline architecture. Avail yourself of the examples in the Enterprise Integration Patterns book.

How to Migrate Your Web Application to the Cloud

photography of cloud and forest trees

So it’s time to move one of your legacy web applications from your on-premise data centre to a public cloud provider such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). This may be a decision you made, or one that your leadership team made. In this post I’ll explain what you need to think about when figuring out how to migrate your web application to the cloud.

Read more

Microservices: When to Use Them

Brown and white long coated small dog wearing eyeglasses on black laptop computer


Microservices are all the rage these days. You’ve seen so many blog posts, technical articles, even job postings calling for microservices experience. So it must be the new way to architect applications, right?

Well, as with everything else in software architecture, it depends. It depends on the context you are dealing with. It depends whether the benefits of microservices outweigh their drawbacks for your situation.

In this post I’ll describe what microservices are, why they’re so good, and what their drawbacks are. I’ll finish off by giving you some guidelines to help you decide whether microservices are right for your situation.

Read more

When to Use the The Layered Architecture Style

Layer cake

A layered architecture style is one where the presentation, business and persistence layers are bundled into one deployment unit. In this article I’ll discuss what it is, and point out the situations where you would choose to adopt it.

This is one of a series of articles where I am going to discuss each of these architecture styles: Layered, Microservices, Pipeline, Microkernel, Service-Based, Event-Driven, Space-Based, and Orchestration-Driven Service-Oriented Architecture (SOA). I’ll provide some insights based on my experiences, and I’ll mention some of the Java / JVM frameworks that support the style we’re talking about.

Read more

Write More Maintainable Software With a Hexagonal Architecture

Adopting the hexagonal architecture pattern produces software that is more maintainable. It enables you to respond to changes with less fuss than many other architectural patterns. In this article I’ll explain why and offer my thoughts on this pattern.

What is a Hexagonal Architecture?

Alistair Cockburn first coined the term Hexagonal Architecture on his blog in 2005. Also known as the Ports and Adapters pattern, it is a layered architecture. It is a way of separating the domain concerns while making unit tests easier to write and changes simpler to accommodate.

Read more

Are Microservices The Same Old SOA?

Are microservices the same old SOA (Service Oriented Architecture)? Isn’t it just a rehash of the SOA hype? Let’s discuss the similarities and differences.

While microservices and Service Oriented Architecture (SOA) are two architecture patterns that deal with services, they have some differences. In fact, despite all the attention you see being given to microservices there are a couple use cases where SOA is the better choice.

So What Is a Microservice?

A microservice a stand-alone process that communicates with its clients using a lightweight mechanism, usually HTTP. Its datastore is a database of some sort. The service is the only application that interacts with its datastore. So let’s say for example you have a CustomerService that reads from and writes to the CUSTOMER table and any related child tables. Only the CustomerService performs SQL actions on those tables; any other services that do something with Customer information must go through the CustomerService using its published interface.

Read more

A Target or a Radar For Your Architecture?

Radar dish

Should you use a target architecture or an architecture radar for your organization? In this post I’ll explain what each one is. I’ll then highlight the differences between them. Finally, I’ll provide my opinion on which one is better and why I think so.

Many organizations, once they get to a certain size, see the need to decide on a particular technology stack. And rightfully so, because a proliferation of various competing technologies leads leads to an organisational drain. IT people need to become conversant and in most cases proficient at JBoss, WebSphere, and WebLogic if all three of these application servers are used. It means the organization ends up going “wide and shallow” across these three instead of “narrow and deep” on just one. In other words, they don’t develop a deep level of expertise on one application server, making problems harder to solve.

Note that this applies to programming languages, databases, application frameworks, and the key libraries an organization uses. It can also apply to server operating systems, routers and other network infrastructure, firewalls, etc.

Read more

What Makes a Successful Software Project?

Man and woman using a laptop computer

Solve a Real Business Problem

Some software projects succeed, and others fail. What’s the difference? What are the clues that indicate whether a software project is going to succeed (or fail)? Here are four characteristics.

Number one, in order for any software project to succeed, it needs to solve a business problem. The magnitude of that problem, in terms of dollar value, needs to be greater than the money (time and effort) spent on the project. Otherwise, you’re just blowing an even bigger wad of cash.

Top Management Support

Number two, it needs the commitment and support of top management. Invariably there will be conflicts with allocation of people, time and money. Without this commitment and support, a loss of these resources can cause a project to be delayed, de-scoped, or both. The end result can be disappointing to the business customers. Having this commitment means having an executive on your side to advocate for your project.

Read more