David Wheeler is often credited with the following aphorism: “All problems in computer science can be solved by another level of indirection."
Too often, Wheeler is misquoted, and software engineers attempt, instead, to add another layer of abstraction to solve their computer science problems. Abstraction is a principle that enables engineers to create reusable, multipurpose modules throughout their software systems, and both abstraction and indirection are crucial concepts in scalable, robust system development.
However, in software engineering, indirection is ultimately the driver for implementing highly decoupled components that perform separate tasks, bringing with it numerous additional organization and cross-team benefits as well.
Indirectly connected components tend to be managed by indirectly connected teams, and that's a good thing. The best way to understand what I mean is through a simple example. Let's picture a large organization that runs a successful e-commerce site. Throughout years of development, the entire platform has been broken up into multiple components. These components could consist of a website, an API, order processing, user engagement and shipping management. If the organization is large enough, it's likely that each of these components is managed by a different team.
Let's assume that the website communicates directly with the order-processing component when a user places an order. In this case, the website must have specific details as to how to handle communicating with the service. It's also likely that the order processing engineering team has had to make several design decisions to accommodate the website's workflow. Over time, these accommodations lead to increased complexity, and eventually, the communication between the two teams begins to break down.
The breakdown occurs because of the entanglement between the two teams. Traditionally, documentation provided a way for engineers to effectively communicate their deliverables to other teams. Documentation is itself a level of indirection. We typically don't want developers sending emails to each other every time there is a need to integrate two or more components. And if that situation does start occurring, we mitigate that issue with tools like Confluence to ensure that the documentation lives in "one place."
This strategy, although effective at times, is not scalable. Engineers are notoriously bad at maintaining technical documents. Luckily, technology has evolved to give us more powerful tools for adding levels of indirection in code, and in infrastructure, that can lessen the burden of maintaining highly detailed documents.
Let's get back to our e-commerce site example and explore how adding a level of indirection, through infrastructure, can optimize the communication between teams and components. Rather than having the website communicate directly with the order processing service, we can have it store the order and publish a message to inform subscribers that an order has been placed. One of those subscribers can be a queue that is processed by the order management service, which asynchronously handles the message.
This new approach has the following advantages:
• Documentation can be a simple diagram and a brief description of the expected message payload.
• Engineering teams can independently optimize and iterate on their components, maintaining separate software development life cycles.
• No team dictates the others' methodologies or workflows.
Some of the trade-offs with this approach include absorbing the added complexity of introducing the queuing infrastructure and properly handling asynchronous tasks. However, these trade-offs are typically negligible as your system scales out to more and more moving parts.
This is just one example of indirection within a software system. There are many others within code, infrastructure, networking, business processes, etc. Including levels of indirection in the short-term will pay off as the system becomes increasingly complex and would otherwise become more difficult to effectively manage.
You can immediately start taking advantage of these strategies by looking for components within your system that are tightly coupled. If you don’t have that level of visibility, then focus on the pain points during a production rollout. I find that production deployments are a great place to find out how intertwined your architecture is.
A symptom of not having proper indirection could be that components have to wait on one another for deployments. This complicates production releases and can make rollbacks difficult. Address those issues by working with the engineering teams to come up with creative solutions to indirectly connect components.
However, there is such a thing as having too much indirection, so be careful not to add complexity to your system if there is no measurable advantage for doing so.
This article was originally published on forbes.com
